AT527033A4 - Method for displaying and updating a panoramic image of a landscape - Google Patents

Abstract

Method and terminal for displaying and updating a panoramic image (2) of a landscape, with a display means and with at least one camera (1), with the angle of view of which a section (6) of the angle of view of the panoramic image can be captured, wherein the position of the camera can be controlled in the range of the angle of view of the panoramic image, with the step: (i) displaying the panoramic image on the display means, characterized by the steps: (ii) panning the camera to a predetermined position and creating a recording, (iii) comparing image properties of the recording with reference values of the section of the panoramic image in place of which the recording is to be inserted, (iv) either inserting the recording in place of the section of the panoramic image if the reference values are met, (v) or adjusting the recording in terms of contrast, zoom factor and/or brightness until the image properties meet the reference values, whereupon the recording is inserted into the section of the panoramic image.

Description

The invention relates to a method for displaying and updating a panoramic image of a landscape, with a display means and with at least one camera, with the angle of view of which a section of the angle of view of the panoramic image can be captured, wherein the position of the camera can be controlled in the range of the angle of view of the panoramic image, with the step: (i) displaying the panoramic image on the display means. The invention also relates to a terminal for displaying and updating a panoramic image of a landscape comprising at least one camera, with the angle of view of which a section of the angle of view of the panoramic image can be captured, and a display means on which the panoramic image can be displayed, wherein a control and processing unit is connected between the camera and the display means, which is designed to (I) display the panoramic image on the display means

to display.

In the current state of the art, live panoramic videos are used in various applications, such as broadcast productions or remote air traffic control (ATC) towers, which are generated by a first set of cameras (the panoramic cameras). At the same time, additional PTZ cameras are used to display details of the scenery shown in the panoramic videos with a larger zoom factor (and thus more clearly visible to the viewer) (example: a landing aircraft in a panoramic video of a remote ATC tower application, which the controller must see in detail in order to be able to judge whether the landing gear of this aircraft is extended). The panoramic cameras are arranged in the immediate vicinity, ideally directly below or above the PTZ camera to be controlled. This has the following disadvantages: A second camera system is required to record the panoramic video; camera systems for recording panoramas with an angle of view of more than 180° are very expensive; the camera — e.g. a PTZ camera — and the panorama camera system must be manually calibrated to each other, which is time-consuming and must be repeated every time there is a change in position or orientation of one of the two cameras; the image processing for panorama generation causes a delay in the

video display, which affects the precise control of the PTZ camera; the

CN112991175 teaches a method and a system for creating a panoramic image from a set of individual shots of a tracking shot with a PTZ camera. Neighboring individual shots are arranged in an overlapping manner so that the image content is not interrupted. The disadvantage here is that there is no solution for arranging the panoramic image once it has been created according to useful aspects.

update.

CN106296584 discloses a method to insert a video image, e.g. from a live camera, into an existing panoramic image. This solution is rigid and can only

process a given size of the video image.

WO2018232412 relates to a method in the field of cartography, with which a sequence of images with increasing zoom factor is stored for sections of a given panoramic image in order to be able to zoom into any position of the panoramic image. A method for updating, monitoring or

Video representation of the panorama or parts thereof is not provided.

The object of the invention is to overcome the disadvantages of the prior art and to generate and update a panoramic image for monitoring or control purposes using simple equipment, whereby the data transmission should also be resource-saving. The solution should simultaneously

low-maintenance, reliable, easy or intuitive to use and cost-effective.

The method according to the invention achieves this by the steps: (ii) panning the camera to a predetermined position and taking a picture,

(li) comparison of image properties of the image with reference values of the section of the panoramic image in whose place the image is to be inserted, (iv) either insertion of the image in the place of the section of the

panoramic image if the reference values are met,

In an advantageous embodiment of the invention, a selection of image pixels of the panoramic image is used as the basis for the reference values, with the contrast values of the image pixels serving as reference values in particular. This makes updates optically more uniform and avoids inserting faulty images. Contrast values are a pixel property that is always present and can be read out.

It is also thought that a panoramic image is created beforehand with the following steps: Creating a set of images by the camera as photos in the angle of view of the panoramic image to be created by repeating the steps:

(a) taking a first photograph at a first position,

(b) Adding metadata to the first photo,

(c) Moving the camera to a next position and stitching the images together side by side with a predetermined overlap and using the metadata. This creates a panoramic image in a format that already corresponds to the system.

Overlapping areas of adjacent images can also be blended in a semi-transparent manner. This also improves the optically uniform presentation.

In a further development of the method, overlapping areas of adjacent images are blended to 10%-30% of the area of the images, which corresponds to a balance between technically safe overlapping but low area losses.

In an advantageous embodiment of the invention, at least one camera

a PTZ camera, as this is a camera type that is easy and safe to control.

to display more than one moving object in the panorama image updated.

An application can also be provided with which a terrain edge on the horizon of a landscape is determined in the panoramic image, and the step of creating a video is carried out by the camera with an adjustable zoom factor depending on the position of the camera and the angle of view of the camera in such a way that the terrain edge, as well as a predetermined angular range of the terrain horizontally below the terrain edge, are in the video. This allows an operator to always see a flying object in relation to the ground below.

and thus assess the situation more reliably.

In a further embodiment of the method, at least one graphic overlay is inserted, and the graphic overlay contains information about objects and/or events in the area of the scenery depicted in the panoramic image, whereby the operator is provided with additional important information about events in the panoramic image, so that he can already see this when looking at this

panoramic image without having to look at separate display instruments.

In an advantageous embodiment of the invention, depth information is visibly inserted in one or more areas of the panoramic image in order to provide the operator with additional important information about the states of

objects or events in the panoramic image.

It is also planned that a video is also shown in the angle of view of the panoramic image and in a separate window or other display device from the displayed panoramic image, whereby the image resolution of the separate display differs from the image resolution of the video inserted into the panoramic image. In this way, a selected section of the panoramic image can be shown to the operator in a larger size, which means that the operator can recognize details better in this larger section than in the panoramic image.

can.

displayed where they are not yet close enough to be visible.

In a further development of the process, the camera is aligned via a user interface depending on click and/or touch commands from an operator. This allows the operator to very easily adjust the section in the

Select the panoramic image to update.

According to an alternative embodiment, the orientation of the camera can be dependent on the viewing direction of the operator, which also allows a very

allows simple and intuitive control of the camera orientation by the operator.

In a further embodiment, the operator's viewing direction is determined using virtual reality (VR) glasses worn by the operator. Advantageously, the control becomes simpler and more intuitive, while the equipment required to implement the method is reduced.

The terminal according to the invention achieves the above-mentioned object together with the above-mentioned advantages in that the control and processing unit is further designed to

(ii) to pan the camera to a predetermined position to take a picture,

(li) to compare the image properties of the image with reference values of the section of the panoramic image to which the image is to be joined,

(iv) insert the image in place of the cropped panoramic image if the second set of reference values is met, or

(v) adjust the image in terms of contrast, zoom factor and/or brightness until the image properties meet the reference values, whereupon the image is merged into the section of the panoramic image.

camera type.

It is also envisaged that the control and processing unit is connected to a receiver for transponder data of an aircraft and/or radar data. This makes it possible to receive information that is contained in the panoramic image

useful as additional information that can be displayed to an operator.

The control and processing unit can also be connected to a user interface. This allows an operator as a user to easily select the section of the panoramic image that is to be updated.

Exemplary embodiments of the invention are shown schematically in the drawings and are described by way of example below with reference to the drawings. Fig. 1 to Fig. 3 show three simplified panoramic images, Fig. 4 shows the combination of a panoramic image from three individual images, Fig. 5 shows a panoramic image with graphic overlays and Fig. 6 shows a panoramic image of an airfield with a high-resolution zoomed image of an aircraft, as well as a corresponding representation within the static panorama (scaled to the correct size and embedded), as well as its separate representation in full resolution.

The procedure and the terminal offer, using the example of airspace or airfield monitoring from a distance (remote tower application), a replacement of the bandwidth-intensive video panorama with a static panorama, which saves around 80% of the transmission bandwidth of previous remote tower applications. At the same time, the air traffic controller is offered almost equivalent situational awareness in order to be able to handle one to three simultaneous flight movements safely and efficiently. The number of flight movements depends on the number of cameras used. A so-called remote tower can now also be used where the available network bandwidth between the airport and the location of the terminal for air traffic control is low. The following describes parts of a

unless one of them can be excluded in the technical context.

According to Fig. 1 to Fig. 3 and the following, a method for controlling a camera 1, such as a pan-tilt-zoom camera (PTZ camera for short), is specified using a 2-dimensional, static panoramic image 2. The panoramic image 2 is generated directly by the PTZ camera without the aid of another camera and is continuously updated. The PTZ is then controlled by click or touch inputs from an operator via a suitable user interface. This interface can be a touch-sensitive screen on which the panoramic image 2 is directly displayed. After receiving the input, the PTZ camera is automatically aligned in the direction of the corresponding position. In addition, the 2-dimensional panoramic image 2 can be stored with depth information and the zoom factor and focus can be controlled automatically.

According to a particularly easy-to-use alternative, it can be provided that the camera is operated depending on the viewing direction of the operator, whereby the viewing direction of the operator is determined with or without virtual reality glasses (VR glasses) worn by the operator.

If VR glasses are used, which the operator wears, they serve simultaneously as a display means and as a control of the PTZ camera by the operator. The current viewing direction of the operator or changes to this viewing direction are recorded using the sensors of the VR glasses and the PTZ camera is aligned according to this viewing direction or its change. In this embodiment, the panoramic image 2 is displayed in the VR glasses. VR glasses are equipped with tilt and motion sensors that enable the determination of the viewing direction and changes to the viewing direction of the operator.

As shown in Fig. 6, a live video from the PTZ camera is superimposed on the previously created static panorama image 2 in order to update the static panorama image 2 at the points where events of importance to the operator are taking place (e.g. an airplane is flying). The live video from the PTZ camera is used for this purpose

parked aircraft at the airfield may not be moved.

To better distinguish between the static panorama image and the embedded live video, the panorama image can also be displayed in black and white while the live video is displayed in color. In addition, recently updated image areas within the panorama can be displayed in color, while areas that were updated a long time ago are displayed in grayscale.

In preparation, the panoramic image 2 is generated by the camera 1 automatically moving to predetermined positions as shown in Fig. 1 to Fig. 3 and 4, which are defined so that individual shots 3 between two adjacent positions overlap by approximately 10 to 30% of the area. At each position, the camera 1 takes a static image (individual shot 3). In addition, the system saves the meta information of the camera 1 (horizontal and vertical alignment, zoom factor) for each individual shot 3.

The individual images 3 are then combined into the entire panoramic image using a well-known software algorithm, taking into account the meta information.

2 and displayed to the user.

The display may be made via any suitable display means, such as a screen, a touch-sensitive screen, a screen of a VR headset,

3 Sr

projector or similar.

The generated panoramic image 2 is 2-dimensional and initially static, i.e. motionless. The invention offers the advantage of continuously updating the panoramic image 2 at the necessary positions or sections 6. The invention deals with the task of determining which events trigger specific updates of the panoramic image 2 and how the updates are then carried out specifically and precisely.

Fig. 4 shows how a panoramic image 2 with a significantly larger angle of view is created from three exemplary individual images 3 each with a specific first angle of view. In this example, an airfield is captured from the airport floor, which represents a terrain 7, from the edge of the terrain 8 and from the sky 9.

The process for generating the panorama is fully automatic. When displayed, the panoramic image 2 according to Fig. 5 can be augmented with additional graphic overlays 10 to enable the user to better orientate themselves in the displayed terrain (for example by coloring areas where aircraft are parked or overlaying them with hatching). The graphic overlays can also include symbols and/or numbers for aircraft or flying objects in the sky 9, whereby the contents of these symbols/numbers can be, for example, the radio registration number, flight altitude or flight speed. In the following, we generally refer to objects 11 that move in the field of view of the panoramic image 2.

In addition, the panoramic image 2 can be provided with additional depth information to enable the zoom factor and focus of the camera 1 to be set accordingly. Particularly with smaller objects (e.g. drones or birds), the autofocus of the camera 1 may not work properly. If depth information about such objects is known, the focus and zoom factor of the camera 1 can be set or adjusted according to this depth information.

At least the autofocus should be supported.

screen.

The depth information is read in via suitable interfaces from appropriate data sources, such as rangefinders, radar systems, image recognition software (for object sizes). For static objects, such as mountains or buildings,

This depth information can be permanently entered into the system.

In the example of the “Remote Tower Application”, the operator is an air traffic controller.

Other use cases could involve any type of user.

The alignment of camera 1 is carried out via input devices, for example by a simple "click" or "touch" within the panoramic image 2. The camera 1, such as a PTZ camera, is then aligned exactly in this direction vertically and horizontally. If additional depth information has been stored, the zoom factor and the focus of the camera are adjusted accordingly according to the stored depth information in order to display the displayed image content precisely, e.g.

in terms of sharpness.

An alternative input device for aligning camera 1 is a sensor that determines the operator's viewing direction on the panorama 2, such as a camera that records the operator's face with a downstream image recognition system.

Software that determines the alignment of the operator's eye axes.

In addition, the orientation of the camera 1 can also be carried out using position data from objects 11, such as an aircraft, which are provided by a third-party sensor (e.g. transponder, radar, etc.). The PTZ video image can be displayed in the original resolution in a separate window 4 next to the panoramic image 2 (Fig. 6) — e.g. to provide even more information about what is happening — but in any case scaled directly in the panoramic image 2. According to Fig. 6, this gives the user the impression that he is controlling the camera 1 via a live panorama.

which would make the system even more intuitive to use and view.

If a separate window 4 is provided, the video in it is usually enlarged, i.e. displayed unscaled in full image resolution, while the video that is inserted in panorama image 2 is always scaled to the size that fits in panorama image 2. A selected section of panorama image 2 can therefore be shown to the operator in a larger size, which means that the operator can see details better in this larger section than in panorama image 2. An example of this is the contours of an aircraft fuselage: in the larger section, the operator can see better than in panorama image 2 whether the landing gear of this aircraft is extended or retracted.

By integrating third-party sensors (e.g. ADS-B transponder, radar), the panoramic image 2 can be augmented with additional real-time data from aircraft and other objects 11 to further increase the operator's situational awareness. In addition, the position data from the third-party sensor can also be used to control the PTZ cameras. Fig. 6 shows an example of a combination 12 of an aircraft with an associated graphic overlay in the (live video) section 6

Transponder and/or radar data are also used to automatically control the

Camera 1 can be used: Using such data, camera 1 can be automatically moved to those areas that are currently important for the operator. For example,

become.

The use of third-party sensors ensures that the camera 1 can already be aimed at the aircraft, even though it is not yet visible in the panoramic image 2. This makes it much easier for the operator to find the aircraft as soon as it comes into view. In the embodiment shown in Figs. 5 and 6, objects 11 are sometimes shown larger and clearer than they are in nature, which also makes it easier for the operator to identify these objects.

can be reliably identified in panoramic image 2.

There can also be two cameras 1 which generate the panoramic image 2 step by step and then update it step by step. Although the equipment required for the method and the system will then be greater, current movements of two independent objects 11 in the angular range of the panoramic image 2 can then be inserted into the panoramic image 2 independently of one another as a video sequence. Even if the sections 6 of both objects 11 overlap at a time, the invention ensures that the optically better of the two sections 6 in terms of contrast, brightness and zoom factor, for example, is displayed - and thus both moving objects 11 remain visible at this particular time - while the other is temporarily discarded.

can be (see below).

If the two objects 11 are flying objects, a temporary overlap of their trajectories in the panoramic image 2 (or even a close approach of both trajectories) does not necessarily mean a collision or a risk of collision, since the flying objects are only too close apart in terms of their spatial depth from the perspective of the panoramic image 2. Whether and how large this real spatial distance, which is initially not visible in the two-dimensional panoramic image, is is then displayed, for example, via depth information graphically attached to the objects and/or a graphic overlay 10, so that the operator can

Sy 5

to intervene.

A major challenge is therefore the overlap of the sections 6 of two PTZ windows when both cameras 1 are aligned in approximately the same direction so that the two image sections overlap. This can be relevant in that in the worst case scenario, namely when there is little contrast and brightness of an object 11 compared to the background or an unfavorable color setting of the camera, an object 11 is only seen by one camera 1 and not by the other. If the latter then covers the image of the first, the object 11 is not visible to the operator at all.

The invention solves this problem by applying the following method: After the static panoramic image 2 is generated by a camera 1, the panoramic image 2 is used as a reference image. Because the system knows the position of the new section 6 within the panoramic image 2, predetermined individual pixels, areas of pixels or all pixels in the PTZ live video image can be assigned to the individual reference pixels in the static panoramic image 2 and compared, for example, in terms of color. The comparison can also be made with regard to contrast, brightness,

zoom factor or combinations of these properties.

The system compares the majority of pixels between the static image and the live PTZ video image and adjusts the color setting (and/or contrast setting, brightness setting, etc.) of camera 1 so that the critical values of the corresponding pixels are as identical as possible. This process is carried out for all cameras 1 used so that all cameras 1 deliver a video image (or still image) with identical color.

In order to carry out this comparison of pixels as best as possible, any offset caused by the different camera positions must be compensated by the software. If two or more individual images 3 overlap, the overlapping areas are blended semi-transparently. Software modules for the graphic implementation of blends of this type with individually adjustable parameters can be found in the state of the art. In addition,

be displayed.

When assessing the risk when observing an object 11 (aircraft, drone, birds, etc.), the reference to the surroundings is always crucial. For this reason, it is necessary that the operator is shown not only the object 11 and its immediate surroundings, but also the landscape below the object 11. In the static panoramic image 2, the system determines the edge of the terrain 8 on the horizon. The system then ensures that the zoom factor of the PTZ cameras is automatically set so that both the object 11 and the edge of the terrain 8, including an adjustable vertical range (i.e. angular range) of the terrain 7 below, are visible in the live video image of the PTZ camera. This works both with automatic and manual tracking of a camera 1. In this way, the operator is able to determine the position of the object 11 in relation to the terrain 7 at any time and immediately and to make sensible decisions regarding airfield control.

1 camera

2 panoramic images

3 Single shot 4 Separate window 6 Detail

7 terrain

8 terrain edge

9 Heavens

10 Graphic Overlay 11 Object

12 Combination of flying object and associated graphic overlay

Claims (19)

1. Method for displaying and updating a panoramic image (2) of a landscape, with a display means and with at least one camera (1), with the image angle of which a section (6) of the image angle of the panoramic image (2) can be captured, wherein the position of the camera (1) can be controlled in the range of the image angle of the panoramic image (2), with the step: (i) displaying the panoramic image (2) on the display means, characterized by the steps: (ii) panning the camera (1) to a predetermined position and taking a picture, (li) comparison of image properties of the recording with reference values of the section of the panoramic image (2) at the location of which the recording is to be inserted, (iv) either inserting the recording at the location of the section (6) of the panoramic image (2) if the reference values are met, (v) or adjusting the image in terms of contrast, zoom factor and/or brightness until the image properties meet the reference values, whereupon the image is joined to the section (6) of the panoramic image (2). 2. Method according to claim 1, characterized in that a selection of image pixels of the panoramic image (2) is used as the basis of the reference values, wherein in particular the contrast values of the image pixels serve as reference values. 3. Method according to claim 1 or 2, characterized in that a panoramic image (2) is generated beforehand with the steps: creating a set of images by the camera (1) as photos in the image angle of the panoramic image (2) to be generated by repeating the steps: (a) taking a first photo at a first position, (b) supplementing the first photo with metadata, (c) moving the camera (1) to a next position, and merging the images next to each other with a predetermined Overlapping and using the metadata. 5. Method according to one of claims 1 to 4, characterized in that a blending of overlapping areas of adjacent recordings takes place on 10%-30% of the area of the recordings. 6. Method according to one of claims 1 to 5, characterized in that at least one camera (1) is a PTZ camera. 7. Method according to one of claims 1 to 6, characterized in that it is carried out with at least two cameras (1), each of which takes pictures in the form of photos and create videos. 8. Method according to one of claims 1 to 7, characterized in that an application is provided with which a terrain edge 8 on the horizon of a landscape is determined in the panoramic image (2), and wherein the step of creating a video by the camera (1) with an adjustable zoom factor depending on the position of the camera (1) and the angle of view of the camera (1) is carried out in such a way that the terrain edge (8), as well as a predetermined angular range of the terrain (7) lie horizontally below the terrain edge (8), in the video. 9. Method according to one of claims 1 to 8, characterized in that at least one graphic overlay (10) is inserted, and wherein the graphic overlay (10) contains in particular information about objects and/or events in the area of the scenery depicted in the panoramic image (2). 10. Method according to one of claims 1 to 9, characterized in that in one or more areas of the panoramic image (2) depth information be inserted visibly. 11. Method according to one of claims 1 to 10, characterized in that in addition a video is recorded in the image angle of the panoramic image (2) and in a 12. Method according to one of claims 1 to 11, characterized in that the orientation of the camera (1) is determined as a function of position data from third-party sensors, in particular from at least one transponder, at least one ADS-B transponder and/or at least one radar. 13. Method according to one of claims 1 to 12, characterized in that the orientation of the camera (1) is controlled via a user interface depending on click and/or touch commands from an operator. 14. Method according to one of claims 1 to 12, characterized in that the orientation of the camera (1) depends on the viewing direction of the operator takes place. 15. Method according to claim 14, characterized in that the viewing direction of the operator is determined using virtual reality (VR) glasses worn by the operator becomes. 16. Terminal for displaying and updating a panoramic image 2 of a landscape comprising at least one camera (1), with the image angle of which a section (6) of the image angle of the panoramic image (2) can be captured, and a display means on which the panoramic image (2) can be displayed, wherein a control and processing unit is connected between the camera (1) and the display means, which is designed to (i) display the panoramic image (2) on the display means, characterized in that the control and processing unit is further designed to (ii) to pan the camera (1) to a predetermined position to take a picture, (li) to compare image properties of the image with reference values of the section (6) of the panoramic image (2) in whose place the image is to be inserted, (v) to adjust the image in terms of contrast, zoom factor and/or brightness until the image properties meet the reference values, whereupon the image is joined to the section of the panoramic image (2). 17. Terminal according to claim 16, characterized in that the at least one Camera (1) is a PTZ camera. 18. Terminal according to claim 16 or 17, characterized in that the control and processing unit is provided with a receiving device for transponder data of an aircraft and/or radar data. 19. Terminal according to one of claims 16 to 18, characterized in that the control and processing unit is connected to a user interface. AM DS