RU2606875C2 - Method and system for displaying scaled scenes in real time - Google Patents

Abstract

FIELD: image forming devices.

SUBSTANCE: invention relates to displaying scaled scenes in real time. Method of displaying scaled scenes in real time, comprising receiving a video stream of frames from one or more groups of video cameras, each group having a master video camera for synchronisation on shooting parameters, which shoot required scene and located at predetermined fixed positions or on calculated positions; performing reconstruction of dynamic elements of a 3D model of a scene based on obtained video stream of frames so that static model is supplemented with new objects together with pixel by pixel referencing of texture coordinates to frames in a set; and forming and superimposing texture of objects on 3D model of scene in real time; displaying views of displaying 3D model of scene; rendering 3D model of scene in accordance with configuration of display devices; displaying rendering result on least one display device.

EFFECT: reduced computational load on processing device when constructing 3D model.

28 cl, 2 dwg

Description

FIELD OF TECHNOLOGY

The invention relates, in general, to the graphic display of multidimensional objects. More specifically, the invention relates to graphical display of a three-dimensional realistic scene from different angles based on a 3D model and video streams.

BACKGROUND

The known method described in patent application US20140015832 "System and method for implementing three-dimensional (3D) technologies", publication date 1/16/2014, patent holder (s) Dmitry Kozko, Ivan Onuchin.

The invention discloses a 3D model reconstruction system for a set of video files shooting a single scene. Video files are synchronized using a global event (for example, a flash), then on each frame the location of each video camera is calculated (it is assumed that the cameras are not fixed absolutely motionless), the 3D model is reconstructed, textures are selected and superimposed on the 3D model, weather phenomena are recognized and simulated. This system is intended mainly for tracking machines.

The disadvantage of this invention is the inability to work in real time and with high resolution, due to the fact that the cameras are not located absolutely fixed and reconstruction occurs on each frame.

The known method described in patent US6940538 "Removing a depth map according to the security camera and the created three-dimensional model", publication date 09/06/2005, patent holder Sony Electronics Inc.

A video synthesis system based on data from a real video camera and with an additional synthetic object. The system consists of one fixed video camera that provides textures for an already prepared model. A synthetic object is introduced into the model. Next, the video is synthesized from the position of a real video camera and with a field of view matching the real video camera, which displays a synthetic object as if it is part of a scene.

The disadvantage of this solution is that only one video camera is used, which does not allow you to shoot a large-scale scene and does not allow you to change the point from which the video is synthesized.

The known method described in patent US6674461 "Advanced view of morphing", publication date 01/06/2004, patent holder Matthew H. Klapman.

The invention discloses the principle of shooting a scene in which several cameras are used, from which an image is taken and superimposed on a synthesized virtual scene.

The disadvantage of this solution is that it is focused on shooting a single object, not a scene, so the authors do not solve the problem of lighting differences in different parts of the scene and setting up video cameras in real time. In addition, the system has a limitation of three cameras, which is a significant narrowing of the scope.

SUMMARY OF THE INVENTION

The present invention addresses the disadvantages inherent in existing solutions.

The technical result is to increase the efficiency of the system in real time, which reduces the computational load on the construction of the model, and also achieves the highest possible modeling accuracy of up to 1 pixel of the input image from the video camera. This technical effect is achieved through the use of both static and dynamic modeling of individual parts of the scene.

The technical result is achieved due to the method of displaying large-scale scenes in real time, which according to the invention is implemented as follows: receive a video stream of frames from one or more groups of cameras, each group having a master video camera for synchronization, shooting the desired scene and located at predetermined fixed positions or at calculated positions; reconstruction of the dynamic elements of the 3D scene model based on the received video stream of frames so that the static model is supplemented with new objects along with pixel-by-pixel mapping of texture coordinates to frames in the set; form and impose textures of objects on the 3D model in real time; set the display angles of the 3D model; render 3D models in accordance with the configuration of display devices; outputting the result of rendering to at least one display device.

After receiving frames from several cameras, they are combined to obtain a set of frames corresponding to one moment in the life time of the scene.

For a group of cameras, their relative position and direction of shooting at any time are known.

The shooting direction of the video cameras is selected based on the need to cover a certain scene, on which both stationary and moving objects are located.

In the group of video cameras, master video cameras are allocated.

At least two cameras in the set are synchronized in exposure, white balance, and frame rate per second in the output image.

At least two cameras can register images sequentially with different settings.

It is possible to simultaneously operate at least two cameras with different parameters and lens settings, for example, focal length, aperture, not limited to this.

Simultaneous operation of video cameras operating in different resolutions and with different frame rates is possible.

After receiving frames from several video cameras, images of at least two cameras are docked to obtain a panoramic view and / or to obtain an image of an object that does not fall into the field of view of one camera.

A static and / or dynamic scene model is compiled based on data received from video cameras located at predetermined fixed positions or at calculated positions.

Compose a static 3D model of the scene in another application designed for 3D modeling.

In some embodiments of the invention, when reconstructing the dynamic elements of a three-dimensional model of a scene, objects are separated from their background based on data on the colors and brightness of the objects or background to model different plans for the scene.

When reconstructing the dynamic elements of a three-dimensional model, objects are selected using data from several cameras using stereo vision algorithms.

When reconstructing the dynamic elements of a three-dimensional scene model, depth maps of the whole or part of the scene are obtained using stereo-vision algorithms.

When applying texture to a three-dimensional model of the scene, a combination of texture fragments from several cameras is performed.

When applying texture to a three-dimensional model of the scene, the textures of one object, at least two cameras, are combined into one texture of the object.

When setting up the viewing angles of the scene model, a manual or automatic selection mode is available.

In automatic mode, the angle is set based on data from external sensors.

In manual mode, the user independently changes the angle during the operation of the system of attributes of the scene model.

When rendering a three-dimensional model of a scene, rendering is performed from one or more angles at the same time.

When outputting the rendering result, additional elements are added to the created scene model that were not present on the original image frames.

When outputting the rendering result, existing scene elements are modified.

When displaying the result of rendering, additionally modeled weather phenomena are added.

This invention can be made in the form of a system for displaying large-scale scenes in real time, including one or more groups of cameras synchronized by shooting parameters, while the cameras are capable of transmitting data to a computer system and can be synchronized by shooting parameters using the master video cameras, at least one master video camera, in each of the aforementioned groups of video cameras, configured to transmit data to a computer system, one or more command processing, one or more storage devices, one or more programs, where one or more programs are stored on one or more storage devices and executed on one or more processors, the one or more programs including instructions for implementing a method for displaying large-scale scenes in real time as described above.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is an example of a building using several cameras, including a master video camera.

Figure 2 shows a block diagram of an implementation of the method.

DETAILED DESCRIPTION OF THE INVENTION

Below will be described the concepts and definitions necessary for the detailed disclosure of the invention.

A 3D scene is a part of the 3D world that must be calculated and displayed on the screen in accordance with the current observation point.

SD-model - a set of parameters and properties that is used to form an object called a three-dimensional model.

Static models refer to objects that are practically unchanged in time or considered in separate time sections.

The dynamic model allows you to see changes in the object over time.

Dynamic models reproduce changes in states (“movement”) of an object, taking into account both external and internal factors.

Rendering is a term in computer graphics that denotes the process of obtaining an image from a model using a computer program.

Lidar is a technology for obtaining and processing information about distant objects using active optical systems using the phenomena of light reflection and scattering in transparent and translucent media.

Texture - a raster image superimposed on the surface of a polygonal model to give it a color, coloring or illusion of relief.

Master camera - a camera that sets the reference settings for exposure and brightness. Based on the reference settings, the settings for a set of video cameras shooting the same scene as the master camera are calculated.

A depth map is an image on which, for each pixel, instead of color, its distance to the camera is stored. The depth map can be obtained using a special depth camera (for example, the Kinect sensor is a kind of such a camera), and can also be built on a stereo pair of images.

In computer graphics and computer vision, three-dimensional reconstruction (Eng. 3D reconstruction) is the process of obtaining the shape and appearance of real objects. The process can be performed by passive or active methods. If the shape of the model can change over time, they talk about non-rigid or spatio-temporal reconstruction. Three-dimensional reconstruction can be used to restore the appearance of objects or landscapes. Used to obtain a three-dimensional scan of the terrain, distance to the ground, determine the speed.

This invention in its various embodiments can be implemented in the form of a method, including that implemented on a computer, in the form of a system or computer-readable medium containing instructions for performing the aforementioned method.

In this invention, a system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems, and any other devices that can perform a given, well-defined sequence of operations (actions, instructions).

By a command processing device is meant an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

The command processing device reads and executes machine instructions (programs) from one or more data storage devices. Hard disk drives (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives can be used as storage devices.

A program is a sequence of instructions intended for execution by a control device of a computer or a device for processing commands.

To achieve the claimed technical result, a method for displaying large-scale scenes in real time is proposed, including the following steps.

First, the scene is shot from the same positions on which the cameras are located, after which they form a static 3D model in an application designed for 3D modeling (for example, Autodesk 3Dmax studio, but not limited to).

A video stream of frames from one or more groups of video cameras is obtained, each group having a master video camera for synchronization, shooting the desired scene and located at predetermined fixed positions or at calculated positions.

A video stream of frames is obtained from a group of cameras (FIG. 2) that shoot the desired scene. In the video camera system, groups of video cameras for which master video cameras are installed can be allocated. The field of view of the master video camera partially overlaps the field of view of each video camera in the group (Figure 1). Such groups of video cameras and master video cameras are allocated and installed based on the need to synchronize the shooting parameters of the video cameras in the group for shooting individual sections of the scene. In the particular case, such a group of cameras may consist of two cameras with a partially overlapping field of view and having synchronized shooting parameters. The video stream of frames is processed and used for dynamic modeling of the scene. Each frame is stamped with a time stamp. By time stamps, sets of frames from all cameras are formed, so that each set corresponds to one moment in the life of the scene. Fragments of the frames corresponding to the parts of the scene where dynamic reconstruction is necessary, are processed for the subsequent selection of textures. For example, if there is an area on which people walk on the stage, then we can turn on the background calculation algorithm (for example, a part of BGS) and “remove” people from the frame.

The video stream of frames can be transmitted in a compressed form, for example, using encoders H.264, VP8, MJPEG, JPEG, JPEG2000.

The video stream of frames can be transmitted as separate files. In this case, standard containers can be used, for example, WebM, OGV, MKV, MP4, TS, JPG, etc.

The dynamic elements of the 3D scene model are reconstructed based on the received video stream of frames so that the static model is supplemented with new objects along with pixel-by-pixel mapping of texture coordinates to frames in the set.

For dynamic fragments of the scene, a partial reconstruction takes place, so that the static model of the scene is supplemented with new objects, along with pixel-by-pixel mapping of texture coordinates to frames in the set. For example, a person stands out by movement, so that we simultaneously get the texture coordinates in the frame for him and his location on the stage. Also in the scene model, for example, weather conditions can be reconstructed.

Form and impose textures of objects on a 3D model corresponding to the current moment of the scene’s life in real time.

For scene elements defined in a static model, textures are also statically linked to fragments of frames from cameras. For example, it is known pixel by pixel where the house is in the frame, since the camera is at a predetermined position, so you can then take this part of the frame and pull it on the house. And for dynamic moving objects, a number of algorithms are used that work in real time, at least on a fragment of the scene, depending on the nature of the expected object. For example, people and machines are distinguished using motion detection algorithms (for example, BGS, MOG). Birds in the sky are isolated by simple contrast analysis or by segmentation algorithms.

The model is updated, and textures corresponding to the current moment of the scene's lifetime are superimposed on the model.

Set the display angles of the model.

The view can be set and changed manually by the user during the operation of the system to obtain the desired view. You can move the virtual video camera, allowing the user to see the scene from the most interesting angles in real time or later in the recording.

Also, the angle can be set automatically based on data from external sensors.

For example, to implement a “virtual” window in an elevator, the angle is set depending on the readings of the elevator height sensor. Before the start of rendering, the readings of external devices are read (Figure 2), and the position of the virtual camera is calculated in accordance with the specified configuration.

The model is rendered in accordance with the configuration of the display devices.

Display devices can also be configured to display different views of the scene. Each view can be displayed immediately by a group of display devices so that the rendered image is stretched to them. In the present invention, several rendering technologies can be applied, but not limited to, with the possibility of combining them together: Z-buffer, scanline, ray tracing, global lighting.

The result of rendering to at least one display device is output.

Video data is displayed on at least one display device via a digital High-Definition Multimedia Interface (HDMI), HD-SDI or Low Voltage Differential Signaling (LVDS) video interface, Fast Ethernet or Gigabit Ethernet network interface.

Only the final processing result is transmitted to the display subsystem on the screens, while the processing cluster of the computing cluster performs the capture from the cameras and all the necessary processing.

The proposed method can be implemented on the system described below, which includes:

- one or more groups of cameras synchronized according to the shooting parameters, while the cameras are configured to transmit data to a computer system and synchronized according to the shooting parameters using the master video camera;

- at least one master video camera, in each of the aforementioned groups of video cameras, configured to transmit data to a computer system;

- a computer system including:

(i) one or more command processing devices;

(ii) one or more storage devices;

(iii) one or more programs,

where one or more programs are stored on one or more data storage devices and executed on one or more command processing devices, and one or more programs include instructions for performing the method according to claim 1.

The video stream is received via the Fast Ethernet or Gigabit Ethernet network interface, through a specialized video transmission interface, for example, High-Defmition Serial Digital Interface (HD-SDI), composite PAL / NTSC analog signal or Universal Serial Bus (USB).

The video stream of frames can be transmitted over wireless networks, such as GSM (Global System for Mobile Communications), CDMA (Code division multiple access), LTE (Long Term Evolution), Wi-Fi (Wireless Fidelity). In some implementations of the present invention, the receipt and / or sending of data is carried out using several technologies described above, or technologies for receiving / transmitting data that will be invented after filing an application for the present invention.

Video data can be recorded to a hard disk (Hard Disk Drive, HDD), a memory card (Memory Card), a solid state drive (Solid State Drive, SSD) with various organization, such as direct attached storage (DAS), storage with network connection (Network Area Storage, NAS), RAID-array of disks (Redundant Array of Independent Disks, RAID), Logical Volume Manager (LVM).

The server can be a separate physical (dedicated) computer or virtual, that is, it is not limited by the implementation, physical configuration, or geographic location of the hardware. In particular, the server may be a plurality of computers combined into a single cloud server.

The server and at least one video source can be combined in a single hardware and / or software.

The rendering result is displayed via the digital video transmission interface High-Definition Multimedia Interface (HDMI), HD-SDI or Low Voltage Differential Signaling (LVDS), Fast Ethernet or Gigabit Ethernet network interface,

At least one monitor can be virtual, that is, video data can be displayed on virtualized equipment, not limited by the implementation, physical configuration, or geographic location of the hardware. For example, a virtual monitor can be implemented in the form of (match) a video wall consisting of many monitors. A virtual monitor can be implemented as (correspond to) a group of monitors of different users (operators) of a situation center.

It will be apparent to those skilled in the art that specific embodiments of a method for displaying large-scale scenes in real time have been described herein for purposes of illustration, various modifications are permissible without departing from the scope and essence of the scope of the invention.

Claims (41)

1. A way to display large-scale scenes in real time, including the following steps: - receive a video stream of frames from one or more groups of cameras, each group having a master video camera for synchronization according to the shooting parameters that shoot the desired scene and located at predetermined fixed positions or at calculated positions; - carry out the reconstruction of the dynamic elements of the 3D model of the scene based on the received video stream of frames so that the static model is supplemented with new objects along with pixel-by-pixel mapping of texture coordinates to frames in the set; - form and impose textures of objects on the 3D model of the scene in real time; - set the display angles of the 3D scene model; - render 3D models of the scene in accordance with the configuration of the display devices; - display the result of rendering on at least one display device. 2. The method according to p. 1, characterized in that after receiving frames from several cameras, they are combined to obtain the entire frame of the object. 3. The method according to p. 1, characterized in that for a group of cameras their relative position and direction of shooting at any time are known. 4. The method according to p. 1, characterized in that the direction of shooting of the cameras is selected on the basis of the need to cover a certain scene on which both stationary and moving objects are located. 5. The method according to p. 1, characterized in that at least two cameras in the group are synchronized by exposure, white balance and frame rate per second in the output image. 6. The method according to p. 1, characterized in that at least two cameras can register images sequentially with different settings. 7. The method according to p. 1, characterized in that it is possible the simultaneous operation of at least two cameras with different parameters and lens settings, for example, focal length, aperture, not limited to this. 8. The method according to p. 1, characterized in that it is possible the simultaneous operation of cameras operating in different resolutions and with different frame rates. 9. The method according to p. 1, characterized in that after receiving frames from several cameras, images of at least two cameras are docked to obtain a panoramic view and / or to obtain an image of an object that is not captured by one camera. 10. The method according to p. 1, characterized in that they comprise a static and / or dynamic model based on data received from cameras located at predetermined fixed positions or at calculated positions. 11. The method according to p. 1, characterized in that they first make up a static 3D model of the scene in another application designed for 3D modeling. 12. The method according to claim 1, characterized in that during the reconstruction of the dynamic elements of the three-dimensional model of the scene, objects are separated from their background based on data on the colors and brightness of the objects or background for modeling different plans of the scene. 13. The method according to p. 1, characterized in that when reconstructing the dynamic elements of a three-dimensional model of the scene, objects are extracted using data from several cameras using stereo vision algorithms. 14. The method according to p. 1, characterized in that when reconstructing the dynamic elements of a three-dimensional model of the scene, depth maps of the whole or part of the scene are obtained using stereo-vision algorithms. 15. The method according to p. 1, characterized in that when applying texture to a three-dimensional model of the scene, combining fragments of textures from several cameras is performed. 16. The method according to p. 1, characterized in that when applying the texture to a three-dimensional model of the scene, the textures of one object are combined with at least two cameras into one object texture. 17. The method according to claim 1, characterized in that when setting the display angles of the model, a manual or automatic selection mode is available. 18. The method according to p. 18, characterized in that in the automatic mode, the angle is set based on data from external sensors. 19. The method according to p. 18, characterized in that in manual mode the user independently changes the angle during the operation of the system within the configurable limits of the model. 20. The method according to p. 1, characterized in that when rendering a three-dimensional model, rendering is performed for at least two angles simultaneously. 21. The method according to p. 1, characterized in that when rendering a three-dimensional model of the scene, rendering is performed using existing rendering technologies, for example, Z-buffer, scanline, ray tracing, global lighting, but not limited to. 22. The method according to p. 1, characterized in that when outputting the rendering result, additional elements are added to the created scene model that were not present in the original image frames. 23. The method according to p. 1, characterized in that the output of the rendering result changes the existing elements of the scene. 24. The method according to p. 1, characterized in that when the output of the rendering is added, additionally simulated weather phenomena are added. 25. A system for displaying large-scale scenes in real time, including: - one or more groups of cameras synchronized according to the shooting parameters, while the cameras are configured to transmit data to a computer system and synchronized according to the shooting parameters using the master video camera; - at least one master video camera in each of the aforementioned groups of video cameras configured to transmit data to a computer system; - a computer system including: (i) one or more command processing devices; (ii) one or more storage devices; (iii) one or more programs, where one or more programs are stored on one or more data storage devices and executed on one or more command processing devices, and one or more programs include instructions for performing the method according to claim 1. 26. The system of claim 25, further comprising at least one separate video camera for obtaining a scene with independent shooting parameters configured to transmit data to a computer system. 27. The system of claim 25, further comprising at least one display for displaying the resulting image. 28. The system of claim 25, further comprising at least one depth camera for obtaining a depth map for reconstructing dynamic elements of a three-dimensional model.

Images