RU2421933C2 - System and method to generate and reproduce 3d video image - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: system to generate and reproduce a 3D video image, comprising the following: a complex of generation, a complex of disparity evaluation, a complex of images synthesis and a reproduction complex. Also a method to evaluate and specify disparity values for a stereo mate is proposed, consisting in making serial iterations of specifying the primary evaluation of disparity and providing for performance of the following operations: primary calculation of disparity is made; disparity specification filter parameters are tuned depending on the iteration number; disparity specification filter core size is evaluated depending on the iteration number; disparity map is confirmed in accordance with parameters and size of the filter core; convergence criteria are calculated and evaluated, at the same time, if they are complied with, then the process of filtering is terminated; post-treatment of disparity map is carried out, and the method to generate virtual images from a colour image and according disparity map of depth with the available camera parameters.

EFFECT: production of a high-quality image based on the accurate depth map formed from stereo mates.

42 cl, 19 dwg

Description

The invention relates to devices and methods for processing stereo images and video streams and can find application, in particular, for obtaining in real time a three-dimensional scene from several cameras, for calculating the disparity between adjacent views and for creating virtual views for reproducing a three-dimensional image on auto-stereoscopic displays .

Three-dimensional television originates from the idea of perceiving the depth of the observed scene on a television screen. To realize this, several stages of video signal processing are required. As a rule, this includes the step of forming a three-dimensional image, the step of transmitting the image, and the playback step.

A three-dimensional image can be formed on the basis of real data or generated on the basis of computer graphics. For the first case, very sophisticated methods for assessing stereodispersity have been developed. At the same time, the developers of such systems were forced to take into account the need for compromise solutions between quality and performance because the calculation of a dense disparity map often requires a large number of iterations of the algorithm. For the second case, depth data; they are easily identified, since the geometry of the scene is fully defined and is stored in the depth buffer (z-buffer).

From the loss of technology, the specifications of various three-dimensional formats are known. Moreover, the most popular are solutions based on the transmission of the stereo effect through separate streams for the left and right eyes, as well as images with a pixel-by-pixel depth map. The first approach is convenient for capturing stereoscopic video images with a stereo camera, while the second approach allows the formation of many views (angles).

Various compression methods have been developed for the three-dimensional format, including using traditional codecs such as MPEG-2, MPEG-4, H.264. Other methods include spatial prediction of p- and b-frames along with conventional temporal prediction.

After decoding, a three-dimensional image must be prepared for visualization on the selected device. An example of a three-dimensional image visualization device is autostereoscopic monitors, which use a lens-raster sheet or optical barriers for spatial separation of rays. In any case, a special procedure, called blending, is required to prepare a mixed image, which is composed of many virtual images representing a scene from various angles (perspectives). From a structural point of view, a three-dimensional monitor is responsible for the adequate separation of light rays corresponding to different perspectives (viewing angles).

US patent applications 20040032980 [1] and 20030007681 [2] disclose a system and method adapted to decode video, which is accompanied by information about the depth of the scene. The task of such a system is to stereo-convert a single two-dimensional video frame. This implies the formation of a pair of stereo images in the presence of a video frame with an appropriate depth map. Such methods [1] and [2] determine the content of the depth map and the possible use of a system adapted to DIBR (depth image based rendering - image representation based on depth data). The disadvantage of this approach is that these systems are designed to generate only one stereo pair, and not to synthesize many types (perspectives). In addition, the method of calculating the depth map is not disclosed. In other words, the depth map for each video frame must be calculated before generating stereo images.

The three-dimensional television system described in US patent application 20050185711 [3] is the closest to the claimed invention in terms of the steps of generating, transmitting and reproducing stereo images. This patent application is selected as a prototype of the claimed invention. Despite the fact that the described system allows the simultaneous capture and encoding of several stereo-video streams, its disadvantage is that these video streams must be specially de-multiplexed for a particular visualization device. In this sense, the system is not universal. Visualization of virtual views is also carried out by averaging the incoming video streams using a weighted sum with constant coefficients, which generates a low quality three-dimensional image.

Thus, the problem to which the claimed invention is directed is to create a system for generating and reproducing a three-dimensional video stream, working with visualization devices known from the prior art, and providing a sufficiently high quality three-dimensional image, i.e. system free from the disadvantages inherent in the prototype [3].

The technical result is achieved by developing a system and method for generating in real time a three-dimensional scene from several cameras, calculating the disparity between adjacent views and generating virtual views (images) for three-dimensional playback on auto-stereoscopic displays.

The inventive system for the formation and playback of three-dimensional video image consists of:

- complex formation, including:

- more than one device for generating video streams;

- Means for storing and transmitting video streams over the network to the modules for extracting stereo frames and evaluating disparity;

- a formation module configured to extract stereo frames from a plurality of video streams;

- a module for pre-processing stereo frames, configured to extract multiple projections of a three-dimensional scene from stereo frames and perform procedures for compensating for lens distortion and image alignment using an epipolar constraint;

- a complex assessment of disparity, including;

- a color converter, configured to convert images into a visually uniform color space:

- the first disparity module, made with the possibility of an approximate assessment of disparity on the set of projections of a three-dimensional scene;

- the second disparity module, configured to re-refine the approximate disparity map;

- a transmission unit configured to pack a disparity map and projections into three-dimensional video frames and, after optionally compressing them, into a three-dimensional playback device;

- a complex of image synthesis, including:

- receiving unit, configured to receive and decompress three-dimensional video frames;

- a synthesis unit configured to generate a plurality of virtual views of the observed three-dimensional scene for playback on a three-dimensional display;

- a reproduction complex, including:

- a digital display with an optical system, configured to spatially separate many virtual views, so that each eye observes its own kind of reproduced three-dimensional scene;

- a multiplexer, configured to prepare a multiplexed image in accordance with the characteristics of the optical system of a three-dimensional display and many virtual views.

The complex of estimation and refinement of disparity values for a stereo pair is based on the execution of successive iterations of refinement of the initial disparity assessment. This method involves the following operations:

- perform the initial calculation of disparity;

- spend adjusting the parameters of the refinement filter disparity depending on the iteration number;

- evaluate the size of the core filter refinement disparity depending on the number of iteration;

- refine the disparity map according to the parameters and dimensions of the filter core;

- calculate and evaluate the criteria for convergence, while if they are followed, then the filtering process is stopped;

- perform post-processing of the disparity card.

The method of generating virtual images from a color image and the corresponding disparity / depth map with known camera parameters, implemented by the above system, consists in performing the following operations:

- form models of virtual cameras;

- determine the function of forming virtual images;

- evaluate the parameters of the function of forming virtual images;

- form virtual images in accordance with the function of forming virtual images;

- eliminate existing dislocations on the generated virtual images.

Thus, the main distinguishing features of the claimed invention from the prototype are that the proposed method and system provide high-quality images based on an accurate depth map formed from stereo frames. At the same time, methods for evaluating the disparity and image synthesis have been developed, which are effectively performed by parallel data calculation modules (for example, FPGA and GPU). This makes it possible to use the system in real time.

The procedure for obtaining and reproducing a three-dimensional video image includes the stage of forming stereo content, the stage of evaluating disparity, the stage of synthesis of virtual images, the stage of reproducing a three-dimensional image. The stage of content formation involves the use of several cameras, providing the creation of multiple video streams of a dynamic scene. The disparity assessment phase involves the use of computing units for real-time disparity assessment, using the resulting video streams. The stage of synthesis of virtual images involves the use of computing units to obtain synthesized virtual images in real time based on the calculated disparity. The step of reproducing a three-dimensional image involves the use of computing units and a display capable of displaying a three-dimensional image based on several virtual images generated in the previous step.

Further, the essence of the claimed technical solution is explained in detail with the involvement of graphic materials.

Figure 1. Schematic diagram of the inventive system with a description of the main stages.

Figure 2. Detailed block diagram of the inventive system.

Figure 3. The structure of the computing system used to evaluate disparity and image synthesis.

Figure 4. A detailed block diagram of a computing system with parallel channels used to evaluate and refine disparity, according to the claimed invention.

Figure 5. The algorithm of the module for pre-processing stereo frames.

6. Examples of different orientations of stereo frames.

7. A way to clarify the disparity map.

Fig. 8. The principle of filtering the disparity map based on the comparison of pixel sections: 8.1 - comparison of two-dimensional blocks, 8.2 - comparison of one-dimensional horizontal blocks, 8.3 - comparison of one-dimensional vertical blocks.

Fig.9. Evaluation of the disparity card and the results of the refinement of the card.

Figure 10. The structure of the computing system used to obtain many virtual images.

11. A method of obtaining virtual images.

Fig. 12. An example of placing virtual cameras.

Fig.13. Modeling a virtual video camera: 13.1 - a virtual image truncated by parallel planes, 13.2 - a plane of a virtual image.

Fig.14. Illustration of obtaining a virtual image with the elimination of disruption in new images.

The inventive system consists of four main complexes, namely:

a stereo content generating complex 101, a disparity assessment complex 102, an image synthesis complex 103, and a reproducing complex 104.

The stereo content generation complex 101 includes a set of (many) video cameras 106 that provide the generation of multiple video streams of the dynamic scene 105. The number of video cameras in the system must be at least two, which allows you to create video frames of the scene taken from different angles of view, that is, create a stereo effect. In the minimum acceptable configuration, the system is equipped with a stereo camera. It is proposed to use a real-time video capture device 107, a drive (memory) 108 or a network device 100 for storing and transmitting (forwarding) video streams to a complex 102 for the purpose of subsequent processing.

The video streams received from the complex 101 are transmitted through intermediate devices 107, 108, or 100 to the input of the complex 102, where a stereo frame 109 is formed from these streams, which is usually composed of several images obtained from a particular video camera. The complex 102 also includes a computing device 1010, which uses the real-time disparity estimation algorithm 1011 based on the stereo frame 109. To solve the problems associated with the high computational complexity of the disparity map estimation method, it is advisable to use additional computing modules that work in conjunction with the computing device 1010

The virtual image synthesis complex 103 includes a computing module 1012 (a presentation synthesis module) using a real-time virtual image synthesis algorithm (method) based on a stereo frame 109 and a disparity map computed using algorithm 1011. The complex 103 generates a certain number of virtual images 1013 in accordance with the parameters of virtual video cameras.

The three-dimensional playback complex 104 also includes a computing module 1014 for multiplexing images, and a display 1015 for three-dimensional reproduction of virtual images 1013 formed in the complex 103. The three-dimensional display 1015 includes an LCD panel 1016 on which the multiplexed image is displayed. An optical device 1017 (lens-raster sheet, parallax barrier, etc.) separates the multiplexed image in space so that each human eye 1018 observes a corresponding virtual image 1013 for normal perception of a three-dimensional scene.

Figure 2 shows a detailed block diagram of a system according to the claimed invention. The stereo content generation complex 201 includes a video capture module 208 that extracts stereo frames 209 from devices responsible for generating stereo content. Typically, stereo frames 209 can be obtained from a set of 205 cameras. In addition, stereo frames 209 can be recorded and saved on a memory device in the format of video files 207. Video capture module 208, in the case of recorded stereo content, decompresses and analyzes the video file to extract the stereo frame 209. Another content provider is network television - IPTV 206, which provides video stream transmission through the network. In this case, the purpose of the video capture module 208 is to connect to the host computer and provide a video stream and extract the stereo frame 209.

The resulting stereo frame 209 typically consists of two or more pictures taken by a particular video camera. Such pictures can be grouped in descending form 602 or form 601 from left to right (see Figure 6). For further processing, such images should be cut and individual images formed corresponding to each image. This procedure is performed in the stereo frame pre-processing module 210. In addition, module 210 performs image correction and lens distortion compensation. The improved images obtained at the output of the complex 201 then go to the input of the disparity assessment complex 202.

The color conversion unit 211 converts the enhanced images into a perceptually uniform color space, such as Lab, for more accurate pixel comparison in the disparity estimation units 212 and 213. The output of the color conversion unit 211 is connected to the initial disparity estimation module 212. The initial assessment of disparity is carried out using traditional stereo image combining algorithms known from the prior art, as well as based on images with chaotic noise. An image with chaotic noise is defined as a grayscale image. The intensities of pixels in images with chaotic noise are arbitrarily selected from a range including pixels from the minimum intensity to the maximum intensity. For eight-bit images, the intensity range is from 0 to 255.

Module 212 generates approximate disparity maps, which are then refined in disparity refinement module 213 according to the claimed invention. The initial disparity assessment module 212 and the disparity refinement module 213 perform stereo alignment and refinement using a computing device with multiple parallel computing units.

The refined high-quality disparity map 214 is a product of the disparity refinement module 213. It will be appreciated that, at a disparity estimation step 202, several disparity cards may be generated for each image extracted from the stereo frame 209. Several disparity cards combined with the corresponding multiple images provide a more accurate synthesis of the virtual image. However, for simplicity, we illustrate the case with one disparity card.

The resulting disparity card 214, together with the corresponding image from the three-dimensional video frame, can be recorded in the storage device as a video file for subsequent transmission to the three-dimensional playback device through the transfer unit 215. It is clear that the transfer can be carried out without optional storage in the storage device. The transmission unit 215 composes the disparity card (s) and the image (s) in three-dimensional video frames and, after optional compression, transmits them to the three-dimensional playback device. Such a transmission can be accomplished by connecting the transmission unit 215 to the receiver module 217 in a three-dimensional playback device. Such a connection can be carried out both by wire and by radio channel.

In the complex 203 for synthesizing images from images and disparity cards extracted from a three-dimensional video frame, a certain number of virtual images corresponding to the number of virtual cameras is obtained using module 217. It should be noted that the number of virtual images may be greater than the number of images obtained in a three-dimensional video frame. The number of virtual images that will be generated depends on the number of views that a three-dimensional display can reproduce. When the number of views and parameters of the virtual cameras are determined, the image synthesis module 217 generates the required number of virtual images 218 in accordance with the claimed method.

A set of virtual images 218 is fed to the input of a three-dimensional playback complex 204, which includes a three-dimensional display 221 and an image multiplexing unit 219. The three-dimensional display 221 is a conventional digital display with an optical system, such as a lenticular sheet, parallax barrier, etc. Such an optical system allows you to see only certain pixels, depending on the position of the observer. Thus, the image multiplexer 219 generates a multiplexed image 220 in accordance with the optical system and the plurality of virtual images 218. Then, the multiplexed image 220 is displayed on a three-dimensional display 221, and the optical system performs demultiplexing by optical means.

Figure 3 shows the structure of a computing device used to evaluate disparity and image synthesis. Such a device includes a data memory unit 301, which is used to store images, disparity cards, views, camera parameters, and intermediate data needed for calculations. The input / output device 303 acts as a communicator with external devices such as a data memory driver, a network, cameras, and a three-dimensional display. An input / output device 303 transmits and receives data from a data memory unit 301.

Computing unit 304 consists of computing modules with mass parallelism, each of which can work with different data in parallel in accordance with the instructions recorded in the programs in block 302 of the program memory. Block 302 program memory stores instructions in accordance with the claimed method for evaluating disparity and image synthesis. The program memory unit 302, the computing unit, and the input / output device are interconnected by a common data bus 305.

Figure 4 shows a detailed block diagram of the parallel computing architecture used to evaluate the disparity and refine the disparity map according to an embodiment of the claimed invention. The input for the disparity assessment consists of refined images and an initial disparity map 403. At least two refined images are needed for disparity assessment. One image is a reference image 401, and the other image is a compatible image 402. The initial disparity map 403 stores the disparity values specified in the previous steps. Such an initial disparity card 403 is specified in accordance with the claimed invention, and as a result, an updated disparity card 410 is obtained.

The method of assessing disparity and refinement according to the claimed invention is designed in such a way that the process of updating the disparity map for one line can be carried out independently of the process of clarifying other lines. This important property can be used by computing device 406 with a large number of modules performing parallel calculations to significantly speed up the process of evaluating disparity. Computing device 406 operates in accordance with program 412 for evaluating disparity and refining it, which is recorded in block 411 of program memory. Program 412 corresponds to a method for assessing disparity and refining it, which is described in detail below.

Module 404 splitting the image into lines divides the incoming image 401, 402, 403 into many lines that can be processed independently, and stores many lines in the buffer 405 of the data memory. It should be understood that many rows can include either horizontal rows of an image or vertical columns of an image. The number of lines depends on the size of the image and is equal to the width or height of the processed image.

Computing device 406 includes several computing modules, each of which processes a set of multiple lines assigned to it in parallel with other modules and generates a corresponding line with specified disparity. Lines with specified disparity are stored in the data memory buffer 408. Row alignment module 409 generates a refined disparity map 410 from strings with refined disparity stored in a data memory buffer 408.

It should be noted that the number of computing modules may not be equal to the number of data sets. To solve this problem, the calculation planning module 407 matches the number of lines generated by the image splitting module 404 with the number of computing modules that are unloaded and can be used for calculations. After the computing module starts processing the data, it is marked as "busy". After the formation of the line with the specified disparity, the computing module is marked as "free". The calculation planning module 407 also controls the row alignment module 409, sending a control signal when new rows with refined disparity become available.

Figure 5 shows the sequence of operations of the module 210 preprocessing stereo frame 501. The stereo frame 501 may have a left-right orientation, as shown in Fig.6, view 601 or "top-bottom", as shown in view 602. Orientation module 502 the stereo frame automatically determines the type of orientation based on the data on the aspect ratio of the frame sides. Then, knowing the orientation of the stereo frame and the number of images, the coordinates of the images 504, 505 in the stereo frame separation unit 503 are determined. The lens distortion compensation module 506 and the refinement module 507 perform standard procedures known in the art. In this manner, a reference image 509 and a compatible image 508 are used, which are used to evaluate the disparity in subsequent steps.

Consider the step-by-step process of implementing the method for evaluating the disparity card and its refinement on a pair of stereo images (Fig. 7). The first step in the method for refining the disparity map is to adapt the filtering strength according to the iteration number (Step 701). In a preferred embodiment of the invention, the equation for adapting the filter strength in accordance with the iteration number (function σ (k)) has a linear form and is represented as:

σ (k) = a ₁ · k · b _1,

where k is the iteration number,

α ₁ , b ₁ - linear coefficients.

Another form of the function σ (k) is also acceptable, which does not affect the scope of protection of the claimed invention.

The next step in the implementation of the method is to evaluate the size of the filter core (Step 702). In a preferred embodiment of the invention, the equation for estimating the size of the filter core in accordance with the number (function KS (k)) has a linear form and is expressed as:

KS (k) = a ₂ · k + b _2,

where k is the iteration number,

a ₂ , b ₂ - linear coefficients.

A different form of the function KS (k) is also acceptable, which does not affect the scope of protection of the claimed invention.

After calculating the filter strength and estimating the size of the filter core, the disparity map is refined for the reference image using information from the combined image (Step 703-Step 704). Hereinafter, the reference image is defined as a color image from a stereo pair for which a disparity assessment has been carried out. And the combined image is defined as another color image of a stereo pair.

The disparity map at the kth iteration is represented as

where d _k (x _c , y _c ) denote the disparity map at the kth iteration for the current pixel with coordinates (x _c , y _c ),

d _k-1 (x _r , y _r ) denote the disparity map at the (k-1) th iteration for the reference pixel with coordinates (x _r = x _c + p, y _r = y _c + s),

w _γ denotes the weight of the reference pixel,

до

в направлении X,index p varies from

before

in the direction of X,

до

в направлении Y,index s varies from

before

in the direction of Y

.the normalization factor is calculated as

.

In a preferred embodiment of the claimed invention, the filter window dimensions, namely L and K, are defined as L = K = KS (k) using the filter core size estimation function calculated in step 702. However, independent dimensioning of L and K is also acceptable. This could be necessary for devices that use line-by-line memory for processing. In such cases, the radius of the filter core in the vertical direction is limited by the number of lines, and the radius of the filter core in the horizontal direction can be arbitrarily set to the desired value.

To reduce the calculations, the disparity filter can be divided into independent operations in two passes. The first pass is line-by-line processing (Step 703). The second pass is column processing (Step 704).

where d _rowk (x _c , y _c ) is the result of a row- _wise filtering procedure,

d _k (x _s , y _s ) - the final result of filtering by columns,

the normalization factor for the line filter is calculated as

and the normalizing factor for the column filter is calculated as

Filtering is performed in a window of size L × K. All pixels that belong to this area are defined as reference pixels. The pixels in the aligned image, which are displayed by disparity vectors from the reference pixels, are defined as compatible pixels (Fig. 8). The proposed filter assigns higher weights for pixels, which are more similar to the current pixel. In a preferred embodiment of the claimed invention, the method of calculating the weight of the disparity filter uses a comparison of a pixel neighborhood, rather than a comparison of individual pixels. This is done in order to strengthen the criteria for pixel similarity.

In this method of assessing and refining disparity, the weight reflects the degree of similarity (similarity) of the current pixel with the reference pixel, as well as the degree of similarity of the reference pixel with a compatible pixel. Accordingly, to determine the similarity of the current pixel with the reference pixel, the neighborhood of the current pixel is compared with the neighborhood of the reference pixel. Then, to determine the similarity of the reference pixel with the compatible pixel, a comparison of the neighborhood of the reference pixel with the neighborhood of the compatible pixel is performed (examples of comparisons are shown in Fig. 8 by arrows). In this case, the weights of the disparity filter are calculated as follows

where C () denotes a function used to compare neighborhoods of a pixel,

σ and σ _t are the parameters of the filter strength regulators. In a preferred embodiment of the claimed invention, they are defined as σ _r = σ _t = σ (k) using the filter strength adaptation function calculated in step 701. However, independent adjustment of σ _r and σ _t is also acceptable, which allows differentiation of the fine for pixels from the reference and compatible images.

To improve the performance of the method with small losses in quality, it makes sense to use a comparison of only reference pixels instead of pixel neighborhoods. This can be considered as a boundary case of the method, when the sizes of the neighborhoods of the pixel, namely, M and N, are equal to unity. Using this approach to calculate filter weight can result in a somewhat noisy disparity map. Therefore, some post-processing is required to eliminate the discrepancies in the disparity map. To this end, the disparity post-processing step is performed (Step 707) for the final disparity assessment.

An important step in the method for clarifying disparity is the calculation and evaluation of the convergence criterion (Step 705 - Step 706). During each iteration, the convergence criteria for the filtering process are evaluated using the method of refining the disparity. In a preferred embodiment of the claimed invention, two methods for evaluating the convergence criterion are described:

- Using the analysis of the difference between adjacent estimates of the disparity map

- Using a hard threshold value of the iteration number of the algorithm.

The first method for checking the convergence of the algorithm is to calculate the residual image (difference) between adjacent estimates of the disparity map. The sum of the residual pixels should not be greater than the threshold T _{dec1 of} convergence of the disparity estimation estimate. It is formulated as follows

,

,

where d _k and d _k-1 are disparity estimates at the kth and (k-1) -th iteration of the algorithm.

The second method for evaluating the convergence criterion can be formulated as a hard threshold value of the number of iterations. If the number of iterations exceeds the threshold T _{dec2 of the} convergence of the disparity estimation, then the filtering process is stopped. It is formulated as follows

k≥T _dec2,

where k is the number of the current iteration of the disparity assessment.

The first criterion for convergence can be used to rigorously evaluate the results of the algorithm. While the use of the second criterion is preferable for devices with a minimal configuration, where the simplicity of the device plays an important role.

The final step in the method for refining disparity is to perform the post-processing of disparity (Step 707). In a preferred embodiment of the claimed invention, a median filter is used as a post-processing means, which is a simple and reliable solution for eliminating impulse noise, as well as small distortions of the disparity map.

Fig.9 shows the results of the assessment of the disparity of the proposed filter according to the invention. The left image in a stereo pair for which a disparity assessment was performed is shown in Fig. 9 (view 9.1). Figure 9 (view 9.2) presents the result of the assessment of the disparity of the prior art filter [4], and Figure 9 (view 9.3) presents the result of the assessment of the disparity of the proposed filter according to the invention. From Fig.9 (view 9.2) and (view 9.3) it can be seen that the inventive method gives the best result, especially for homogeneous (smooth) portions of the image and areas with a periodic structure. At the same time, the effectiveness of the method is comparable with the efficiency of the known filter [4].

Figure 10 shows a detailed block diagram of a parallel computing system used to generate multiple images (views) according to this embodiment of the claimed invention. Input data for generating a plurality of images is generated from the reference color image 1001, the disparity card 1002 and the camera parameters 1003. At the system output, N images 1010 are synthesized.

The method of generating virtual images according to the claimed invention is designed in such a way that any line of N virtual images can be synthesized independently of other lines. Such an important property can be used by computing device 1006 with a large number of modules performing parallel computing to significantly accelerate the synthesis of virtual images. Computing device 1006 operates in accordance with a program 1012 for generating a plurality of images that are recorded in a program memory 1011. Program 1012 corresponds to a method for generating virtual images, which is described in detail below.

Module 1004 splitting the image into lines splits the incoming image 1001, 1002 into sets of lines that can be processed independently, while such many lines are stored in the buffer 1005 of the data memory. Keep in mind that many lines consist of horizontal lines of the reference image. The number K of row sets depends on the size of the images and may be equal to the height of the reference image. For devices with a minimal configuration, the K coefficient may be less than the height of the reference image. In such a case, the reference image should be loaded into the data storage unit 1005 via a separate data loader.

Computing device 1006 includes several computing modules, each of which processes a set of multiple lines assigned to it in parallel with other modules and generates a corresponding line of N images. During processing of the lines of the reference image, one computing module of the computing device 1006 generates N corresponding lines of virtual images. In other words, the computing module generates a corresponding line of the virtual image for each virtual image. Thus, the data memory buffer 1008 must be able to write K · Width · N values, where N is the number of virtual images, Width is the width of the reference image, K is the number of lines available in the data memory buffer 1005. After all the lines of N virtual images have been generated, the line combining unit 1009 will generate N images 1010 from the image lines stored in the data memory buffer 1008.

It should be noted that the number of computing modules may not correspond to the number of data sets. To solve this problem, the calculation planning module 1007 maps the number of lines generated by the image splitting module 1004 to the number of computing modules free to use in the calculations. After the computing module starts processing the data, it is marked as "busy". When a string of N images is formed, the computing module is marked as “free”. The calculation scheduling unit 1007 also controls the line combining unit 1009, sending a control signal when new lines of the plurality of images are ready.

Consider the phased operation of the method of generating virtual images based on the reference image received from the video camera and the corresponding disparity card (Figure 11). For normal processing, the method requires the availability of data on the parameters of the video camera, namely, the projection parameters of the camera, as well as the orientation parameters of the camera in space, which are essential for simulating a virtual camera. These parameters can be predefined and stored in ROM. Another possibility is the transfer of these parameters together with the image and disparity data. The first step of the virtual image generation method is to set the number of virtual images to be generated (Step 1101). The number of images can be strictly determined and recorded in ROM. This can be claimed in a situation where data on the playback device, as well as on the number of images to be generated, are known in advance. However, this requires reprogramming the ROM in accordance with the new number of images when changing the playback device. You can also write these virtual images to RAM. In this case, data should be provided on the number of virtual images and on the parameters of the video camera.

It is known from the prior art that the projection transformation of a real camera is described by a projection matrix M, which consists of two matrices, a projection matrix K and a camera orientation matrix in space [R | t]. Where K defines the projection rule, that is, how the three-dimensional point of the real world is displayed on the image plane inside the camera, while the matrix [R | t] is responsible for describing the position of the camera relative to the origin of the world coordinate system. In the future, we define the projection matrix M of the camera as

M = K · [R | t],

where K is the projection matrix,

[R | t] is the matrix of the orientation of the camera in space.

The projection matrix K is defined as

_,

_,

where (f _x , f _y ) is the focal length for modeling a rectangular photosensitive element,

(x ₀ , y ₀ ) - the main point, the coordinate of the center of the image.

,The camera orientation matrix in space [R | t] is defined as

,

where R is the rotation matrix describing the orientation of the camera with respect to the origin of the world coordinate system;

t is the shift vector from the origin of world coordinates.

Since the goal is to generate a set of images for sequentially outputting the image stream to the reproducing apparatus, it is possible to simplify the initial data required for normal processing. The following parameters are necessary for the successful generation of multiple images.

1. The matrix K inherent in the video camera.

2. The distance b between adjacent virtual images.

3. The number n of images (angles) located on the side of the real camera (left or right side).

The configuration of virtual and real cameras is shown in Fig. 12. If n is the number of images on one side of a real video camera, then the total number of images can be expressed as N = 2 · n + 1. Without loss of generality, we can arbitrarily fix the beginning of the world coordinate system in the center of a real camera (C ₀ in FIG. 12).

Then the rotation matrix is reduced to the identity matrix:

On the other hand, it is understood that the optical centers of all virtual cameras must lie on one horizontal line. Therefore, the shift vector for cameras that are located on the right hand of a real camera can be formulated as:

where i is the number of virtual images located on the right hand of a real camera. Therefore, the shift vector for cameras that are located on the left hand of a real camera can be formulated as:

where i is the number of virtual images located on the left hand of the real camera. As a result, the matrix [R | t] is represented as:

The next step in implementing the method of generating virtual images is to simulate virtual cameras by the number of images (Step 1102). The main goal of modeling virtual cameras is to establish a correspondence between a real camera and a virtual camera. This allows you to correctly submit virtual images to any device that supports the usual projection equations. Modeling a virtual camera consists in constructing a projection matrix P based on the matrix K of the real camera, as well as constructing a surveillance matrix V based on the matrix [R | t] of the real camera. The advantage of this description of a virtual camera is the ability to combine real and synthesized content in automatic mode.

Consider the process of constructing the matrices P and V in detail.

The matrices P and V should be considered as analogues of the matrices K and [R | t]. The purpose of the matrix P is to describe the conversion from a three-dimensional image to a two-dimensional image. In other words, P describes the projective transformation of a three-dimensional set of points contained in a virtual camera-truncated image from a virtual camera into a set of two-dimensional points. One important difference between the matrix P and the matrix [R | t] is the ability to save not only two-dimensional coordinates, as in [R | t], but also to save the z-coordinate of a three-dimensional point, thus making integration with synthesized content or merging possible various real images.

where N and F are the foreground and background of the cone of view of the virtual camera. The viewing cone is shown in FIG. 13 (view 13.1).

L and R are the coordinates of the left and right boundaries of the plane of the virtual image (Fig.13 (13.2).

T and B are the coordinates of the upper and lower boundaries of the plane of the virtual image (Fig.13 (13.2).

It is necessary to determine the parameters of the matrix P corresponding to the parameters of the matrix K. For this, the equations are determined:

,

,

,

,

where W is the width of the image plane;

H is the height of the image plane.

Using the above equations and setting the corresponding values for N and F, we can derive 1 projection matrix P corresponding to the matrix K. Now we can derive the observation matrix V of virtual images from the matrix [R | t] of the camera. The purpose of the observation matrix V coincides with the purpose of the matrix [R | t] - that is, both matrices are used to describe the location of the camera with respect to the zero point of the world coordinate system. Matrix V is defined as follows:

.

.

From here the complete projection equation is derived:

G = P · V,

where G - is the full projection matrix of the virtual camera.

After the corresponding G-matrix is output for each virtual camera, it is possible to start generating a plurality of images in accordance with the virtual imaging function (Step 1104). The virtual image generation function is designed to create multiple virtual views (images) obtained from an image captured by a real camera and a depth map (disparity), along with previously calculated projection matrices G _{N of} virtual cameras. Hereinafter, G _{N is} described as a set containing a matrix G for each virtual camera. In order to generate a virtual image from the real image and the corresponding depth map, it is necessary to re-project the image pixel back into three-dimensional space, and then project the reconstructed three-dimensional points in the direction of the virtual view described by its projection matrix G (Fig. 14). Re-projecting a pixel into three-dimensional space can be formulated as

where x is a vector describing the uniform pixel coordinates (x, y, 1);

X is a vector describing the uniform coordinates (X, Y, Z, 1) of a three-dimensional point for pixel x;

z is a scalar representing the depth value from the depth map corresponding to pixel x;

M ⁺ is the pseudo inversion of the projection matrix M of the video camera, so MM ⁺ = I;

C is a vector representing the coordinates of the center of the camera. Given the previous considerations about the location of virtual cameras in space, the vector is described as (t _x , 0, 0, 1).

For each pixel in the reference image, it is necessary to reconstruct the three-dimensional point X using equation (1). After that, the reconstructed point cloud should be projected onto the projection plane of the virtual camera. The projection from three-dimensional space into two-dimensional is as follows:

where G is the projection matrix of the virtual camera, as defined previously; X is a vector describing a three-dimensional point corresponding to a reference color image;

Y is a vector representing the homogeneous coordinates of a two-dimensional point projected onto the plane of the virtual camera. Along with two-dimensional coordinates, the z-coordinate of the three-dimensional point X is preserved. The vector is described as follows (x _v , y _v , z _v , 1).

Now, it is possible to re-formulate equations (1) and (2) to obtain a more compact formulation of the virtual imaging function. To do this, replace the vector X in equation (2) with its definition from equation (1) and obtain the following expression after opening the brackets

Now, it is possible to divide equation (3) into parts that depend on the input data (i.e., the value of the depth and image coordinates) and parts that are independent of the data and, therefore, can be pre-calculated at the stage of evaluating the parameters of the virtual image formation function ( Step 1103). Parameters independent of the data include the scalar product of the matrices G and C, which is defined as the vector A (A = G · C). The second parameter, which can be calculated in advance, is the scalar product of the matrices G and M ⁺ . The resulting matrix B is defined as B = G · M ⁺ . Now we can re-formulate equation (3) as follows

Equation (4) represents the function of generating virtual images according to the described method for generating virtual images. Using equation (4), a virtual image associated with a specific virtual camera can be easily calculated by transferring the image and depth data to the coordinates represented by x. For all virtual cameras N, matrices A and B must be calculated and stored. Then, the algorithm for generating a virtual image synthesizes virtual images according to camera parameters, starting with A ₁ and B ₁ , and ending with A _N and B _N.

Once all the necessary virtual images have been synthesized, the virtual image generation method performs the dislocation processing (Step 1105). An illustration of disclusion is shown in Fig. 14. In this drawing, the areas of disclusion are shown in black. Disclusion is an area in a virtual image that becomes visible from a virtual point of view, as opposed to a reference image, where this area is obscured by some kind of object. Since the correct processing of dislocations requires additional information, for example, another image of the scene, captured from a different shooting point, to fill the "holes" that have appeared. However, when the virtual images are not spaced far from the reference image, interpolation from the boundary pixels of the dislocation region gives quite acceptable results. In the process of generating virtual images, the dislocation area can be marked with a digital sign for further processing.

A disclusion filter is formulated as a special case of a bilateral filter, when filter weights are calculated only for pixels that lie outside the disclusion region. The result of summing the weights is attributed to the pixels that lie in the area of disclusion. This type of filter distributes information from certain areas of the image to undefined areas of the image (disocclusions). The filter can be used repeatedly with increasing radius for iterative filling in the area of disclusion. A disruption filter for a pixel I (x, y) in a rectangular window with size K in the horizontal direction and size L in the vertical direction is formulated as

where I (x + i, y + j) is the original image pixel;

I ^∈ (x, y) filtered pixel;

W (x + i, y + j) is the weight of the pixel I (x + i, y + j). Weight W is formulated as follows

where ΔС _pq is the distance between the points p (x, y) and q (x + i, y + j) in the color space, calculated as the Euclidean distance; ^ΔJ _pq is the Euclidean distance between the points p and q of the image area, c ₁ and c ₂ are some predetermined values related to the smoothing and refinement effect. According to equation (6), if the intensity of the original pixel is marked as a dislocation I (x + i, y + j) = mark, then the corresponding weight assumes a zero value. It follows that the disclosure pixels will not participate in the filtering process defined by equation (5) and the refined image I ^∈ (x, y) will be formed only with the help of reliable pixels weighted by color similarity and spatial distance.

For large areas of disclusion, you can apply the disclusion filter according to equation (5) several times with a variable filter core size. The variable kernel size for the DKS (k) disclosure filter, which is adaptive to the iteration number, is defined as

_{DKS (k) = a d ·} k + b d,

where k is the number of iterations,

a _d , b _d - linear coefficients.

It is also possible to use another form of the function DKS (k). But this does not affect the scope of protection of the claimed invention.

Another improvement in the disclusion filter includes the ability to implicitly use recursion when only one buffer is needed to enter the set I and output the set I ^∈ . In this case, the filtering result of the pixel I (x-1, y-1) of the disclusion can be used to filter the next pixel I (x, y) of the disclosure. This can significantly reduce the number of iterations.

A third improvement in the disruption filter is to separately calculate the filter when image rows and then columns are processed. A separable filter is defined as

where I _row (x, y) means the result of filtering performed line by line;

I ^∈ _sep (x, y) is the final result of filtering performed on image columns.

LINKS

[1] US patent application No. 2004/0032980 P.V. Harman "Image Conversion and Encoding Techniques".

[2] US patent application No. 2003/0007681 R.G. Baker "Image Conversion and Encoding Techniques".

[3] US patent application No. 2005/0185711, H. Pfister, W. Matusik "3D television system and method".

[4] WO patent application No. 2008/041167, F. Boughorbel, "Method and Filter for Recovery of Disparities in a Video Stream".

Claims (42)

1. The system for the formation and playback of three-dimensional video images, consisting of:
formation complex, including:
more than one video stream generation device;
means for storing and transmitting video streams over the network to the modules for extracting stereo frames and evaluating disparity;
a generating module configured to extract stereo frames from a plurality of video streams;
a module for pre-processing stereo frames, configured to extract multiple projections of a three-dimensional scene from stereo frames and perform procedures for compensating for lens distortion and image alignment using an epipolar constraint;
the disparity assessment complex, which includes:
a color converter configured to convert images into a visually uniform color space:
a first disparity module configured to approximate disparity on a plurality of projections of a three-dimensional scene;
a second disparity module configured to re-refine the approximate disparity map;
a transmission unit configured to pack the disparity map and projections into three-dimensional video frames and, after optional compression, transfer them to a three-dimensional playback device;
a complex of image synthesis, including:
a receiving unit configured to receive and decompress three-dimensional video frames;
a synthesis unit configured to generate a plurality of virtual views of the observed three-dimensional scene for playback on a three-dimensional display;
playback complex, including:
a digital display with an optical system, configured to spatially separate many virtual views, so that each eye observes its own kind of reproduced three-dimensional scene;
a multiplexer configured to prepare a multiplexed image in accordance with the characteristics of an optical system of a three-dimensional display and a plurality of virtual views. 2. The system according to claim 1, characterized in that the device for generating video images are an arbitrary set of video cameras observing the same region of space, configured to form multiple video streams of a three-dimensional dynamic scene; means for transmitting stereo images over a network of a storage device configured to store stereo images in the form of video files. 3. The system according to claim 1, characterized in that the digital display is equipped with an optical system configured to implement the principles of obtaining the effect of a three-dimensional image through the use of a lens-raster sheet or parallax barrier. 4. The system according to claim 1, characterized in that the plurality of video images are stereo pairs of a dynamic scene. 5. The system according to claim 1, characterized in that it comprises means for synchronizing multiple cameras. 6. The system according to claim 1, characterized in that the color converter is configured to convert RGB images into a Lab color space. 7. The system according to claim 1, characterized in that the first disparity module is configured to initialize the disparity card through the stereo-matching procedure. 8. The system according to claim 1, characterized in that the first disparity module is configured to initialize the disparity card with arbitrary values. 9. The system according to claim 1, characterized in that for each projection of a three-dimensional scene, a separate disparity map is provided. 10. The system according to claim 1, characterized in that the storage means includes the ability to store video files consisting of disparity cards, grouped with the corresponding projection of a three-dimensional scene. 11. The system according to claim 1, characterized in that the disparity assessment blocks and the playback blocks are made in the form of a computing device consisting of mass parallelism calculation blocks, each of which is designed for parallel data processing in accordance with the procedure recorded in the program memory. 12. The system according to claim 11, characterized in that the computing device is configured to clarify disparity values for each individual line of the disparity card independently of other lines of the card. 13. The system of claim 12, wherein the line on the disparity map is both an image row and an image column. 14. The system of claim 1, wherein the stereo frame pre-processing module is configured to automatically determine the orientation of the stereo frame based on the aspect ratio of the stereo frame. 15. A method for evaluating and clarifying disparity values for a stereo pair, which consists in carrying out successive iterations of the refinement of the initial disparity assessment and providing for the following operations:
perform a primary calculation of disparity;
adjusting the parameters of the disparity refinement filter depending on the iteration number;
evaluate the size of the core of the disparity refinement filter depending on the iteration number;
refine the disparity map according to the parameters and dimensions of the filter core;
convergence criteria are calculated and evaluated, and if they are met, the filtering process is stopped;
perform post-processing of the disparity card. 16. The method according to clause 15, wherein the original stereo image is converted from RGB color space to Lab color space. 17. The method according to clause 15, wherein the initial disparity is represented as a noise image, the image being represented as a grayscale image, and the pixel intensity in the image is set randomly in the range from the minimum pixel intensity to the maximum pixel intensity, which for 8 -bit images corresponds to an intensity range from 0 to 255. 18. The method according to p. 15, characterized in that the adjustment of the disparity filter in accordance with the iteration is performed according to the formula σ (k) = a ₁ · k · b ₁ ,
where k is the iteration number; a ₁ , b ₁ - linear coefficients. 19. The method according to clause 15, wherein the evaluation of the size of the filter cell in accordance with the iteration is performed according to the formula
KS (k) = a ₂ · k + b _2,
where k is the iteration number; a ₂ , b ₂ - linear coefficients. 20. The method according to p. 15, characterized in that the disparity filtering at the k-th iteration is performed according to the formula

where d _k (x _c , y _s ) denotes the disparity map at the kth iteration for the current pixel with coordinates (x _s , y _s );
d _k-1 (x _r , y _r ) denotes the disparity map at the (k-1) th iteration for the reference pixel with coordinates (x _r = x _c + p, y _r = y _c + s),
w _r denotes the weight of the reference pixel,
index p varies from

before

in the direction of X,
index s varies from

before

in the direction of Y
the normalization factor is calculated as

. 21. The method according to claim 20, characterized in that the disparity filtering is performed in stages, and at the first stage, line-by-line image processing is performed, and at the second stage, the processing is performed in columns, namely

where d _rowk (x _c , y _c ) is the result of a row- _wise filtering procedure;
d _k (x _s , y _s ) - the final result of filtering by columns,
the normalization factor for the line filter is calculated as

and the normalizing factor for the column filter is calculated as

22. The method according to any one of paragraphs.20 and 21, characterized in that the weight of the disparity refinement filter is calculated as follows

where C () denotes a function used to compare the environment of a pixel;
σ _r is the penalty weight parameter for the reference pixel in the reference image;
σ _t is the penalty weight parameter for the combined pixel in the combined image;
(x _r , y _r ) - coordinates of the reference pixel;
(x _t , y _t ) - coordinates of the combined pixel. 23. The method according to item 22, wherein the function used to compare the environment of the pixel is defined as

where I _c (x _c , y _c ) denotes the intensity of the current pixel with coordinates x _c and y _c ;
I _r (x _r , y _r ) denotes the intensity of the reference pixel with coordinates x _r and y _r ;
index i varies from

before

in the direction of X;
index j varies from

before

in the direction of Y;
σ _n is a Gaussian parameter for controlling the weight of the pixel in accordance with its position relative to the position of the current pixel, and the pixels distant from the central pixel are assigned lower weights than the pixels closer to the central pixel, which are assigned higher weight values, while normalizing the multiplier is calculated as

24. The method according to clause 15, wherein the evaluation of the convergence criterion is performed according to the formula

where d _k and d _k-1 denote disparity estimates at the k-th and (k-1) -th iterations of the algorithm;
T _dec1 is the threshold value for the convergence of the disparity assessment. 25. The method according to clause 15, wherein the convergence criterion is evaluated according to the formula k≥T _dec2 , where k is the sequence number of the current iteration of the disparity assessment; T _dec2 is the threshold value for the convergence of the disparity assessment. 26. The method according to clause 15, wherein the disparity processing post-processing is performed using a median filter. 27. A method of generating virtual images from a color image and the corresponding disparity / depth map with known camera parameters, consisting of the following operations:
form models of virtual cameras;
determine the function of forming virtual images;
evaluate the parameters of the function of forming virtual images;
form virtual images in accordance with the function of forming virtual images;
eliminate existing occlusions on the generated virtual images. 28. The method according to item 27, wherein the virtual cameras are described in terms of matrices of real cameras, and the simulation of virtual cameras is performed according to the formula G = P · V, where G is the full projection matrix for the virtual camera; P is the projection matrix of the virtual camera, the matrix P describing how the three-dimensional point is displayed on the plane of the virtual image is derived from the matrix K of the real camera; V is a virtual camera matrix, and the matrix V, which determines the position and orientation of the virtual camera in three-dimensional space relative to the reference point of the world coordinate system, is determined based on the matrix [R | t] of the real camera. 29. The method according to p, characterized in that the projection matrix P of the virtual camera is defined as

where N and F are the foreground and background of the cone of view of the virtual camera;
L and R are the coordinates of the left and right boundaries of the plane of the virtual image;
T and B are the coordinates of the upper and lower boundaries of the plane of the virtual image. 30. The method according to clause 29, wherein the parameters L, R, B and T of the matrix P are determined in accordance with the parameters of the internal matrix K of the real camera as follows

where W is the image width;
- image height;
(x ₀ , y ₀ ) - coordinates of the center of the image on the image plane;
(f _x , f _y ) is the focal length modeling a photosensitive element of a rectangular shape. 31. The method according to p. 28, characterized in that the matrix V of the virtual camera is determined in accordance with the matrix [R | t] of the real camera as

where the submatrix R and the vector t are taken from the matrix [R | t] of the real camera, which describes the orientation and position of the camera relative to the reference point of the world coordinate system. 32. The method according to item 27, wherein the function of forming virtual images is represented as Y = A + zBx,
where the vector A (A = G · C) is defined as the scalar product of the projection matrix G and the coordinate vector C of the center of the camera;
the matrix B (B = G · M ⁺ ) is defined as the scalar product of the matrix G and the pseudoinverse projection matrix M ^{+ of the} real camera;
x is a uniform vector representing the coordinates of the pixel in the image of the real camera, x = (x, y, 1);
z is the scalar corresponding to the depth value obtained from the disparity / depth map with the x coordinate;
Y is a uniform vector representing the coordinate of the pixel in the virtual image, Y = (x, y, z, 1), where the third element corresponds to the depth of the pixel. 33. The method according to p, characterized in that they evaluate the parameters for the function of forming virtual images in the form of a preliminary calculation of the matrices A and B for each virtual camera. 34. The method according to item 27, wherein the elimination of disclusion on the generated virtual images is carried out in the form of occlusion filtering, while the filter for the pixel I (x, y) of the virtual image in a rectangular window with a size K in the horizontal direction and with size L vertically represent as

where I (x + i, y + j) is the original image pixel;
I ^∈ (x, y) - filtered pixel;
W (x + i, y + j) - pixel weight I (x + i, y + j). 35. The method according to clause 34, wherein the weight of the filter is calculated as

where ΔC _pq is the distance between the points p (x, y) and q (x + i, y + j) in the color space;
ΔJ _pq is the distance between the points p and q in the image area;
f (ΔC _pq , ΔJ _pq ) is the function of calculating the pixel weight of the image. 36. The method according to clause 35, wherein the function of calculating the pixel weight of the image f (ΔC _pq , ΔJ _pq ) is defined as

,
where c ₁ and c ₂ are some predefined values associated with the effect of smoothing and improvement. 37. The method according to clause 35, wherein the distance between points in the color space ΔC _{pq is} calculated as the Euclidean distance. 38. The method according to clause 35, wherein the distance between the pixels in the image area ΔJ _{pq is} calculated as the Euclidean distance. 39. The method according to clause 34, wherein the filter for eliminating disclusion is calculated separately as

where I _row (x, y) represents the result of filtering performed on image lines;
I ^∈ (x, y) the final filtering result performed on the image columns. 40. The method according to clause 34, wherein the filter for eliminating occlusion is applied k times. 41. The method according to clause 34, wherein the size of the filter core in the horizontal direction K and the size of the filter core in the vertical direction L are determined using the function DKS (k), which determines the radius of the filter core in accordance with the iteration number k. 42. The method according to paragraph 41, wherein the function DKS (k) is defined as DKS (k) = a _d · k + b _d , where k is the iteration sequence number; a _d , b _d - linear coefficients.

Images