CN119906810B - A real-time hole filling method for high-resolution naked-eye 3D video generation - Google Patents

Abstract

The invention discloses a real-time hole filling method in high-resolution naked eye 3D video generation, and aims to solve the problem of holes when a plurality of virtual viewpoints are mapped in real time by utilizing image information of a plurality of reference viewpoints. The method fills holes mainly through voting fusion technology, and preprocessing is carried out before fusion by adopting the following steps of improving the mapping address error caused by calculation, carrying out single-row filtering on the mapping result, carrying out neighbor interpolation and the like, and filling the holes in the single mapping result one by one. And then, adopting a voting fusion method to fuse complementary mapping results of each virtual viewpoint so as to effectively fill holes of each virtual viewpoint. And after fusion, filling the hole of the current virtual viewpoint by using the pixel data of the adjacent virtual viewpoint in the process of synthesizing the video source according to the naked eye 3D screen parameters. According to the invention, through three filling steps, the cavity filling efficiency is effectively improved, and high-quality naked eye 3D video is output.

Description

Real-time cavity filling method in high-resolution naked eye 3D video generation

Technical Field

The invention relates to a real-time cavity filling method in high-resolution naked eye 3D video generation, and belongs to the technical field of three-dimensional reconstruction.

Background

In the three-dimensional scene reconstruction process, the difficulty and the computational complexity of information acquisition can be remarkably reduced by utilizing a virtual viewpoint generation algorithm. By inputting a small number of reference viewpoint images and combining the depth information as guidance, a virtual view of an arbitrary viewpoint can be generated within a certain range. After the generated virtual viewpoint views are synthesized, the effect of generating corresponding views according to the viewpoint changes of observers is realized, and therefore the three-dimensional scene is successfully reconstructed.

According to the method, corresponding views are generated through virtual views, holes tend to occur in the synthesis process, with the increase of the angle of view, a large number of holes exist in the generated virtual views due to the fact that the information is lost and calculation errors, the holes can be divided into holes outside the view and shielding holes, wherein the holes outside the view are generally distributed at the left side or right side boundary of the generated virtual views, the holes occur in the virtual views mainly because the imaging range of the virtual views exceeds the imaging range of an input reference view, the reference view cannot provide enough boundary information to fill the boundary area additionally seen in the virtual views, the shielding holes are black holes at the boundary of the foreground and the background, the large-scale loss of the foreground structure is also shown, the loss is shown as small cracks at the position where depth changes relatively gradually, the serious tear on the foreground structure is shown, the holes occur in the virtual views mainly because the shielding relation of objects changes in the process of generating the virtual views, some objects or structures blocked in the input view are exposed in the generated virtual views, but the input view cannot provide effective information to fill the boundary area which is exposed, and thus the holes are exposed.

In order to effectively fill a large number of holes generated in the virtual viewpoint generation process, a real-time hole filling method in high-resolution naked eye 3D video generation is provided. According to the method, two view point images with fixed included angles are taken as input, virtual view points corresponding to the same angle are calculated, and the two complementary mapping virtual view point views are fused. And before and after fusion, simple cavity filling is carried out, and finally, video conforming to naked eye 3D screen parameters is output.

Disclosure of Invention

The invention aims to provide a real-time hole filling method in high-resolution naked eye 3D video generation, and in the process of inputting RGB images of a plurality of reference viewpoints and corresponding depth images thereof to generate a naked eye 3D video source, the method aims to effectively reduce holes outside the vision field and shielding holes in the video source by filling holes before virtual viewpoint fusion, filling holes after fusion and filling holes after fusion, so that the display quality of the naked eye 3D video source is improved.

In order to achieve the above purpose, the technical scheme of the invention is as follows, and the real-time cavity filling method in high-resolution naked eye 3D video generation is characterized by comprising the following steps:

Step 1) filling a cavity before fusion, namely inputting orthogonal projection RGB images of a plurality of reference viewpoints and corresponding depth images, mapping the plurality of virtual viewpoints according to the depth information of the depth images of the reference viewpoints, and generating virtual views of different viewpoint positions. Filling holes in the virtual view by adopting a preprocessing strategy, wherein the method comprises the steps of improving mapping errors, performing single-row filtering and then performing neighbor interpolation and the like so as to pre-fill the holes;

Step 2) fusing filling holes, namely fusing the hole filling results of two complementary virtual views by adopting a voting fusion method for each virtual view to further fill the hole area in the virtual view;

and 3) filling the cavity after fusion, namely selecting pixels from different virtual viewpoints to synthesize according to parameters of the naked eye 3D screen in the naked eye 3D video generation process. When the virtual view point to which the current pixel belongs has a hole, the pixels in other virtual view points are preferentially used for filling.

As an improvement of the present invention, the reference viewpoint image input in the step 1) is obtained by a surrounding camera array, each camera is distributed and arranged along an arc, the number of reference viewpoints is greater than or equal to 2, the left and right reference views are used for determining the viewing angle of the naked eye 3D video, and the generated multiple virtual viewpoints are located in the viewing angle range covered by the left and right reference views.

As an improvement of the present invention, in the step 1), the process of mapping the plurality of virtual views according to the depth information of each reference view depth map is performed in a row unit, and the hole filling process is also performed in a row unit.

As an improvement of the invention, the virtual viewpoint mapping process in the step 1) can generate a hole outside the visual field and an occlusion hole, and the occlusion hole is mainly filled before fusion. The specific filling method comprises the steps of calculating the mapping address of the virtual viewpoint according to the depth information of the reference viewpoint and the pixel position, eliminating the cavity caused by calculation errors by rounding the mapping address up and down, or removing discrete points in the virtual viewpoint by single-line filtering, and further filling by using neighbor interpolation on the basis.

As an improvement of the present invention, in the step 2), each virtual viewpoint contains two mapping results from mapping data of reference viewpoints on the left and right sides of the virtual viewpoint position, respectively.

As an improvement of the present invention, the fusion filling mechanism in the step 2) can effectively fill the cavity. According to the method, a voting fusion mode is adopted, and voting fusion is carried out based on the cavity flag bit and the mapping angle on the premise that effective information in each mapping result is extracted. Taking the example of generating a left virtual viewpoint, the fusion process can be expressed as the following formula:

Where θ is the optical axis angle between the generated virtual view and the middle reference view, θ _max is the optical axis angle between the left reference view and the middle reference view, and i represents the position of the current pixel in the virtual view row. P _V ⁱ denotes a fusion result of the i-th pixel to generate the virtual view, P _M ⁱ and P _L ⁱ denote mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint, respectively, and F _M ⁱ and F _L ⁱ denote hole flag bits of the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint, respectively. When a certain pixel contains effective information in the mapping results of a plurality of reference viewpoints, auxiliary modes such as deep arbitration and the like can be introduced based on the method so as to further optimize the voting fusion process.

As an improvement of the invention, the step 3) adopts a single-line filtering method to fill the holes in the fused virtual view. Specifically, a filtering window with the size of 1×4 is selected, and the filtering processing is synchronously performed on the hole zone bit and the pixel offset coordinate of the current mapping row, so as to ensure the effective filling of the hole area and the smooth transition of the image.

As an improvement of the present invention, the step 3) is directed to three-dimensional display devices with different parameters and different types, and the coding and reorganizing process can be flexibly adjusted and scheduled according to the viewpoint arrangement rules of each device, so as to adapt to different display requirements and optimize the video output effect.

As an improvement of the present invention, the step 3) selects an adjacent virtual view or a reference view to fill the hole of the current virtual view. On the basis of considering the pixel height consistency of the adjacent view points, other virtual view points with a closer distance from the current virtual view point are preferably selected.

Compared with the prior art, the invention has the following advantages:

1. The scheme provides a hole filling method based on row processing. According to the method, the whole frame of image data is not required to be stored, and the real-time calculation can be completed only by storing single-row data, so that the cavity filling efficiency is improved. The method can meet the real-time hole filling requirement of naked eye 3D video with 8K/30p even higher resolution and frame rate.

2. According to the scheme, the holes are filled orderly through three steps, and the holes of different types and the causes thereof are processed respectively. Before fusion, part of small holes caused by small-range shielding are filled rapidly to improve the integrity of an initial picture, holes generated outside a visual field and larger holes caused by large-area shielding are filled in a fusion stage, image consistency and quality in a fusion process are ensured, after fusion, the left-over holes still existing in the current viewpoint are filled complementarily by using effective pixels of other viewpoints, and especially, the holes which cannot be filled in a fusion mapping result are filled, so that the integrity of an image is further improved. The staged ordered processing mode effectively solves the filling problem of different cavities, and ensures the high-quality presentation of the final image.

3. The single-line filtering method provided by the scheme only needs to store four continuous pixel data, and compared with the traditional filtering method, the single-line filtering method has the advantage that the hardware storage space is remarkably saved. Taking 3×3 median filtering as an example, conventional methods for outputting one filtered data require storing at least two lines of pixel data, resulting in a significant increase in hardware storage pressure. The scheme effectively reduces the occupation of hardware resources by optimizing the storage requirement, and provides a better solution for realizing efficient real-time processing.

Drawings

FIG. 1 is a schematic flow chart of an algorithm of the present invention;

fig. 2 is a schematic diagram showing the arrangement of left, middle and right reference views and virtual views in embodiment 1;

FIG. 3 is a schematic diagram of eliminating voids caused by calculation errors in embodiment 1;

FIG. 4 is a diagram of filling holes according to mapping addresses in embodiment 1;

FIG. 5 is a schematic illustration of single-line filtering in example 1;

fig. 6 is a virtual viewpoint fusion diagram in embodiment 1;

FIG. 7 is a schematic diagram of the post-fusion filled voids of example 1.

Detailed Description

The application is further illustrated below in conjunction with specific embodiments, it being understood that these embodiments are meant to be illustrative of the application and not limiting of the scope of the application, and that various equivalent modifications to the application will fall within the scope of the application as defined in the appended claims after reading the application.

Embodiment 1 referring to fig. 1 to 7, a real-time hole filling method in high resolution naked eye 3D video generation, when an input reference viewpoint is three viewpoints of left, middle and right, the method comprises the following steps:

Step 1) inputting orthogonal projection RGB images of the left, middle and right three reference viewpoints and corresponding depth images. These reference viewpoint images are acquired by a surrounding camera array, wherein the left and right reference viewpoints determine the field angle of the naked eye 3D video. And mapping the plurality of virtual viewpoints according to the depth information of each reference viewpoint. The arrangement of the reference view and the virtual view is shown in fig. 2, wherein the left virtual view is mapped by the left reference view and the middle reference view, and the right virtual view is mapped by the right reference view and the middle reference view. The field angle of the naked eye 3D video is 2 theta _max.

The mapping mode of the virtual view is to traverse the row pixels of the reference viewpoint, and the mapping position of each pixel in the virtual view is calculated according to the position of the pixel and the corresponding depth information. Since the calculation result may be decimal, by rounding up and down the calculation result, the hole caused by the calculation error can be effectively filled. A specific mapping diagram is shown in fig. 3.

In the mapping of row pixels of a reference view to row pixels of a virtual view, it may occur that pixels of adjacent reference views are mapped to non-adjacent virtual view pixels, as shown in fig. 4. If the pixels of the reference view are not edge pixels, two non-adjacent virtual view pixels may be used to fill the hole in between.

After the virtual view mapping is completed, scattered points affecting the visual effect may appear in the image. Each virtual view pixel has a hole flag bit, and in the subsequent fusion process, the hole flag bits directly determine the validity of the current virtual view pixel. By processing the hole flag bit by using a single-line filtering method, storage and calculation resources can be effectively saved. The specific process is shown in fig. 5. In the processing process, a 1*4 filtering window is selected, a ③ part is taken as a filtering center, the mapping coordinates of each point of the filtering window are assumed to be l1, l2, l3 and l4 respectively, the cavity zone bits are f1, f2, f3 and f4 respectively, the difference threshold value of the mapping coordinates is T (T > 0), m1, m2 and m3 are respectively the comparison results of the absolute values of the difference values of the mapping coordinates of l3 and other three pixels and T, if |l3-l1| is less than or equal to T, m1 is taken as 1, otherwise, the processes of 0, m2 and m3 are taken and the like.

When the window ③ satisfies the following equation:

and judging ③ as a scattered point in the non-hollow area, and updating the coordinates of the scattered point by using the adjacent value in the window.

When the following formula is satisfied at ③ in the window:

the position ③ is a scattered point in the hole area, and although the hole flag bit is 1, the position is actually an invalid value, the hole flag bit needs to be corrected, and the coordinate value does not need to be updated.

Step 2) the same virtual view corresponds to two virtual views, wherein the virtual view of the left virtual view is mapped by the left reference view and the middle reference view, the virtual view of the right virtual view is mapped by the right reference view and the middle reference view, and the two virtual views of the same view are fused according to a preset fusion rule so as to fill the holes in each virtual view. Taking a left virtual viewpoint as an example, a specific fusion block diagram is shown in fig. 6, where V _L2M is a virtual view obtained by mapping a left reference view to the middle, V _M2L is a virtual view obtained by mapping a middle reference view to the left, and V _L is a result obtained by fusing two virtual views.

The fusion filling hole adopts a voting fusion mode, and under the premise of ensuring that effective information in each mapping result can be extracted, voting fusion is carried out based on a hole zone bit and a mapping angle, and a left virtual viewpoint is generated as an example, and the fusion process can be represented as the following formula:

Where θ is the optical axis angle between the generated virtual view and the middle reference view, θ _max is the optical axis angle between the left reference view and the middle reference view, and i represents the position of the current pixel in the virtual view row. P _V ⁱ denotes a fusion result of the i-th pixel to generate the virtual view, P _M ⁱ and P _L ⁱ denote mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint, respectively, and F _M ⁱ and F _L ⁱ denote hole flag bits of the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint, respectively. When a certain pixel contains effective information in the mapping results of a plurality of reference viewpoints, auxiliary modes such as deep arbitration and the like can be introduced based on the method so as to further optimize the voting fusion process.

And 3) when rearranging the fused virtual viewpoints according to the parameters of the naked eye 3D screen, if the virtual viewpoint to which the current pixel belongs is a hole, selecting the pixels at the same positions in other viewpoints for filling. The filling process is shown in fig. 7, where V1-Vn are each a row of pixels of the fused virtual viewpoint, and Pixel is a row of pixels in the rearranged image. In the filling process, the pixels of adjacent virtual views or left, middle and right reference views are preferentially selected to fill the holes of the current virtual view. Considering the high consistency of the pixels of the adjacent viewpoints, the pixels of other virtual viewpoints close to the current virtual viewpoint are preferably selected as much as possible for filling.

The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.

Claims (7)

1. A real-time hole filling method for generating high-resolution naked-eye 3D video, comprising the following steps: Step 1) Fill holes before fusion: Multiple orthogonally projected RGB images of reference viewpoints and their corresponding depth images are input. Multiple virtual viewpoints are mapped based on the depth information of the depth map of each reference viewpoint to obtain mapping coordinates and hole flags. Based on the coordinates and flags, virtual views of different viewpoint positions are generated. Holes in the virtual views are filled using a preprocessing strategy, including improving the mapping error caused by calculation and applying a nearest neighbor interpolation method after single-line filtering of the mapping results to pre-fill the holes. Step 2) Hole filling by fusion: For each virtual viewpoint, which consists of two complementary virtual views, the hole filling results of these two virtual views are fused using a voting fusion method to further fill the hole area in the virtual viewpoint; Step 3) Filling holes after fusion: During the naked-eye 3D video generation process, pixels are selected from different virtual viewpoints for synthesis based on the parameters of the naked-eye 3D screen. If there are holes in the virtual viewpoint to which the current pixel belongs, pixels from other virtual viewpoints are used to fill them, with pixels from adjacent virtual viewpoints being given priority. In step 1), the virtual viewpoint mapping process generates out-of-view holes and occlusion holes. Before fusion, the occlusion holes are primarily filled. The specific filling method includes: first, calculating the mapping address of the virtual viewpoint based on the depth information and pixel position of the reference viewpoint, and then rounding the mapping address up or down to eliminate holes caused by calculation errors; second, removing discrete points in the virtual viewpoint through single-line filtering, and then further filling them using nearest neighbor interpolation. In step 2), each virtual viewpoint includes two mapping results, which are respectively derived from the mapping data of the reference viewpoints on the left and right sides of the virtual viewpoint position. A complementary mapping pair is formed based on the mapping data on the left and right sides, which can effectively eliminate large-area holes. 2. The real-time hole filling method for high-resolution glasses-free 3D video generation according to claim 1, characterized in that: the reference viewpoint images input in step 1) are acquired by a surround camera array, each camera arranged along an arc, the number of reference viewpoints being greater than or equal to two, the left and right reference views being used to determine the field of view of the glasses-free 3D video, and the multiple virtual viewpoints generated being within the field of view angle covered by the left and right reference views. 3. The real-time hole filling method for generating high-resolution naked-eye 3D video according to claim 1, wherein the mapping of multiple virtual viewpoints according to the depth information of each reference viewpoint depth map is performed in real time on a line-by-line basis, and the hole filling process is also performed in real time on a line-by-line basis. 4. The real-time hole filling method for high-resolution naked-eye 3D video generation according to claim 1 is characterized in that the fusion filling mechanism in step 2) can effectively fill the holes. Voting fusion is performed based on the hole flag and mapping angle to generate the left virtual viewpoint, while ensuring the extraction of valid information from each mapping result. The fusion process is expressed as the following formula: (1) in is the optical axis angle between the generated virtual view and the intermediate reference view, is the angle between the left reference view and the middle reference view, represents the row position of the pixel in the current virtual view, Indicates the The fusion result of the virtual view is generated by pixels. and Respectively represent The mapping result of the middle reference view and the left reference view of pixels at the virtual viewpoint, and Respectively represent When a pixel contains valid information in the mapping results of multiple reference viewpoints, auxiliary methods such as depth arbitration can be introduced based on this method to further optimize the voting fusion process. 5. The real-time hole filling method for high-resolution glasses-free 3D video generation according to claim 1, characterized in that: step 3) uses a single-row filtering method to fill the holes in the fused virtual view. Specifically, a filter window of size 1×4 is selected, and the hole flag and pixel offset coordinates of the current mapping row are simultaneously filtered to ensure effective filling of the hole area and smooth image transition. 6. The real-time hole filling method for high-resolution naked-eye 3D video generation according to claim 1, wherein in step 3), the encoding and reorganization process is flexibly adjusted and scheduled according to the viewpoint arrangement rules of each device for different parameters and different types of 3D display devices. 7. The real-time hole filling method for high-resolution naked-eye 3D video generation according to claim 1, characterized in that in step 3) selecting adjacent virtual viewpoints or reference viewpoints to fill the holes of the current virtual viewpoint, taking into account the high consistency of pixels of adjacent viewpoints, preferably other virtual viewpoints that are closer to the current virtual viewpoint.