CN1296873C - Travel-in-picture method based on relative depth computing - Google Patents

Abstract

The invention discloses a picture-in-picture swimming method based on relative depth calculation. The steps are: (1) determine the number of vanishing points in the image; divide the image into foreground and background; for the foreground part, prepare a foreground mask map; for the background part of the image, make a background image and construct a spider grid; (2) according to the number of vanishing points The purpose is to calculate the relative depth and the relative size of the model on the background image, including three sub-steps: determining the viewpoint, obtaining data from the spider grid, and calculating the relative depth and relative size; (3) according to the values of the relative depth and model size, carry out Modeling of the foreground and background; and use the background image and the foreground mask map to texture map the background and foreground model respectively, and position the camera, and render the output image. This method can accurately calculate the relative size and relative position of the 3D model from the 2D image for more regular buildings, and can better solve the fusion problem when the two technologies are switched in the application of the combination of panorama and picture-in-picture .

Description

In-Picture Game Based on Relative Depth Calculation

Technical field

The invention relates to a picture-in-picture swimming method based on relative depth calculation.

Background technique

Image-Based Rendering (IBR) is a virtual reality technology that draws from a series of captured images or unconnected points to obtain a better three-dimensional roaming effect. It has extensive and in-depth applications in film and television special effects production and entertainment advertising. But at the same time, in general, the amount of data required to express a scene by this technology is too large, and sampling is too difficult. Y.Horry proposed Tour Into the Picture (TIP) in 1997. This technology can simply roam a two-dimensional photo, which better solves the sampling problem in IBR technology. The picture-in-picture technology gives a simple 3D model, which is composed of some 3D polygonal frames and billboards, so that a variety of pleasing high-quality 3D animations can be generated. This technology has been widely used in entertainment, cultural relics protection, advertising and other aspects.

Picture-in-stream technology can generally be divided into the following four steps:

1. Image preprocessing. The input image is distinguished from the foreground and the foreground, and the foreground mask and background image are manually made. The so-called foreground mask image is a two-dimensional grayscale image formed by cutting out the foreground part of the image, the foreground part is set to white, and the other parts are black. The background picture is the background after the foreground is removed, and the foreground hollow part is manually filled with patterns and colors that are coordinated with the surrounding background. This is the preparation to start the TIP making process.

2. Build a spider grid. Build a spider grid on the background image, which can provide basic geometric information for modeling. The spider grid consists of a vani shing point, an inner window, an outer window, and rays emitted from the vani shing point. Because the perspective phenomenon is that the near side is larger than the far side, the straight lines parallel to each other in the 3D scene are not necessarily parallel in the 2D image, and finally converge into a point, which is the so-called vanishing point. The inner window is the back rectangle that the viewpoint cannot pass through, and the outer window is the border of the image. The rays starting from the vanishing point and passing through the four vertices of the inner window together with the inner window divide the original image into five parts—the inner window and the four polygons up, down, left, and right. Correspondingly, they are respectively named rear wall, ceiling, floor, left wall, and right wall.

3. Remodeling. For the background, the five segments into which the spider grid is divided correspond to the five contiguous rectangles in the modeling process. Remodeling is based on three assumptions: 1) the adjacent rectangles among the five rectangles are perpendicular to each other; 2) the rear wall rectangle is parallel to the viewing plane in the 3D environment; 3) the floor rectangle is perpendicular to the 3D environment at the viewing plane. In the reconstruction process, the relevant position information of the spider grid in the image is obtained first, and the three-dimensional position calculation of each key point (vertex of the rectangle in the modeling process) is performed. Among them, information such as ceiling height is an estimated size. Modeling is carried out according to the calculated position information of each key point, and a three-dimensional box model (box model) of the background is obtained. For the foreground, based on the form of the billboard, it is modeled using hierarchical polygons, and the three-dimensional position calculation is similar to that of the background.

4. Texture mapping and rendering. Assume that the textures in the 3D rectangles are all inherited from the corresponding 2D polygons. Any 2D point in the rendered output image can find its corresponding point in the 3D model, which in turn can find its 2D corresponding point in the original image. For the texture mapping of the foreground, in addition to the texture correspondence, different transparency values of the foreground object and the surrounding background must be set according to the information provided by the mask map. Therefore, for each point rendered, the corresponding point can be found in the original image, and the result of texture mapping can be obtained by mapping the color of the point.

The swimming-in-picture technology starts from an image and obtains the roaming effect in the 3D model, giving people a brand-new visual effect of "people swimming in the picture". However, in the application process, especially when the picture-in-picture technology is combined with the panorama technology, whether the reconstructed 3D model can obtain the same visual effect as the original picture without changing the position of the viewpoint in the picture-in-picture technology has raised more questions. High requirements, which are essentially more precise requirements for the position and size of the reconstructed 3D model. In the modeling process of the picture-in-picture game technology, the original height equivalent estimation method is difficult to meet the demand, which directly affects the user's sense of reality in the picture-in-picture game process.

Contents of Invention

The purpose of the present invention is to provide a method for in-picture swimming based on relative depth calculation.

Taking an image as input for roaming, the steps are:

(1) Determine the number of vanishing points in the image; divide the image into the foreground and background; for the foreground part, cut out the foreground from the background, and prepare the foreground mask map; for the background part of the image, fill in the cut out part of the foreground, make a background image, and Construct a spider grid;

(2) Define the image plane as a plane perpendicular to the ground, the original image is located on the image plane, and the bottom edge of the original image is placed on the intersection line between the image plane and the ground plane, and the relative depth is the distance between the image plane and the back wall of the model Depth: Calculate the relative depth of the background image and the relative size of the model according to the number of vanishing points, including the three sub-steps of determining the viewpoint, obtaining data from the spider grid, and calculating the relative depth and the relative size of the model;

(3) Model the foreground and background according to the values of the relative depth and the relative size of the model; use the background image and the foreground mask to map the texture of the background and foreground model respectively, position the camera, and render the output image.

The invention is simple and easy to use, and its beneficial effect is that the more accurate relative position and relative size of the model in the picture can be obtained through scientific calculation, so that the user can get the same visual effect as the original image under the condition that the position and angle of the viewpoint remain unchanged 3D model perspective output image. Especially in the application of the combination of picture-in-game and panorama technology, jump-free switching between panorama and picture-in-game models can be obtained.

The present invention is tested experimentally, an accurate three-dimensional model is constructed, and camera sampling is performed. Using the new picture-in-picture method (one vanishing point image) on the sampled image, the 3D model is reconstructed. In the sampling where the original relative depth is 100.00, Table 1 is the result calculated by the present invention, and the average error is 2.86%:

Table 1 Results and error analysis Experiment 1 Experiment 2 Experiment 3 relative depth 97.33 96.73 97.37 error 2.67% 3.27% 2.63%

In the process of re-rendering the reconstructed 3D model to form an output image, the influence of errors in its relative depth and relative size will be further reduced, so that the same visual effect as the original image can be obtained.

Description of drawings

Fig. 1 is a schematic flow chart of the method of picture-in-picture based on relative depth calculation;

Figures 2a-d are schematic diagrams of spider grids in the midstream method;

Fig. 3 is a schematic diagram of the definition of "relative depth" of the present invention;

Fig. 4 is a schematic diagram of calculating the relative depth and model relative size of a vanishing point image of the present invention;

Fig. 5 is a schematic diagram of the actual modeling details of the left wall of the present invention;

Figures 6a-c are schematic diagrams of output image effects of a vanishing point image of the present invention after computational modeling and rendering;

Fig. 7 is a schematic diagram of calculating the relative depth and model relative size of two vanishing point images of the present invention;

8a-c are output images of the roaming process of two vanishing point images of the present invention.

Detailed ways

The technical scheme adopted by the present invention to solve the technical problem is: define the image plane as a plane perpendicular to the ground. The original image is located on the image plane, and the bottom edge of the original image is placed on the intersection line between the image plane and the ground plane. In this scheme, the original image placed in this way is called the reference image. Introduce the concept of "Relative Depth (RD)", that is, the distance depth between the model and the image plane. In the painting-in-picture technique, "relative depth" is the distance depth between the image plane and the back wall of the model. According to the principle of perspective geometry, on the basis of constructing the spider grid, the relative depth of the model can be calculated from the definition of the reference image. On this basis, the relative dimensions of the model can be further calculated. The so-called relative size means that the calculated size is the size of the model corresponding to the size of the reference image, rather than the original size of the object in the real environment. The calculation of the relative depth determines the position and size of the in-picture game model, and also determines the depth limit of the user's roaming.

Since it is difficult to calculate depth from a single image, this requires some auxiliary conditions or the model itself has certain geometric features (such as vertical or parallel lines and planes, etc.). We propose the present invention based on the following assumptions:

(1) The left and right walls and the back wall of the model are perpendicular to the ground.

(2) The image plane is perpendicular to the ground.

(3) The original image is obtained by horizontal sampling of the camera.

According to different image features, the present invention summarizes the relative depth and model size calculation methods of images with different numbers of vanishing points. The basic steps of relative depth calculation are:

1. Determine the viewpoint. A fixed viewpoint is determined, and switching between the reference image and the in-picture model is performed at this viewpoint. During the roaming process, the generated new image is different from the original image, including field of view size, resolution, and viewing angle. Only at this fixed viewpoint can the new generated image get the same visual effect as the original image. This means that the user must return to this viewpoint position when switching from the in-picture model to the original image.

2. Obtain the data required for relative depth calculation from the spider grid.

3. Calculate the relative depth and relative size according to different numbers of vanishing points.

The steps of the new picture-in-picture method are as follows:

(1) Determine the number of vanishing points in the image; manually divide the foreground and background of the image; for the foreground part, cut out the foreground from the background, and prepare the foreground mask map; for the background part of the image, fill in the cut out part of the foreground to make a background image, And construct the spider grid;

(2) Define the image plane as a plane perpendicular to the ground, the original image is located on the image plane, and the bottom edge of the original image is placed on the intersection line between the image plane and the ground plane, and the relative depth is the distance between the model and the image plane Depth: Calculate the relative depth of the background image and the relative size of the model according to the number of vanishing points, including three sub-steps: determining the viewpoint, obtaining data from the spider grid, and calculating the relative depth and relative size;

(3) Model the foreground and background according to the values of the relative depth and model size; use the background image and the foreground mask to map the texture of the background and foreground model respectively, position the camera, and render the output image.

As shown in Figure 1, first, determine the vanishing point type of the original image to determine different relative depths and model relative size calculation methods; distinguish its foreground and background, and preprocess the foreground and background respectively to form a foreground mask and a background image. Then, on the basis of constructing a spider grid on the background image, the model structure is determined according to different types of vanishing points, and the relative depth and relative size of the model are calculated. Afterwards, separate modeling and texture mapping of the foreground and background are performed based on the calculated data. Finally, position the camera and render the resulting image.

Figure 2(a) shows the components of the spider grid: the vanishing point, the inner window, the outer window, and the rays starting from the vanishing point. Figure 2(b) shows an example of constructing a spider grid on a background image of vanishing points. Figure 2(c) shows that the construction of the spider grid divides the background image into five parts: the back wall, the left and right walls, the ceiling and the ground, which will directly correspond to the sides of the box model in the modeling process. Figure 2(d) illustrates the corresponding relationship between each vertex after modeling and the relevant vertices in the spider grid, and gives the appearance of the box model after modeling.

As shown in Figure 3, "relative depth" is defined as the distance depth between the model and the image plane. In the painting-in-picture technique, "relative depth" is the distance depth between the image plane and the back wall of the model. The image plane is perpendicular to the ground plane, and the bottom edge of the reference image is on the ground plane. The "relative depth" determines the position of the model (that is, the position of the back wall, that is, the depth limit of roaming), which is actually the length of the ground rectangle in the 3D box model.

In the embodiment shown in FIG. 4, the input image is a vanishing point image. In Figure 4, the upper left corner is the background image, the upper right corner is the modeling side view, and the lower left corner is the modeling top view. The position of the viewpoint is on a straight line perpendicular to the image plane and passing through the vanishing point, and the distance f from the viewpoint to the image plane can be set according to the application requirements, so that the position of the viewpoint is determined. The relevant data information required for relative depth calculation can be obtained from the spider grid of the background image: the distance m ₁ from the bottom edge of the inner window to the bottom edge of the outer window, the distance vh from the vanishing point to the bottom edge of the outer window (ie, the height of the viewpoint), The distance m ₂ from the point to the top edge of the inner window, the distance p ₁ from the vanishing point to the left side of the inner window, and the distance p ₂ from the vanishing point to the right side of the inner window. According to the principle of perspective geometry, the relative depth can be calculated $d=\frac{m_1\&Center\; Dot;f}{\mathrm{vh}-m_1},$ model height $h=\mathrm{vh}+\frac{m_2\&Center\; Dot;(f+d)}{f},$ model width $w=w_1+w_2=\frac{(p_1+p_2)\&Center\; Dot;(f+d)}{f}.$ This gives the position and main relative dimensions of the 3D box model. In addition, there are some modeling details that need attention, such as the specific size of each 3D rectangle. As shown in the top view of Figure 4 and Figure 5, taking the left wall as an example, its 3D model is not a d×h rectangle, but the viewing angle of its 3D space must be considered, which should be (dn ₁ )×h rectangle. Similarly, the right wall model should be a (dn ₂ )×h rectangle, and the ceiling model should be a (dn ₃ )×h rectangle. Among them, it can be obtained from Figure 5 that ${\mathrm{no}}_1=\frac{(w_1-p_3)\&Center\; Dot;f}{p_3},{\mathrm{no}}_2=\frac{(w_2-p_4)\&Center\; Dot;f}{p_4},{\mathrm{no}}_3=\frac{(h-\mathrm{height})\&Center\; Dot;f}{(\mathrm{height}-\mathrm{vh})}.$

FIG. 6 is a comparison of output image effects of the embodiment shown in FIG. 4 . Figure 6(a) is the input image of this example, Figure 6(b) is the appearance of the model obtained after TIP modeling the input image, and Figure 6(c) is the enlargement of the texture part in (b). It can be seen that, under the condition that the position and angle of the viewpoint remain unchanged, the output model with the same visual effect as the original input image can be obtained through the calculation and modeling of the present invention.

Fig. 7 shows another embodiment, the input image is an image of two vanishing points. In Figure 7, the upper left corner is the background image, the upper right corner is the modeling side view, and the lower left corner is the modeling top view. According to perspective geometry, the two vanishing points in the image must be on the same horizontal line, which is called the vanishing line. The position of the viewpoint is on a straight line perpendicular to the image plane and passing through the midpoint of the line connecting the two vanishing points. The distance f from the viewpoint to the image plane can be set according to the application requirements, which determines the position of the viewpoint. The relevant data information required for relative depth calculation can be obtained from the spider grid of the background image. Calculated to get: $d_1=\frac{(\mathrm{vh}-m_1)\&Center\; Dot;f}{m_1},d_2=\frac{(\mathrm{vh}-m_2)\·f}{m_2},d_3=\frac{(\mathrm{vh}-m_3)\&Center\; Dot;f}{m_3},$

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; vh</span>}<span\; class="notranslate">\; +</span>\frac{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; .</span>$

FIG. 8 is the output image effect during the roaming process of the embodiment in FIG. 7 after computational modeling. Figure 8(a) constructs a grid for the two vanishing point images and extracts relevant calculation data. Figure 8(b) is the original input image, and Figure 8(c) is the output image at a certain angle during the roaming process, which is a new viewpoint visual effect that cannot be obtained in the original image.

Claims (5)

1. A method for swimming in a picture based on relative depth calculation, characterized in that: roaming is performed with an image as input, and the steps are: (1) Determine the number of vanishing points in the image; divide the image into foreground and background; for the foreground part, cut out the foreground from the background, and prepare a foreground mask map; for the background part of the image, fill in the cut out part of the foreground, and make a background image, And construct the spider grid; (2) Define the image plane as a plane perpendicular to the ground, the original image is located on the image plane, and the bottom edge of the original image is placed on the intersection line between the image plane and the ground plane, and the relative depth is the distance between the image plane and the back wall of the model Depth: Calculate the relative depth of the background image and the relative size of the model according to the number of vanishing points, including the three sub-steps of determining the viewpoint, obtaining data from the spider grid, and calculating the relative depth and the relative size of the model; (3) Model the foreground and background according to the values of the relative depth and the relative size of the model; use the background image and the foreground mask map to texture map the background and foreground models respectively, position the camera, and render the output image. 2. A method of midstream based on relative depth calculation according to claim 1, characterized in that said viewpoint determination is: the viewpoint is a fixed point for switching between the midstream model and the reference image, and for a vanishing point image, Its viewpoint is located at the vanishing point and is perpendicular to the straight line of the image plane, and the distance f from the viewpoint to the image plane can be determined arbitrarily; The distance f from the viewpoint to the image plane can be determined arbitrarily on the line connecting the midpoint of the vanishing point and perpendicular to the image plane. 3. A mid-picture method based on relative depth calculation according to claim 2, characterized in that the calculation of the relative depth and model relative size of the background image according to the number of vanishing points is: for an image of a vanishing point, Obtain the following data information from the spider grid of the background image: the distance m ₁ from the bottom edge of the inner window to the bottom edge of the outer window, the distance vh from the vanishing point to the bottom edge of the outer window, which is the height of the viewpoint, and the distance from the vanishing point to the top edge of the inner window m ₂ , the distance from the vanishing point to the left of the inner window p ₁ , the distance from the vanishing point to the right of the inner window p ₂ , the distance from the vanishing point to the left of the outer window p ₃ , the distance from the vanishing point to the right of the outer window p ₄ , the height of the outer window, From the sampling point to the left wall of the model is w ₁ , to the right wall of the model is w ₂ , and according to the principle of perspective geometry, the relative depth of the 3D box model and the relative size of the model are calculated as follows: relative depth $d=\frac{m_1\&Center\; Dot;f}{\mathrm{vh}-m_1},$ model height $h=\mathrm{vh}+\frac{m_2\&CenterDot;(f+d)}{f},$ model width $w=w_1+w_2=\frac{(p_1+p_2)\&Center\; Dot;(f+d)}{f}.$ 4. A mid-picture method based on relative depth calculation according to claim 2, characterized in that the calculation of the relative depth and model relative size of the background image according to the number of vanishing points is: for the image of two vanishing points , the following data information is obtained from the spider grid of the background image: the distance from the off-line to the bottom of the inner window m ₁ , the distance from the off-line to the bottom of the left wall m ₂ , the distance from the off-line to the bottom of the right wall m 3 , and the distance from the off-line to the bottom of the right wall m ₃ . The distance m ₄ to the top edge of the outer window, the distance vh from the vanishing line to the bottom edge of the outer window, that is, the height of the viewpoint, the width p ₁ of the inner window, the distance p ₂ from the viewpoint to the left side of the outer window, and the distance p ₃ from the viewpoint to the right side of the outer window, After calculation: the relative depth of the back wall $d_1=\frac{(\mathrm{vh}-m_1)\&Center\; Dot;f}{m_1},$ The relative depth of the outer edge of the left wall $d_2=\frac{(\mathrm{vh}-m_2)\&Center\; Dot;f}{m_2},$ The relative depth of the outer edge of the right wall $d_3=\frac{(\mathrm{vh}-m_3)\&Center\; Dot;f}{m_3},$ rear wall width $w_1=\frac{(d_1+f)\&Center\; Dot;p_1}{f},$ Width of the outer edge of the left wall $w_2=\frac{(d_2+f)\&Center\; Dot;p_2}{f},$ Right wall outer edge width $w_3=\frac{(d_3+f)\&CenterDot;p_3}{f},$ model height $h=\mathrm{vh}+\frac{m_4\&Center\; Dot;(f+d_1)}{f}.$ 5. A mid-picture method based on relative depth calculation according to claim 3, characterized in that said calculation of a vanishing point image to obtain a three-dimensional box model is: for a vanishing point image, after obtaining the relative depth of the box model and After the relative size data of the model, confirm the specific size of each three-dimensional rectangle of the box model. The left wall model is a (dn ₁ )×h rectangle, the right wall model is a (dn ₂ )×h rectangle, and the ceiling model is (dn ₃ ) ×h rectangle, where, ${\mathrm{no}}_1=\frac{(w_1-p_3)\&CenterDot;f}{p_3},$ ${\mathrm{no}}_2=\frac{(w_2-p_4)\&CenterDot;f}{p_4},$ ${\mathrm{no}}_3=\frac{(h-\mathrm{height})\&CenterDot;f}{(\mathrm{height}-\mathrm{vh})}.$

Images