TWI736335B - Depth image based rendering method, electrical device and computer program product - Google Patents

Abstract

A depth image based rendering method includes: obtaining a two-dimensional image and a corresponding depth image, and warping the two-dimensional image according to the depth image to obtain a warped image which includes at least one hole region; filling the hole region from up to down to generate a first filled image; filling the hole region from down to up to generate a second filled image; classifying each pixel of the hole region as a first angle type or a second angle type, filling the pixel by adopting the first filled image if the pixel belongs to the first angle type or by adopting the second filled image if the pixel belongs to the second angle type.

Description

Based on depth image generation method, electronic device and computer program product

This disclosure relates to a depth-based image generation method, which uses a weighted 3D decimal warp calculation and a double-path gradient hole filling method.

Comfortable 3D imaging is already one of the hot trends in science and technology. In recent years, the entertainment industry, medical systems, etc. have all applied 3D imaging in daily life. However, comfortable 3D image display requires multi-view images. Unfortunately, multi-view cameras are expensive and difficult to transmit, resulting in lack of content and degradation of quality. Based on the depth image based rendering (DIBR) technology, a multi-view 3D image can be generated from a single 2D image and a depth map. The existing traditional depth image based rendering technology includes three main processing stages: depth image preprocessing; 3D curvature shift (3D Warping); Hole Filling. Depth map preprocessing is to smooth the depth map to reduce the horizontal depth variation, thereby reducing the voids after 3D warping. The 3D curve shift uses the principle of optical projection. The pixels in the 2D image move linearly and horizontally according to the depth of field value. The pixels with a larger depth of field move more, and vice versa. 3D warping will produce many holes, The above-mentioned hole filling step is to fill the holes generated by 3D warping through methods such as interpolation, extrapolation, and repair. How to improve the above process is a topic of concern to those skilled in the art.

An embodiment of the present invention provides a depth-based image generation method suitable for an electronic device. The depth-based image generation method includes: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and pixelizing the two-dimensional image according to the depth image Warping to obtain a warped image, the warped image including at least one hollow area; filling the hollow area from top to bottom to generate a first filled image; filling the hollow area from bottom to top to generate a second filled image; And each pixel in the cavity area is divided into a first angle category or a second angle category. If it is the first angle category, the first padded image is used, and if it is the second angle category, the second padded image is used to fill the pixels.

In some embodiments, the above step of filling the cavity area from top to bottom to generate the first filled image includes: for the first pixel of the cavity area, calculating a horizontal gradient value according to the upper pixel and the upper left pixel of the first pixel, And calculate a vertical gradient value based on the left pixel and the upper left pixel of the first pixel; if the horizontal gradient value is greater than the vertical gradient value, the upper pixel is used to fill the first pixel; and if the vertical gradient value is greater than the horizontal gradient value, the left Pixels to fill the first pixel.

In some embodiments, the above step of filling the cavity area from bottom to top to generate the second filled image includes: for the first pixel of the cavity area, calculating a horizontal gradient value according to the lower pixel and the lower left pixel of the first pixel, And calculate a vertical gradient value based on the left pixel and the bottom left pixel of the first pixel; if the horizontal gradient value is greater than the vertical gradient value, use the bottom pixel to fill the first pixel; and if the vertical gradient value is greater than the horizontal gradient value, use the left Pixels to fill the first pixel.

From another perspective, the above step of classifying each pixel in the cavity area into the first angle category or the second angle category includes: for each pixel in the cavity area, if the pixel is a hole pixel and the pixel on the left and above If the pixel is a non-hole pixel, classify this pixel as the first angle category; for each pixel in the hole area, if the pixel is a hole pixel and the left pixel belongs to the first angle category, then this pixel is classified as the first angle category ; For each pixel in the hole area, if the pixel is a hole pixel, the lower pixel belongs to the first angle category, and the left pixel is a non-hole pixel, classify this pixel as the first angle category; and classify the remaining hole pixels in the hole area Classified as the second angle category.

In some embodiments, the step of performing pixel warping on the two-dimensional image according to the depth image to obtain the warped image includes: calculating a decimal point position according to the depth value in the depth image, and filling the depth value to the decimal point position To generate the depth image after the curvature shift, and fill the corresponding pixel value in the two-dimensional image into the decimal point position to generate the two-dimensional image after the curvature shift; for an integer point position, calculate the maximum depth value within the vicinity of the integer point position , And delete the depth values in the neighboring range that differ from the maximum depth value by more than a critical value; and weighting the pixel values at the integer point positions according to the pixel values corresponding to the remaining depth values in the neighboring range.

In some embodiments, the above is based on the remaining depth in the adjacent range The step of calculating the pixel value at the position of the integer point corresponding to the pixel value of the value includes: calculating the Gaussian distance between the position of the integer point and the position of the remaining depth value in the adjacent range; and using the Gaussian distance as a weight to add up the remaining depth The pixel value corresponding to the value is used to calculate the pixel value at the integer point position.

In some embodiments, the aforementioned proximity range includes a horizontal direction and a vertical direction.

From another perspective, an embodiment of the present invention provides an electronic device including a memory and a processor. The memory stores a plurality of instructions, and the processor executes these instructions to complete the above-mentioned depth-based image generation method.

From another perspective, an embodiment of the present invention provides a computer program product that is loaded and executed by an electronic device to complete the above-mentioned depth-based image generation method.

Using the above-mentioned depth-based image generation method can reduce holes and also reduce the amount of calculation.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

100: electronic device

110: processor

120: memory

210,230,240,250: 2D image

220: Depth image

310, 320, 330, 401~405: steps

411: Two-dimensional image after warping

412: Depth image after warping

410: Image after curve shift

420: Hollow Area

I(x,y): pixel

D(x,y): depth of field value

(

,y):曲移後深度影像

(

,y ): Depth image after curve shift

(

,y):曲移後二維影像

(

,y ): Two-dimensional image after warping

(x,y):曲移後影像

( x,y ): image after shifting

R _x : Proximity range

(x,y):最大深度值

( x,y ): maximum depth value

,

):集合 N (

,

):gather

610,620,630: steps

611: First Filled Image

621: second post-fill image

631: image

[Fig. 1] is a schematic diagram showing an electronic device according to an embodiment.

[Fig. 2] is a schematic diagram showing a depth-based image generation method according to an embodiment.

[Fig. 3] is a flowchart of a depth-based image generation method according to an embodiment Schematic.

[FIG. 4A] and [FIG. 4B] are schematic diagrams illustrating the flow of weighted 3D decimal warp calculation according to an embodiment.

[FIG. 5A and FIG. 5B] are partial schematic diagrams showing a two-dimensional image after a curve shift and a depth image after a curve shift according to an embodiment.

[Fig. 6] is a schematic diagram illustrating a method for filling a double-path gradient hole according to an embodiment.

Regarding the “first”, “second”, etc. used in this text, it does not particularly mean the order or sequence, but only to distinguish elements or operations described in the same technical terms.

FIG. 1 is a schematic diagram illustrating an electronic device according to an embodiment. 1, the electronic device 100 may be a smart phone, a tablet computer, a personal computer, a notebook computer, a server, an industrial computer, or various electronic devices with computing capabilities, etc. The present invention is not limited thereto. The electronic device 100 includes a processor 110 and a memory 120, where the processor 110 can be a central processing unit, a microprocessor, a microcontroller, an image processing chip, a special application integrated circuit, etc., and the memory 120 can be random access Memory, read-only memory, flash memory, floppy disk, hard disk, optical disk, flash drive, tape, or a database accessible through the Internet, which stores multiple commands, which the processor 110 will execute These instructions complete a depth-based image generation method, which will be described in detail below.

Fig. 2 illustrates a depth-based image generation method according to an embodiment Schematic diagram. Please refer to FIG. 2, the depth-based image generation method is used to generate two-dimensional images with different viewing angles according to a two-dimensional image 210 and a depth image 220 corresponding to the two-dimensional image 210. For example, the viewing angle of the two-dimensional image 210 is set to viewing angle 0, and the depth image 220 includes the depth value measured from viewing angle 0. In this embodiment, the closer to the lens, the greater the depth of field value, but the invention is not limited thereto. According to the two-dimensional image 210 and the depth image, a two-dimensional image 230 of view angle 1, a two-dimensional image 240 of view angle 2, and a two-dimensional image 250 of view angle -1 can be generated, and so on. Here, viewing angle 1 to viewing angle 3 are used to represent the right viewing angle, and viewing angle -1 to viewing angle-3 are used to represent the left viewing angle, but the present invention is not limited thereto. For example, if there is a foreground object in the two-dimensional image 210, the foreground object will shift to the left when viewed from the right perspective, and the foreground object will shift to the right when viewed from the left perspective. The closer the current object is to the lens The greater the displacement will be.

Here, the two-dimensional image 210 and the depth image 220 may be obtained through an image sensor and a depth sensor, respectively, or may be obtained from an existing database or other devices. The aforementioned image sensor may include a Charge-coupled Device (CCD) sensor, a Complementary Metal-Oxide Semiconductor (Complementary Metal-Oxide Semiconductor) sensor, or other suitable photosensitive elements. The above-mentioned depth sensor may include dual cameras, artificial intelligence depth generation network, structured light sensing device, or any device that can sense the depth of the scene.

FIG. 3 is a schematic flowchart of a depth-based image generation method according to an embodiment. Please refer to FIG. 3, in step 310, the depth image 220 is preprocessed, for example, smoothing is performed to reduce the change of the depth value. Any pre-processing can be used, and the present invention does not limit the content of step 310. In step 320, pixel curvature is performed on the two-dimensional image 210 according to the preprocessed depth image 220. In particular, this embodiment proposes a weighted 3D fractional curvature calculation. After performing step 320, a warped image is obtained. The warped image includes one or more void regions. These void regions are filled in step 330. In this embodiment, a dual-path gradient hole filling method is used to fill the void regions. Later, a virtual perspective image may be generated, such as a two-dimensional image 230. Hereinafter, an embodiment will be given to describe step 320 and step 330 in detail.

4A and 4B are schematic diagrams illustrating the flow of weighted 3D decimal warp calculation according to an embodiment. Please refer to FIG. 4A. Here, the pixels in the two-dimensional image 210 are represented as I(x, y), the depth value of the preprocessed depth image 220 is represented as D(x, y), and x and y respectively represent the X coordinates And Y coordinate. The above-mentioned step 320 includes steps 401 to 405. In step 401, a decimal point position is calculated according to each depth value in the depth image 220. Specifically, a disparity displacement value can be calculated according to the depth value, as shown in the following mathematical formula 1.

(x,y)為視差位移值。K表示小數點位置的小數點精確度，K可為2、4、8或任意正整數。m表示視角，例如為圖2所示的-3~-1與1~3的其中之一。P _c 為景深視差參數(depth disparity ratio)，可依照相機及立體顯示器的參數來決定。

表示捨去小數的整數。上述的數學 式1是將傳統的視差位移值乘上K倍，經過四捨五入的運算以後再除以K倍，但值得注意的是視差位移值

(x,y)可以是整數也可以有小數，若以K=4為例，小數值可為.00、.25、.50、.75等四個值。接下來，將x座標加上視差位移值

(x,y)便可以得到一個小數點位置。 in

( x,y ) is the parallax displacement value. K represents the decimal point accuracy of the decimal point position, K can be 2, 4, 8 or any positive integer. m represents the viewing angle, for example, one of -3 to -1 and 1 to 3 shown in FIG. 2. P _c is the depth disparity ratio, which can be determined according to the parameters of the camera and the stereo display.

Represents an integer with decimals rounded off. The above mathematical formula 1 is to multiply the traditional parallax displacement value by K times, and then divide by K times after the rounding operation, but it is worth noting that the parallax displacement value

( x, y ) can be an integer or a decimal. If K=4 is taken as an example, the decimal value can be four values such as .00, .25, .50, and .75. Next, add the parallax displacement value to the x coordinate

( x,y ) can get a decimal point position.

(

,y)即是曲移後深度影像，x+

(x,y)為小數點位置。值得注意的是Y座標不需要位移。 In step 402, the original depth of field value D ( x, y ) is moved to the above-mentioned decimal point position to generate a depth image after curvature shift. This step can be expressed as the following mathematical formula 2.

(

,y ) is the depth image after warping, x +

( x,y ) is the position of the decimal point. It is worth noting that the Y coordinate does not need to be displaced.

(

,y)即是曲移後二維影像。 In step 403, the corresponding pixel value in the two-dimensional image 210 is filled in the decimal point position to generate the two-dimensional image after the curvature shift. This step can be expressed as the following mathematical formula 3.

(

,y ) is the two-dimensional image after the warp shift.

(x,y)，則可以保留較大的深度值，捨棄較小的深度值。 If the position of the decimal point overlaps, that is, different x corresponds to the same x +

( x,y ), the larger depth value can be kept, and the smaller depth value can be discarded.

(x,y)可以包括K個小數，這等同於曲移後二維影像

(

,y)的寬度會是二維影像I(x,y)的K倍，如曲移後二維影像411所示。相同的，曲移後深度影像

(

,y)的寬度會是深度影像D(x,y)的K倍，如曲移後深度影像412所示。 Since the parallax displacement value in Mathematical formula 1

( x, y ) can include K decimals, which is equivalent to a two-dimensional image after warping

(

, y ) will be K times the width of the two-dimensional image I ( x, y ), as shown in the two-dimensional image 411 after warping. Same, depth image after warp shift

(

The width of , y ) will be K times the depth image D ( x, y ), as shown in the depth image 412 after the warp shift.

(

,y)與曲移後深度影像

(

,y)，在此繪示了局部(

=98~101)的像素值與深度值，值得注意的是較粗的標線表示整數點位置，如

=98、

=99、

=100、

=101等，而其他的標線表示小數點位置，如.25、.50、.75等。在此例子中，景深值“30”被填入

=99.50的位置，景深值“121”被填入

=99.75的位置，而景深值“120”被填入

=100.25。每個景深值都對應至一個像素值，景深值“30”是對應至像素值“68”，景深值“121”是對應至像素值“215”，景深值“120”是對應至像素值“212”。接下來要計算每個整數點位置上的像素值。 5A and 5B are partial schematic diagrams illustrating a two-dimensional image after a curvature shift and a depth image after a curvature shift according to an embodiment. Please refer to Figure 5A, since there is no displacement on the Y coordinate, the X coordinate is used to represent the two-dimensional image after the shift

(

,y ) and the depth image after shifting

(

,y ), here is a part (

=98~101) pixel value and depth value, it is worth noting that the thicker marking line indicates the position of the integer point, such as

=98,

=99、

=100,

=101, etc., while other marking lines indicate the position of the decimal point, such as .25, .50, .75, etc. In this example, the depth of field value "30" is filled in

=99.50 position, the depth of field value "121" is filled in

=99.75 position, and the depth of field value "120" is filled in

=100.25. Each depth value corresponds to a pixel value. The depth value "30" corresponds to the pixel value "68", the depth value "121" corresponds to the pixel value "215", and the depth value "120" corresponds to the pixel value "212". The next step is to calculate the pixel value at each integer point position.

4A and 5A, in step 404, for each integer point position in the depth image after the warping, the maximum depth value within the vicinity of the integer point position is calculated, and the difference between the adjacent range and the maximum depth value is deleted Depth value exceeding a critical value. If the position of the integer point is x, the adjacent range can be expressed as the following mathematical formula 4.

所形成的集合。在此實施例中K=4，因此鄰近範圍R _x 可以設定為左方2個小數點位置到右方2個小數點位置，以整數點位置“100”為例，鄰近範圍R _x 表示從

=99.50至

=100.50的範圍。 接下來可以根據鄰近範圍R _x 的像素值來決定整數點位置

=100上的像素值，如果有多個像素被位移至此鄰近範圍R _x ，則保留與最大景深值相差一臨界值內的景深值，因為較大的景深值在前景，會遮蓋位於背景的像素值。具體來說，鄰近範圍R _x 內的最大深度值為“121”，表示為

(x,y)，如以下數學式5所示。鄰近範圍R _x 內剩餘的景深值形成一個前景景深值集合，表示為以下數學式6。 R _x represents the adjacent range, and also represents all within this range

The resulting collection. In this embodiment, K=4, so the adjacent range R _x can be set from 2 decimal point positions on the left to 2 decimal point positions on the right. Taking the integer point position "100" as an example, the adjacent range R _x means from

=99.50 to

= 100.50 range. Next, the integer point position can be determined according to the pixel value of the neighboring range R _x

=100 pixel value, if multiple pixels are shifted to this adjacent range R _x , the depth value within a critical value from the maximum depth value is retained, because the larger depth value is in the foreground and will cover the pixels in the background value. Specifically, the maximum depth value in the adjacent range R _x is "121", which is expressed as

( x, y ), as shown in Math 5 below. The remaining depth values in the adjacent range R _x form a foreground depth value set, which is expressed as the following mathematical formula 6.

S(x,y) is the foreground depth value set. In this example, "121" and "120" are included, and the depth value "30" is deleted. ε _D is a critical value and can be set to any suitable value.

(

,y)中對應的像素值也會被刪除，剩餘的景深值與像素值如圖5B所示，剩餘的像素值可以表示為以下數學式7。在此例子中剩下的像素值為“215”與“212”。 In addition, the two-dimensional image after the curve shift

(

The corresponding pixel value in ,y ) will also be deleted. The remaining depth value and pixel value are shown in FIG. 5B, and the remaining pixel value can be expressed as the following mathematical formula 7. The remaining pixel values in this example are "215" and "212".

It should be noted that the adjacent range R _x in FIGS. 5A and 5B only includes the horizontal direction, but in some embodiments, it may also include the vertical direction. For example, the aforementioned proximity range R _x can be replaced with R _x,y , which is expressed as the following mathematical formula 8. The above-mentioned Mathematical Formula 5 can be replaced with the following Mathematical Formula 9.

4A, the next step 405, the pixel value may be calculated on an integer position pixel value remaining within proximity of a depth value corresponding to R _x. Because the pixel value that is closer to the position of the integer point is more important, in this embodiment, the Gaussian distance between the position of the integer point and the remaining depth value in the adjacent range can be calculated first, and then the Gaussian distance is used as the weight to add these values. The pixel value corresponding to the remaining depth value is used to calculate the pixel value at the integer point position. In other embodiments, the Gaussian distance can also be replaced with other linear or nonlinear distance weights. Step 405 can be expressed as the following mathematical formula 10 and mathematical formula 11.

,

)為鄰近範圍R _x,y 內剩餘像素值的X座標與Y座標所形成的集合。雖然圖5A與圖5B只採用了X方向的鄰近範圍，但上述的數學式10、11是更泛用(general)的表示，不只採用X方向上的相鄰像素值，也採用了Y方向上的相鄰像素值。在計算出整數點 位置上的像素值以後可以得到曲移後影像

(x,y)，例如圖4B的曲移後影像410，其中包括至少一個空洞區420。在此實施例中是要產生出左方的視角，因此影像中的物件會往右位移，在物件的左方會產生空洞區420。 Where (x, y) is the X coordinate and Y coordinate of the integer point position. σ is a parameter, which represents the variance of the Gaussian distribution. N (

,

) Is the set formed by the X and Y coordinates of the remaining pixel values in the adjacent range R _x,y. Although Figures 5A and 5B only use the proximity range in the X direction, the above mathematical formulas 10 and 11 are more general expressions. Not only the adjacent pixel values in the X direction are used, but also the Y direction is used. The neighboring pixel value. After calculating the pixel value at the integer point position, the warped image can be obtained

( x, y ), for example, the warped image 410 in FIG. 4B includes at least one void area 420. In this embodiment, a left perspective is to be generated, so the object in the image will shift to the right, and a void area 420 will be generated on the left of the object.

Because the conventional pixel shift method only uses integer values, when there is a decimal point, the decimal point has to be discarded, so many additional holes are generated. On the contrary, the above method retains the decimal point, so the generation of holes can be greatly reduced.

(x,y)，可以根據上方像素與左上方像素計算水平梯度值，如以下數學式12所示，另外可根據左方像素與左上方像素計算垂直梯度值，如以下數學式13所示。 Next, perform the double-path gradient hole filling method, as shown in Figure 6. In this embodiment, two directions are used to fill the image 410 after the warping. Specifically, in step 610, the void in the warped image 410 is filled from top to bottom to generate the first filled image 611. When scanning the shifted image 410 from top to bottom and from left to right, it can be ensured that the upper pixel, the upper left pixel and the left pixel of the hole have values (not a hole). Therefore, for a pixel in the hole area

( x, y ), the horizontal gradient value can be calculated based on the upper pixel and the upper left pixel, as shown in the following equation 12, and the vertical gradient value can be calculated based on the left pixel and the upper left pixel, as shown in the following equation 13.

(x-1,y-1)為左上方像素，

(x,y-1)為上方像素，

(x-1,y)為左方像素。若水平梯度值

大於垂直梯度值

，表示在像素

(x,y)周圍存在垂直邊緣，因此採用上方像素

(x,y-1)來填補像素

(x,y)。另一方面，如果垂直梯度值

大於水平梯度值

，表示在像素

(x,y)周圍存在水平邊緣，則採用左方像素

(x-1,y)來填補像素

(x,y)。換言之，步驟610是根據以下數學式14來進行由上往下填補。 in

( x -1 ,y -1) is the upper left pixel,

( x,y -1) is the upper pixel,

( x -1 , y ) is the left pixel. If the horizontal gradient value

Greater than the vertical gradient value

, Expressed in pixels

There are vertical edges around ( x,y ), so the upper pixels are used

( x,y -1) to fill the pixels

( x,y ). On the other hand, if the vertical gradient value

Greater than the horizontal gradient value

, Expressed in pixels

If there is a horizontal edge around (x,y ), the left pixel is used

( x -1 , y ) to fill the pixels

( x,y ). In other words, step 610 is to fill in from top to bottom according to the following formula 14.

(x,y)即是第一填補後影像611。在此實施例中當如果垂直梯度值

等於水平梯度值

時是採用左方像素，但在其他實施例也可以採用上方像素，本發明並不在此限。 The resulting image

( x,y ) is the first padded image 611. In this embodiment, if the vertical gradient value

Equal to the horizontal gradient value

In this case, the left pixel is used, but in other embodiments, the upper pixel may also be used, and the present invention is not limited to this.

(x,y+1)、左方像素

(x-1,y)與左下方像素

(x-1,y+1)有值(不是空洞)，因此可以根據下方像素與左下方像素計算水平梯度值，如以下數學式15所示，並且根據左方像素與左下方像素計算垂直梯度值，如以下數學式16所示。 On the other hand, in step 620, the hollow area in the warped image 410 is filled from bottom to top to generate the second filled image 621. If scanning from bottom to top, from left to top, the pixels below

( x,y +1), left pixel

( x -1 ,y ) and the bottom left pixel

( x -1 , y +1) has a value (not a hole), so the horizontal gradient value can be calculated according to the lower pixel and the lower left pixel, as shown in the following mathematical formula 15, and the vertical gradient is calculated according to the left pixel and the lower left pixel The value is shown in the following mathematical formula 16.

大於垂直梯度值

，表示像素

(x,y)周圍存在垂直邊緣，因此採用下方像素

(x,y+1)來填補像素

(x,y)。如果垂直梯度值

大於水平梯度值

，採用 左方像素

(x-1,y)來填補像素

(x,y)。換言之，步驟620是根據以下數學式17進行由下往上填補。 If the horizontal gradient value

Greater than the vertical gradient value

For pixels

There are vertical edges around ( x,y ), so the pixels below are used

( x,y +1) to fill the pixels

( x,y ). If the vertical gradient value

Greater than the horizontal gradient value

, Using pixels on the left

( x -1 , y ) to fill the pixels

( x,y ). In other words, step 620 is to fill in from bottom to top according to the following formula 17.

(x,y)即是第二填補後影像621。值得注意的是，當像素周圍有45度的邊緣時，第一填補後影像611的填補效果較好，當像素周圍有135度的邊緣時，第二填補後影像621的填補效果較好。因此，在步驟630中會將空洞區的每一個像素分為兩個角度類別，接下來可以根據角度類別來從第一填補後影像611與第二填補後影像621擇一以填補對應的像素。實作上我們採用一個遮罩M(x,y)，如果是空洞區則在對應的位置填入0，非空洞區則填入255，如以下數學式18所示。 The resulting image

( x, y ) is the second padded image 621. It is worth noting that when there are 45-degree edges around the pixels, the filling effect of the first padded image 611 is better, and when there are 135-degree edges around the pixels, the filling effect of the second padded image 621 is better. Therefore, in step 630, each pixel in the cavity area is divided into two angle categories, and then one can be selected from the first padded image 611 and the second padded image 621 according to the angle category to fill the corresponding pixel. In practice, we use a mask M(x,y). If it is a hollow area, fill in 0 at the corresponding position, and fill in 255 in a non-hole area, as shown in the following mathematical formula 18.

For each pixel in the hole area, if the left pixel and the upper pixel are non-hole pixels, the corresponding pixel is classified into a 45 degree angle category, and 128 is filled in the corresponding position in the mask M(x,y), This judgment can also be expressed as Equation 19 below.

[Math 19] M(x,y)=128,if M(x,y)=0 and M(x-1,y)=M(x,y-1)=255

Next, based on the concept of warping, the hole pixels will extend horizontally, extending the 45-degree pixel category to the right. This horizontal extension can be expressed as the following number School formula 20.

[Math 20] M(x,y)=128,if M(x,y)=0 and M(x-1,y)=128

Finally, the vertical extension is performed, as shown in the following mathematical formula 21.

[Math 21] M(x,y)=128,if M(x,y)=0 and M(x,y+1)=128 and M(x-1,y)=255

After extending vertically, the remaining positions marked as "0" in the mask M(x,y) are classified into the angle category of 135 degrees. If a pixel belongs to the angle category of 45 degrees, the result of the first padded image 611 is used for padded; if a pixel belongs to the angle category of 135 degrees, the result of the second padded image 621 is used for padded, this step can indicate It is the following mathematical formula 22.

(x,y)例如為圖6的影像631，在各個角度類別都填補的很好。 The final image

( x, y ) is, for example, the image 631 in FIG. 6, which is well filled in all angle categories.

In this embodiment, a mask M(x, y) is used to classify pixels, but in other embodiments, any data structure and any symbol can also be used for classification. From another perspective, for each pixel in the cavity area, if this pixel is a hole pixel and the left and upper pixels are non-hole pixels, classify this pixel as a 45 degree angle category; if this pixel is a hole pixel And the pixel on the left belongs to the angle category of 45 degrees, then the pixel is classified as the angle category of 45 degrees; if the pixel is a hole pixel, the pixel below belongs to the angle category of 45 degrees, and the pixel on the left is a non-hole pixel, then This pixel is classified into an angle category of 45 degrees; after the above steps, the remaining hole pixels are classified into an angle category of 135 degrees. The present invention does not limit what numbers, symbols or letters are used to mark the above-mentioned angle categories.

From another perspective, the present invention also proposes a computer program product. This product can be written in any programming language and/or platform. When the computer program product is loaded into the computer system and executed, the above-mentioned computer program product can be executed. Based on the depth image generation method.

In the above-mentioned depth-based image generation method, a more natural and low-computing synthetic virtual perspective image can be obtained compared to the existing methods.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to those defined by the attached patent application scope.

210,230: Two-dimensional image

220: Depth image

310, 320, 330: steps

Claims (7)

A depth-based image generation method, suitable for an electronic device. The depth image generation method includes: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and performing pixel shifting on the two-dimensional image according to the depth image To obtain a curved image, wherein the curved image includes at least one cavity area; the at least one cavity area is filled from top to bottom to generate a first filled image, and the at least one cavity area is filled from top to bottom to The step of generating the first padded image includes: for a first pixel of the at least one cavity area, calculating a first horizontal gradient value according to the upper pixel and the upper left pixel of the first pixel, and according to the first pixel The left pixel and the upper left pixel calculate a first vertical gradient value; if the first horizontal gradient value is greater than the first vertical gradient value, use the upper pixel to fill the first pixel; and if the first vertical gradient value Greater than the first horizontal gradient value, the left pixel of the first pixel is used to fill the first pixel; the at least one cavity area is filled from bottom to top to generate a second filled image, wherein the bottom to top fills the The step of generating the second padded image with at least one cavity area includes: for a second pixel of the at least one cavity area, calculating a second horizontal gradient value according to the lower pixel and the lower left pixel of the second pixel, and according to Calculating a second vertical gradient value between the left pixel of the second pixel and the bottom left pixel; If the second horizontal gradient value is greater than the second vertical gradient value, use the lower pixel to fill the second pixel; and if the second vertical gradient value is greater than the second horizontal gradient value, use the left of the second pixel Square pixels to fill the second pixel; and classify each pixel in the at least one cavity area into a first angle category or a second angle category, if it is the first angle category, the first filled image is used, and if it is the first angle category The second angle category uses the second padded image to fill the pixel. The depth-based image generation method according to claim 1, wherein the step of classifying each pixel of the at least one cavity area into the first angle category or the second angle category includes: for each of the at least one cavity area For the pixels, if the pixel is a hole pixel, and the left and upper pixels of the pixel are non-hole pixels, the pixel is classified into the first angle category; for each of the pixels in the at least one hole area, If the pixel is a hole pixel and the left pixel of the pixel belongs to the first angle category, then the pixel is classified as the first angle category; for each of the pixels in the at least one hole area, if the pixel is a hole The pixel, the pixel below the pixel belong to the first angle category and the left pixel of the pixel is a non-hole pixel, classify the pixel as the first angle category; and classify the remaining hollow pixels in the at least one cavity area as the The second angle category. The method for generating a depth-based image according to claim 1, wherein the step of performing pixel warping on the two-dimensional image according to the depth image to obtain the warped image includes: calculating a decimal number according to a depth value in the depth image Point position, filling the depth value into the decimal point position to generate a curved depth image, and filling the corresponding pixel value in the two-dimensional image into the decimal point position to generate a curved two-dimensional image; For an integer point position, calculate a maximum depth value in a neighboring range of the integer point position, and delete the depth value in the neighboring range that differs from the maximum depth value by more than a critical value; and according to the remaining in the neighboring range The pixel value corresponding to the depth value calculates the pixel value at the position of the integer point. The depth-based image generation method according to claim 3, wherein the step of calculating the pixel value at the integer point position according to the pixel value corresponding to the remaining depth value in the adjacent range includes: calculating the integer point position and the adjacent range The Gaussian distance between the positions of the remaining depth values; and using the Gaussian distance as a weight to sum the pixel values corresponding to the remaining depth values to calculate the pixel value at the integer point position . The depth-based image generation method according to claim 4, wherein the proximity range includes a horizontal direction and a vertical direction. An electronic device includes: a memory storing a plurality of instructions; and a processor for executing the instructions to complete a plurality of steps: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and Perform pixel warping on the two-dimensional image according to the depth image to obtain a warped image, wherein the warped image includes at least one cavity area; filling the at least one cavity area from top to bottom to generate a first filled image , Wherein the step of filling the at least one cavity area from top to bottom to generate the first filled image includes: for a first pixel of the at least one cavity area, calculating a first pixel based on the upper pixel and the upper left pixel of the first pixel The first horizontal gradient value, and a first vertical gradient value is calculated according to the left pixel of the first pixel and the upper left pixel; if the first horizontal gradient value is greater than the first vertical gradient value, the upper pixel is used to fill in The first pixel; and if the first vertical gradient value is greater than the first horizontal gradient value, the left pixel of the first pixel is used to fill the first pixel; the at least one cavity area is filled from bottom to top to generate The second padded image, wherein the step of filling the at least one cavity area from bottom to top to generate the second padded image includes: for a second pixel of the at least one cavity area, according to the lower pixel and Calculating a second horizontal gradient value for the bottom left pixel, and calculating a second vertical gradient value according to the left pixel of the second pixel and the bottom left pixel; If the second horizontal gradient value is greater than the second vertical gradient value, use the lower pixel to fill the second pixel; and if the second vertical gradient value is greater than the second horizontal gradient value, use the left of the second pixel Square pixels to fill the second pixel; and classify each pixel of the at least one cavity area into a first angle type or a second angle type, if it is the first angle type, the first filled image is used, and if it is the first angle type The second angle category uses the second padded image to fill the pixel. A computer program product loaded and executed by an electronic device to complete multiple steps: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and performing pixel shifting on the two-dimensional image according to the depth image Obtain a curved image, wherein the curved image includes at least one cavity area; fill the at least one cavity area from top to bottom to generate a first filled image, wherein fill the at least one cavity area from top to bottom to generate The step of the first padded image includes: for a first pixel of the at least one cavity area, calculating a first horizontal gradient value according to the upper pixel and the upper left pixel of the first pixel, and according to the left of the first pixel The square pixel and the upper left pixel calculate a first vertical gradient value; if the first horizontal gradient value is greater than the first vertical gradient value, use the upper pixel to fill the first pixel; and if the first vertical gradient value is greater than The first horizontal gradient value, using the The left pixel of the first pixel fills the first pixel; fills the at least one cavity region from bottom to top to generate a second filled image, wherein fills the at least one void region from bottom to top to generate the second fill The step of the rear image includes: for a second pixel of the at least one cavity area, calculating a second horizontal gradient value according to the lower pixel and the lower left pixel of the second pixel, and calculating a second horizontal gradient value according to the left pixel of the second pixel and the The bottom left pixel calculates a second vertical gradient value; if the second horizontal gradient value is greater than the second vertical gradient value, the bottom pixel is used to fill the second pixel; and if the second vertical gradient value is greater than the second horizontal Gradient value, using the left pixel of the second pixel to fill the second pixel; and dividing each pixel of the at least one cavity area into a first angle category or a second angle category, if it is the first angle category The first padded image is used, and if it is the second angle category, the second padded image is used to fill the pixel.

Images