CN101867810B - Method for pre-processing deep video sequence - Google Patents

Abstract

The invention discloses a preprocessing method of a depth video sequence, which performs transformation processing on the depth video sequence to be processed, then uses the relevant information of the corresponding color video sequence to perform time smoothing processing, and then performs inverse transformation processing to obtain the preprocessed On the basis of maintaining the rendering performance of the virtual viewpoint image, the preprocessed depth video sequence can effectively improve the temporal correlation of the depth video sequence, greatly improve the compression coding efficiency of the depth video sequence, and save Code flow can reach 14.43% ~ 37.07%.

Description

A Preprocessing Method of Depth Video Sequence

technical field

The present invention relates to a processing method of a video signal, in particular to a preprocessing method of a depth video sequence.

Background technique

In a multi-view video system, a multi-view video signal is mainly composed of a multi-view color video sequence signal and a multi-view depth video sequence signal corresponding to the multi-view color video sequence signal. Depth video is very important auxiliary information in a multi-view video system. , which can be acquired by a depth camera or calculated by a depth estimation method. Since the depth camera is very expensive, most of the multi-view depth video sequences currently used for testing are calculated by a depth estimation method. Depth video represents the distance information from its corresponding color video scene to the camera imaging plane, and it quantizes the actual distance value to [0, 255]. In order to promote the application and reduce the cost, the depth video used for virtual viewpoint image rendering is not suitable for depth estimation at the receiving end, but needs to be collected or estimated at the sending end and then compressed and encoded before being transmitted to the receiving end. Therefore, the compression of depth video Coding is critical to multiview video systems.

Multi-view coded signals contain temporal, inter-view and spatial correlations. These correlations are fully utilized in the encoding process to obtain high compression efficiency. However, compared with the corresponding color video sequence, the depth video sequence information acquired by the depth camera or obtained by depth estimation has more inconsistencies in the time direction. This inconsistency reduces the temporal correlation of the depth video, and finally This leads to a decrease in coding and compression efficiency of depth video. In addition, since in multi-view video signals, most of the sequences are correlated in the time direction stronger than the correlation between viewpoints. Correspondingly, in multi-view video coding, a large number of viewpoint temporal prediction coding structures based on temporal reference are adopted. The decrease of temporal correlation of depth video leads to the increase of viewpoint correlation. This kind of predictive coding structure based on temporal reference will not be able to fully tap the correlation in depth video signal and cannot obtain ideal compression performance.

Contents of the invention

The technical problem to be solved by the present invention is to provide a multi-viewpoint depth that can effectively improve the temporal correlation of depth video sequences and greatly improve the compression coding efficiency of depth video sequences on the basis of maintaining the virtual viewpoint image rendering performance. Preprocessing methods for video sequences.

The technical solution adopted by the present invention to solve the above technical problems is: a preprocessing method for a multi-viewpoint depth video sequence. The multi-viewpoint video signal is mainly composed of a color video sequence and a depth video sequence corresponding to the color video sequence. The preprocessing method includes the following steps:

① Record the depth video sequence to be preprocessed as D(m, n, k), where m represents the horizontal resolution of the depth video frame in the depth video sequence D(m, n, k), and n represents the depth video sequence D The vertical resolution of the depth video frame in (m, n, k), m × n represents the resolution of the depth video frame in the depth video sequence D (m, n, k), and k represents the depth video sequence D (m , n, the frame number of the depth video frame contained in k);

② Transform the depth video sequence D(m, n, k) to obtain the transformed depth video sequence, denoted as D′(m, k, n), where the transformed depth video sequence D′( m, k, the resolution of the depth video frame in n) is m × k, and the frame number of the depth video frame contained in the depth video sequence D' (m, k, n) after transformation process is n;

③Sequentially perform temporal smoothing processing on each column of pixels in each frame of depth video frame in the transformed depth video sequence D′(m,k,n) to obtain the smoothed depth video sequence, denoted as D″(m , k, n), wherein, the resolution of the depth video frame in the smoothed depth video sequence D″(m, k, n) is m×k, and the smoothed depth video sequence D″(m, k , the frame number of the depth video frame contained in n) is n;

④ Perform inverse transformation processing on the smoothed depth video sequence D″(m, k, n) to obtain the preprocessed depth video sequence, denoted as D″′(m, n, k), where, after preprocessing The resolution of the depth video frames in the depth video sequence D"'(m, n, k) is m×n, and the depth video frames contained in the preprocessed depth video sequence D"'(m, n, k) The number of frames is k.

In the described step ②, the specific process of transforming the depth video sequence D (m, n, k) is:

②-1, sequentially take out the luminance component Y of m pixels in the i-th row of all depth video frames in the depth video sequence D (m, n, k), wherein, 1≤i≤n;

②-2, take the luminance component Y of the k rows of pixels taken out in order as the luminance component Y of the i-th frame depth video frame in the depth video sequence D' (m, k, n) after the transformation process, and then transform the The values of the first chrominance component U and the second chrominance component V of all depth video frames in the subsequent depth video sequence D′(m, k, n) are set to 128, where 1≤i≤n.

In the described step 3., the specific process of carrying out time smoothing processing to each column pixel in each frame of depth video frame in the depth video sequence D' (m, k, n) after the transformation process is as follows:

③-1. Note that the color video sequence corresponding to the depth video sequence D (m, n, k) to be preprocessed is C (m, n, k), wherein, in the color video sequence C (m, n, k) The resolution of the color video frame is m×n, and the number of color video frames contained in the color video sequence C (m, n, k) is k; The same transformation process is performed on the color video sequence C (m, n, k), and the transformed color video sequence is obtained, denoted as C'(m, k, n), where the transformed color video sequence The resolution of the color video frames in the sequence C'(m,k,n) is m×k, and the number of color video frames contained in the transformed color video sequence C'(m,k,n) is n ;

③-2. Define the pth depth video frame in the transformed depth video sequence D′(m, k, n) as the current depth video frame, where the initial value of p is 1, 1≤p≤n ;

③-3. Process the current depth video frame in units of columns, define the qth column of pixels currently being processed in the current depth video frame as the current column, where the initial value of q is 1, 1≤q≤m;

③-4. Classify each pixel according to the depth value of each pixel in the current column, and obtain several first pixel sets, wherein the specific process of classification is: a1. Classify the first pixel in the current column into the sth In the first pixel set, define the second pixel in the current column as the current depth pixel; b1. determine the difference between the depth value of the current depth pixel and the average value of the depth values of all pixels in the sth first pixel set Whether the absolute value is less than the set first threshold, if yes, classify the current depth pixel into the sth first pixel set, and then perform step d1, otherwise, classify the current depth pixel into the s+1th first pixel set in the pixel set, and continue to execute; c1. make s'=s+1, s=s'; d1. use the next pixel in the current column as the current depth pixel, and return to step b1 to continue execution until all The pixel is processed, and the initial value of s is 1, and the initial value of s′ is 0;

③-5, define the column corresponding to the current column in the color video sequence C' (m, k, n) after the conversion process as the corresponding column, and classify each pixel according to the brightness component value of each pixel in the corresponding column, and obtain Several second pixel sets, wherein the specific process of classification is: a2. Classify the first pixel in the corresponding column into the t-th second pixel set, and define the second pixel in the corresponding column as the current color pixel ; b2. determine whether the absolute value of the difference between the brightness component value of the current color pixel and the average value of the brightness component values of all pixels in the tth second pixel set is less than the second threshold value set, if so, then the current color The pixel is included in the tth second pixel set, and then step d2 is performed, otherwise, the current color pixel is included in the t+1th second pixel set, and continues to execute; c2. Make t′=t+1, t=t'; d2. Use the next pixel in the corresponding column as the current color pixel, and return to step b2 to continue until all pixels in the corresponding column are processed, where the initial value of t is 1, and the initial value of t' is 0;

③-6. Reclassify each pixel in the current column according to the first pixel set and the second pixel set to obtain several third pixel sets, wherein the specific process of reclassification is: a3. Pixels are classified into the vth third pixel set, and the second pixel in the current column is defined as the current depth pixel; b3. Determine whether the current depth pixel and its previous pixel belong to the same first pixel set and the corresponding column Whether the color pixel corresponding to the current depth pixel and its previous pixel belong to the same second pixel set, if yes, then classify the current depth pixel into the vth third pixel set, and then perform step d3, otherwise, Classify the current depth pixel into the v+1th third pixel set, and continue to execute; c3. make v'=v+1, v=v'; d3. use the next pixel in the current column as the current depth pixel , and return to step b3 to continue until all pixels in the current column are processed, wherein the initial value of v is 1, and the initial value of v' is 0;

③-7. Taking the average value of the depth values of all pixels included in each third pixel set in the current column as the depth values of all pixels in the third pixel set to obtain the smoothed column pixels;

③-8. Use the next column in the current depth video frame as the current column, and then return to step ③-4 until all columns in the current depth video frame are processed;

③-9, the next depth video frame in the transformed depth video sequence D′ (m, k, n) is used as the current depth video frame, and then returns to perform step ③-3 until the transformed depth video sequence All depth video frames in D'(m, k, n) are processed.

The first threshold is 10, and the second threshold is 5.

In the described step ④, the specific process of inverse transforming the smoothed depth video sequence D″ (m, k, n) is:

4.-1, take out the luminance component Y of the m pixels of the jth row of all depth video frames in the depth video sequence D" (m, k, n) after smoothing in turn, wherein, 1≤j≤k;

④-2, take the luminance component Y of the n rows of pixels taken out in sequence as the luminance component Y of the jth frame depth video frame of the depth video sequence D"' (m, n, k) after preprocessing, and then preprocess The values of the first chrominance component U and the second chrominance component V of all depth video frames in the subsequent depth video sequence D″'(m, n, k) are set to 128, where 1≤j≤k.

Compared with the prior art, the present invention has the advantage of transforming the depth video sequence to be processed, then using the relevant information of the corresponding color video sequence to perform time smoothing processing, and then performing inverse transformation processing to obtain the preprocessed depth Video sequence, the preprocessed depth video sequence can effectively improve the time correlation of the depth video sequence on the basis of maintaining the rendering performance of the virtual viewpoint image, greatly improving the compression coding efficiency of the depth video sequence, and saving the code stream It can reach 14.43% to 37.07%.

Description of drawings

Fig. 1 is a block flow diagram of the present invention;

Figure 2a is the depth video frame of the first frame of the depth video sequence corresponding to the seventh viewpoint in the "Door Flowers" test video sequence;

Fig. 2b is the depth video frame obtained after the depth video frame shown in Fig. 2a is transformed;

Figure 2c is the first color video frame of the color video sequence corresponding to the seventh viewpoint in the "Door Flowers" test video sequence;

Fig. 2 d is the color video frame obtained after the color video frame shown in Fig. 2c is transformed;

Figure 3a is a schematic diagram of the classification results of the current column;

Figure 3b is a schematic diagram of the classification results of the corresponding columns;

Fig. 3c is a schematic diagram of the result of reclassifying the current column shown in Fig. 3a;

Fig. 4 is the smoothed depth video frame obtained after the transformed depth video frame shown in Fig. 2b is temporally smoothed;

Fig. 5 is the depth video frame obtained after the depth video frame after the smoothing processing shown in Fig. 4 is processed by inverse transformation;

Figure 6a is a schematic diagram of the comparison of the encoding rate-distortion performance of the original depth video sequence and the preprocessed depth video sequence of the Book Arrival test sequence;

Fig. 6b is a schematic diagram of the coding rate-distortion performance comparison between the original depth video sequence and the preprocessed depth video sequence of the Door Flowers test sequence;

Figure 6c is a schematic diagram of the rate-distortion performance comparison between the original depth video sequence and the preprocessed depth video sequence of the Leave Laptop test sequence;

Figure 6d is a schematic diagram of the comparison of the rate-distortion performance of the original depth video sequence and the preprocessed depth video sequence of the Alt Moabit test sequence;

Figure 7a is the original frame S ₈ T ₀ of the Door Flowers test sequence;

Figure 7b is the S ₈ T ₀ obtained after drawing the color video and the corresponding depth video using the original viewpoints 7 and 10;

Fig. 7c is S ₈ T ₀ obtained after rendering using the original color videos of viewpoints 7 and 10 and the corresponding preprocessed depth videos.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

Due to the inconsistency of the depth video sequence in the multi-view video signal in time, the weak correlation is caused. Therefore, in order to improve the coding efficiency of the depth video sequence, the present invention proposes a preprocessing method for the multi-view depth video sequence. , to preprocess the depth video sequence before encoding it to enhance its temporal coherence.

The flow process of the preprocessing method of the multi-viewpoint depth video sequence proposed by the present invention is as shown in Figure 1, and it comprises the following steps:

① Record the depth video sequence to be preprocessed as D(m, n, k), where m represents the horizontal resolution of the depth video frame in the depth video sequence D(m, n, k), and n represents the depth video sequence D The vertical resolution of the depth video frame in (m, n, k), m × n represents the resolution of the depth video frame in the depth video sequence D (m, n, k), and k represents the depth video sequence D (m , n, k) is the number of depth video frames contained in the frame.

② Transform the depth video sequence D(m, n, k) to obtain the transformed depth video sequence, denoted as D′(m, k, n), where the transformed depth video sequence D′( The resolution of the depth video frames in m, k, n) is m×k, and the number of depth video frames included in the transformed depth video sequence D′(m, k, n) is n.

In this specific embodiment, the specific process of transforming the depth video sequence D (m, n, k) is:

②-1. Sequentially extract the luminance components Y of m pixels in the i-th row of all depth video frames in the depth video sequence D(m, n, k), where 1≤i≤n.

②-2, take the luminance component Y of the k rows of pixels taken out in order as the luminance component Y of the i-th frame depth video frame in the depth video sequence D' (m, k, n) after the transformation process, and then transform the The values of the first chrominance component U and the second chrominance component V of all depth video frames in the subsequent depth video sequence D′(m, k, n) are set to 128, where 1≤i≤n.

Figure 2a shows the depth video frame of the first frame of the depth video sequence corresponding to the 7th viewpoint in the "Door Flowers" test video sequence, and Figure 2b shows the depth obtained by transforming the depth video frame shown in Figure 2a In the video frame, the light and dark parts shown in the circles in Figure 2b correspond to the moving parts, and the parts shown in the boxes in Figure 2b reflect the tiny This is the embodiment of the temporal inconsistency of the depth video frame D(m,n,k).

③ The weak temporal correlation of the depth video sequence is reflected in the depth of each column of pixels in the depth video frame with a resolution of m×k in the transformed depth video sequence D′(m, k, n) Therefore, the present invention proposes to perform temporal smoothing processing on each column of pixels in each frame of depth video frame in the transformed depth video sequence D′(m, k, n) sequentially to obtain the smoothed depth The video sequence is denoted as D″(m, k, n), wherein the resolution of the depth video frame in the smoothed depth video sequence D″(m, k, n) is m×k, and the smoothed The number of depth video frames contained in the depth video sequence D″ (m, k, n) is n. In the depth video sequence D′ (m, k, n) after the transformation process, each frame of the depth video frame Temporal smoothing of each column of pixels can effectively enhance the temporal consistency of the transformed depth video sequence D′(m,k,n), and can preserve the transformed depth video sequence D′(m, k, the moving part in n).

In this specific embodiment, the specific process of performing temporal smoothing processing on each column of pixels in each frame of the depth video frame in the transformed depth video sequence D'(m, k, n) in turn is as follows:

③-1. Note that the color video sequence corresponding to the depth video sequence D (m, n, k) to be preprocessed is C (m, n, k), wherein, in the color video sequence C (m, n, k) The resolution of the color video frame is m×n, and the number of color video frames contained in the color video sequence C(m,n,k) is k, where the color video sequence C(m,n,k) and depth The video sequence D (m, n, k) is corresponding, and they have the video frame of the same number of frames and the color video frame has the same resolution as the depth video frame; then adopt the same resolution as the depth video sequence D (m, n, k) The same transformation process is performed on the color video sequence C (m, n, k), and the transformed color video sequence is obtained, denoted as C'(m, k, n), where the transformed color video sequence The resolution of the color video frames in the sequence C'(m,k,n) is m×k, and the number of color video frames contained in the transformed color video sequence C'(m,k,n) is n .

In this specific embodiment, the specific process of transforming the color video sequence C (m, n, k) is as follows: first take out the i-th row of all color video frames in the color video sequence C (m, n, k) sequentially The luminance components Y of the m pixels in , where 1≤i≤n; then take the luminance components Y of the k rows of pixels taken out in order as the color video sequence C′(m, k, n) after the transformation process The luminance component Y of the i frame color video frame, then the values of the first chrominance component U and the second chrominance component V of all color video frames in the color video sequence C' (m, k, n) after the transformation process are equal Set to 128, where 1≤i≤n.

Figure 2c shows the first color video frame of the color video sequence corresponding to the seventh viewpoint in the "Door Flowers" test video sequence, and Figure 2d shows the color video frame obtained after the color video frame shown in Figure 2c is transformed. In the video frame, the light and dark parts shown in the circles in Figure 2d correspond to the moving parts, the parts shown in the boxes in Figure 2d are obviously different from the parts shown in the boxes in Figure 2b, and the parts shown in the boxes in Figure 2d Partially reflects that each column of pixels in a color video frame is substantially consistent.

③-2. Define the pth depth video frame in the transformed depth video sequence D′(m, k, n) as the current depth video frame, where the initial value of p is 1, 1≤p≤n .

③-3. The current depth video frame is processed in units of columns, and the qth column of pixels currently being processed in the current depth video frame is defined as the current column, where the initial value of q is 1, 1≤q≤m.

③-4. Classify each pixel according to the depth value of each pixel in the current column, and obtain several first pixel sets, wherein the specific process of classification is: a1. Classify the first pixel in the current column into the sth In the first pixel set, define the second pixel in the current column as the current depth pixel; b1. determine the difference between the depth value of the current depth pixel and the average value of the depth values of all pixels in the sth first pixel set Whether the absolute value is less than the set first threshold, if so, it is considered that the depth value of the current depth pixel has not changed drastically, and the current depth pixel is considered to belong to the same category as the previous pixel, so the current depth pixel is classified as the sth In a set of pixels, then execute step d1, otherwise, it is considered that the depth value of the current depth pixel has changed drastically, and it is considered that the current depth pixel does not belong to the same class as the previous pixel, so the current depth pixel is classified into the s+1th In a pixel set, and continue to execute; c1. make s'=s+1, s=s'; d1. use the next pixel in the current column as the current depth pixel, and return to step b1 to continue until the current column All pixels are processed, and the initial value of s is 1, and the initial value of s′ is 0. Suppose the result of the current column classification is shown in Figure 3a, the first 3 pixels in Figure 3a belong to the first first pixel set, the middle 5 pixels belong to the second first pixel set, and the last 2 pixels belong to the third first pixel set A collection of pixels.

③-5, define the column corresponding to the current column in the color video sequence C' (m, k, n) after the conversion process as the corresponding column, and classify each pixel according to the brightness component value of each pixel in the corresponding column, and obtain Several second pixel sets, wherein the specific process of classification is: a2. Classify the first pixel in the corresponding column into the t-th second pixel set, and define the second pixel in the corresponding column as the current color pixel ; b2. determine whether the absolute value of the difference between the brightness component value of the current color pixel and the average value of the brightness component values of all pixels in the tth second pixel set is less than the second threshold value set, if so, then the current color The pixel is included in the tth second pixel set, and then step d2 is performed, otherwise, the current color pixel is included in the t+1th second pixel set, and continues to execute; c2. Make t′=t+1, t=t'; d2. Use the next pixel in the corresponding column as the current color pixel, and return to step b2 to continue until all pixels in the corresponding column are processed, where the initial value of t is 1, and the initial value of t' is 0. Assuming that the result of the corresponding column classification is shown in Figure 3b, the first three pixels denoted by a in Figure 3b belong to the first second pixel set, the middle three pixels denoted by b belong to the second second pixel set, and the middle The three pixels denoted by c belong to the third second pixel set, and the last pixel denoted by d belongs to the fourth second pixel set.

③-6. Reclassify each pixel in the current column according to the first pixel set and the second pixel set to obtain several third pixel sets, wherein the specific process of reclassification is: a3. Pixels are classified into the vth third pixel set, and the second pixel in the current column is defined as the current depth pixel; b3. Determine whether the current depth pixel and its previous pixel belong to the same first pixel set and the corresponding column Whether the color pixel corresponding to the current depth pixel and its previous pixel belong to the same second pixel set, if yes, then classify the current depth pixel into the vth third pixel set, and then perform step d3, otherwise, Classify the current depth pixel into the v+1th third pixel set, and continue to execute; c3. make v'=v+1, v=v'; d3. use the next pixel in the current column as the current depth pixel , and return to step b3 to continue until all pixels in the current column are processed, wherein the initial value of v is 1, and the initial value of v′ is 0.

According to the pixel sets shown in Figure 3a and Figure 3b, the result obtained by reclassifying the current column shown in Figure 3a is shown in Figure 3c. It can be seen from Figure 3c that the current column has been re-divided into five third pixels Set, that is, the 1st-3rd pixels in the current column are classified as the 1st third pixel set, the 4th-6th pixels are classified as the 2nd third pixel set, and the 7th-8th pixels are classified as the 3rd pixel set In the third pixel set, the 9th pixel is classified into the 4th third pixel set, and the 10th pixel is classified into the 5th third pixel set.

③-7. Taking the average value of the depth values of all the pixels included in each third pixel set in the current column as the depth value of all the pixels in the third pixel set to obtain the smoothed column pixels.

③-8. Set the next column in the current depth video frame as the current column, and then return to step ③-4 until all columns in the current depth video frame are processed.

③-9, the next depth video frame in the transformed depth video sequence D′ (m, k, n) is used as the current depth video frame, and then returns to perform step ③-3 until the transformed depth video sequence All depth video frames in D'(m, k, n) are processed.

In the present invention, in the process of classifying each column of pixels in the transformed depth video sequence D'(m,k,n), the corresponding transformed color video sequence C'(m,k,n) is used , n) The main purpose of the luminance component of the corresponding column pixels is to preserve the information of those areas whose depth changes are subtle in time but the luminance changes in the color video are strong, so that the depth video sequence can be smoothed in time and maintain Depth information for key areas. Therefore, temporally smoothing each column of pixels in each depth video frame in the transformed depth video sequence D′(m, k, n) is beneficial to the compression coding of the depth video sequence and to the virtual viewpoint The drawing of the image.

In this specific embodiment, the first threshold can take a value of 10, and the second threshold can take a value of 5, wherein the reason why the first threshold is greater than the second threshold is that the color video sequence C(m, n, k) is temporally It is more consistent than the depth video sequence D(m,n,k).

Fig. 4 provides the smoothed depth video frame obtained after the transformed depth video frame shown in Fig. 2b is temporally smoothed, and the transformed depth video sequence D'(m, k, n), in addition to maintaining the information of the motion region in the depth video sequence, the pixels of the frame in the vertical direction have stronger consistency.

④ Perform inverse transformation processing on the smoothed depth video sequence D″(m, k, n) to obtain the preprocessed depth video sequence, denoted as D″′(m, n, k), where, after preprocessing The resolution of the depth video frames in the depth video sequence D"'(m, n, k) is m×n, and the depth video frames contained in the preprocessed depth video sequence D"'(m, n, k) The number of frames is k.

In this specific embodiment, the specific process of performing inverse transformation processing on the smoothed depth video sequence D″(m, k, n) is:

④-1. Sequentially extract the luminance components Y of the m pixels in the jth row of all depth video frames in the smoothed depth video sequence D″(m, k, n), where 1≤j≤k.

④-2, take the luminance component Y of the n rows of pixels taken out in sequence as the luminance component Y of the jth frame depth video frame of the depth video sequence D"' (m, n, k) after preprocessing, and then preprocess The values of the first chrominance component U and the second chrominance component V of all depth video frames in the subsequent depth video sequence D″'(m, n, k) are set to 128, where 1≤j≤k.

FIG. 5 shows the depth video frame obtained after the smoothed depth video frame shown in FIG. 4 is inversely transformed.

In order to verify the validity and feasibility of the preprocessing method of the present invention, the BookArrival, Door Flowers, Leave Laptop and Alt Moabit test sequences provided by HHI were first selected, and then the seventh-order values of these sequences were calculated by using the existing depth estimation method DERS. The depth videos of the 7th viewpoints are taken as the original depth video sequences, and then the depth videos of the 7th viewpoint of these sequences are processed by the preprocessing method proposed by the present invention to obtain the preprocessed depth video sequences.

Here, the performance of the preprocessing method proposed by the present invention will be measured from the compression efficiency of the depth video sequence and the rendering of the virtual viewpoint image.

In terms of depth video sequence compression efficiency, the inventive method can save 14.43%～37.07% code rate, and table 1 has listed the original depth video sequence of above-mentioned each sequence and the depth video sequence after preprocessing under the same condition (BasisQP= 22) Code rate comparison of encoding. Since the preprocessing method of the present invention mainly enhances the temporal correlation of the depth video sequence, it is more effective for the B frame, and the B frame can save 19.16%-41.28% of the code rate. The effectiveness of the preprocessing method of the present invention is mainly determined by the degree of inconsistency in time of each original depth video sequence. Since the Alt Moabit depth video sequence has large temporally inconsistent regions in the foreground and background regions, the preprocessing method of the present invention can greatly improve the encoding and compression efficiency of the sequence. Fig. 6a, Fig. 6b, Fig. 6c and Fig. 6d respectively show the coding rate-distortion performance comparison charts of the original depth video sequence and the preprocessed depth video sequence of the BookArrival, Door Flowers, Leave Laptop and Alt Moabit test sequences, from Fig. 6a, 6b, 6c and 6d, it can be seen that the preprocessing method of the present invention can greatly improve the rate-distortion performance.

Table 1 Comparison of coding bit rates between the original depth video sequence and the preprocessed depth video sequence

In terms of virtual viewpoint image rendering, the preprocessing method of the present invention does not have much influence on the virtual viewpoint image rendering. Figure 7a shows the original frame S ₈ T ₀ of the Door Flowers test sequence; Figure 7b shows the S ₈ T ₀ obtained after rendering using the original color video of viewpoint 7 and 10 and the corresponding depth video; Figure 7c shows S ₈ T ₀ obtained after rendering by using the original color videos of viewpoints 7 and 10 and the corresponding preprocessed depth videos. From a subjective perspective, the quality of the images shown in Figure 7b and Figure 7c is almost the same. In order to reflect the difference between the two more clearly, Table 2 lists the peak signal-to-noise ratio (PSNR) of the virtual viewpoint image drawn using the original depth video and the preprocessed depth video relative to the original viewpoint image, and two comparison situations, It can be seen from Table 2 that the PNSR of the luma component Y is almost unchanged, and the PNSR of the first chrominance component U and the second chrominance component V are almost the same, and some have a slight increase.

Table 2 Quality comparison of virtual viewpoint image rendering (dB)

Claims (2)

1. a preprocessing method of multi-viewpoint depth video sequence, multiviewpoint video signal is made up of color video sequence and the corresponding depth video sequence with described color video sequence, it is characterized in that this preprocessing method comprises the following steps: ① Record the depth video sequence to be preprocessed as D(m, n, k), where m represents the horizontal resolution of the depth video frame in the depth video sequence D(m, n, k), and n represents the depth video sequence D The vertical resolution of the depth video frame in (m, n, k), m × n represents the resolution of the depth video frame in the depth video sequence D (m, n, k), and k represents the depth video sequence D (m , n, the frame number of the depth video frame contained in k); ② Transform the depth video sequence D(m, n, k) to obtain the transformed depth video sequence, denoted as D′(m, k, n), where the transformed depth video sequence D′( m, k, the resolution of the depth video frame in n) is m × k, and the frame number of the depth video frame contained in the depth video sequence D' (m, k, n) after transformation process is n; Here, the specific process of transforming the depth video sequence D(m, n, k) is: ②-1, sequentially take out the luminance component Y of m pixels in the i-th row of all depth video frames in the depth video sequence D (m, n, k), wherein, 1≤i≤n; ②-2, take the luminance component Y of the k rows of pixels taken out in order as the luminance component Y of the i-th frame depth video frame in the depth video sequence D' (m, k, n) after the transformation process, and then transform the The values of the first chrominance component U and the second chrominance component V of all depth video frames in the depth video sequence D' (m, k, n) after are set to 128, wherein, 1≤i≤n; ③Sequentially perform temporal smoothing processing on each column of pixels in each frame of depth video frame in the transformed depth video sequence D′(m,k,n) to obtain the smoothed depth video sequence, denoted as D″(m , k, n), wherein, the resolution of the depth video frame in the smoothed depth video sequence D″(m, k, n) is m×k, and the smoothed depth video sequence D″(m, k , the frame number of the depth video frame contained in n) is n; Here, the specific process of performing temporal smoothing processing on each column of pixels in each frame of depth video frame in the transformed depth video sequence D′(m, k, n) in turn is as follows: ③-1. Note that the color video sequence corresponding to the depth video sequence D (m, n, k) to be preprocessed is C (m, n, k), wherein, in the color video sequence C (m, n, k) The resolution of the color video frame is m×n, and the number of color video frames contained in the color video sequence C (m, n, k) is k; The same transformation process is performed on the color video sequence C (m, n, k), and the transformed color video sequence is obtained, denoted as C'(m, k, n), where the transformed color video sequence The resolution of the color video frames in the sequence C'(m,k,n) is m×k, and the number of color video frames contained in the transformed color video sequence C'(m,k,n) is n ; ③-2. Define the pth depth video frame in the transformed depth video sequence D′(m, k, n) as the current depth video frame, where the initial value of p is 1, 1≤p≤n ; ③-3. Process the current depth video frame in units of columns, define the qth column of pixels currently being processed in the current depth video frame as the current column, where the initial value of q is 1, 1≤q≤m; ③-4. Classify each pixel according to the depth value of each pixel in the current column, and obtain several first pixel sets, wherein the specific process of classification is: a1. Classify the first pixel in the current column into the sth In the first pixel set, define the second pixel in the current column as the current depth pixel; b1. determine the difference between the depth value of the current depth pixel and the average value of the depth values of all pixels in the sth first pixel set Whether the absolute value is less than the set first threshold, if yes, classify the current depth pixel into the sth first pixel set, and then perform step d1, otherwise, classify the current depth pixel into the s+1th first pixel set in the pixel set, and continue to execute; c1. make s'=s+1, s=s'; d1. use the next pixel in the current column as the current depth pixel, and return to step b1 to continue execution until all The pixel is processed, and the initial value of s is 1, and the initial value of s′ is 0; ③-5, define the column corresponding to the current column in the color video sequence C' (m, k, n) after the conversion process as the corresponding column, and classify each pixel according to the brightness component value of each pixel in the corresponding column, and obtain Several second pixel sets, wherein the specific process of classification is: a2. Classify the first pixel in the corresponding column into the t-th second pixel set, and define the second pixel in the corresponding column as the current color pixel ; b2. determine whether the absolute value of the difference between the brightness component value of the current color pixel and the average value of the brightness component values of all pixels in the tth second pixel set is less than the second threshold value set, if so, then the current color The pixel is included in the tth second pixel set, and then step d2 is performed, otherwise, the current color pixel is included in the t+1th second pixel set, and continues to execute; c2. Make t′=t+1, t=t'; d2. Use the next pixel in the corresponding column as the current color pixel, and return to step b2 to continue until all pixels in the corresponding column are processed, where the initial value of t is 1, and the initial value of t' is 0; ③-6. Reclassify each pixel in the current column according to the first pixel set and the second pixel set to obtain several third pixel sets, wherein the specific process of reclassification is: a3. Pixels are classified into the vth third pixel set, and the second pixel in the current column is defined as the current depth pixel; b3. Determine whether the current depth pixel and its previous pixel belong to the same first pixel set and the corresponding column Whether the color pixel corresponding to the current depth pixel and its previous pixel belong to the same second pixel set, if yes, then classify the current depth pixel into the vth third pixel set, and then perform step d3, otherwise, Classify the current depth pixel into the v+1th third pixel set, and continue to execute; c3. make v'=v+1, v=v'; d3. use the next pixel in the current column as the current depth pixel , and return to step b3 to continue until all pixels in the current column are processed, wherein the initial value of v is 1, and the initial value of v' is 0; ③-7. Taking the average value of the depth values of all pixels included in each third pixel set in the current column as the depth values of all pixels in the third pixel set to obtain the smoothed column pixels; ③-8. Use the next column in the current depth video frame as the current column, and then return to step ③-4 until all columns in the current depth video frame are processed; ③-9, the next depth video frame in the transformed depth video sequence D′ (m, k, n) is used as the current depth video frame, and then returns to perform step ③-3 until the transformed depth video sequence All depth video frames in D'(m, k, n) are processed; ④ Perform inverse transformation processing on the smoothed depth video sequence D″(m, k, n) to obtain the preprocessed depth video sequence, denoted as D′″(m, n, k), where, after preprocessing The resolution of the depth video frames in the depth video sequence D′″(m, n, k) is m×n, and the depth video frames contained in the preprocessed depth video sequence D′″(m, n, k) The number of frames is k; Here, the specific process of inverse transforming the smoothed depth video sequence D″(m, k, n) is: 4.-1, take out the luminance component Y of the m pixels of the jth row of all depth video frames in the depth video sequence D" (m, k, n) after smoothing in turn, wherein, 1≤j≤k; ④-2, take the luminance component Y of the n rows of pixels taken out in order as the luminance component Y of the jth frame depth video frame of the depth video sequence D' "(m, n, k) after preprocessing, and then preprocess The values of the first chrominance component U and the second chrominance component V of all depth video frames in the subsequent depth video sequence D′″(m, n, k) are set to 128, where 1≤j≤k. 2 . The method for preprocessing a multi-view depth video sequence according to claim 1 , wherein the first threshold is 10, and the second threshold is 5. 3 .

Images