CN106028046A - Lagrange multiplier correction method for multi-view deep video encoding - Google Patents

Abstract

The invention discloses a Lagrangian multiplier correction method for multi-viewpoint depth map encoding, which mainly solves the problem that the influence of the texture quality of the same viewpoint on the depth map Lagrangian multiplier is not considered in the prior art, which leads to The overall encoding performance of 3D video is not high. The implementation plan is: before encoding multi-view depth video, construct a correction factor according to the quantization parameter Q _d of the depth video to be encoded and the quantization parameter Q _t used in the same-view texture video encoding; use this correction factor to encode the existing depth The adopted Lagrangian multiplier is corrected; the corrected Lagrangian multiplier is used in the rate-distortion optimization process of the depth map coding. The present invention improves the overall coding performance of 3D video, and can be used to code 3D video in any texture and depth quantization parameter QP combination mode.

Description

Lagrangian multiplier correction method for multi-view depth video coding

technical field

The invention belongs to the technical field of video coding, and in particular relates to a method for selecting Lagrangian multipliers, which can be used in the rate-distortion optimization process of multi-viewpoint video depth sequence coding.

Background technique

The real depth perception and immersive visual enjoyment brought by 3D video have led to a sharp increase in people's demand for 3D applications. 3D applications, multi-viewpoint video technology, and virtual viewpoint synthesis have become hot spots in academic and commercial research and development. one. The 3D Video Coding Joint Team JCT-3V, jointly established by two video standardization organizations, the International Motion Picture Experts Group MPEG and the Video Coding Experts Group VCEG, aims to allow the joint team to develop the next-generation 3D video coding standard based on the high-efficiency video coding standard HEVC 3D-HEVC.

The 3D-HEVC standard uses the multi-view plus depth MVD format as its data format. The MVD data format usually includes texture videos of multiple viewpoints and corresponding depth videos, and the bit stream obtained by encoding these data is sent to the decoding end. Using the DIBR technology based on the depth map, the decoder synthesizes the reconstructed texture video and the depth video of different viewpoints to obtain the texture video of the required virtual viewpoint. Theoretically, the encoding of each viewpoint can be encoded using the HEVC encoding framework at the same time, but for some unique features of the depth map itself, new encoding tools are developed to improve the overall encoding performance of 3D video.

The coding framework of 3D-HEVC is shown in Figure 1. First, the texture of the basic viewpoint is coded, and then its depth map is coded. After one frame of the basic viewpoint is coded, the texture and depth of each non-basic viewpoint are coded sequentially. , and so on until all video sequences are encoded. Since there is a lot of cross-redundant information among multi-view videos, inter-view information is used to improve compression efficiency when encoding non-basic views. In the current general test environment of 3D-HEVC, the texture and depth quantization parameters QP of the basic viewpoint are a set of fixed combinations, as shown in Table 1. The QP of the texture and depth of the non-basic viewpoint is based on the QP value corresponding to the basic viewpoint plus ΔQP, and the default value of ΔQP is 3, which can be set in the encoding configuration file.

In the depth map encoding process, the best encoding mode and parameters are selected through the rate-distortion optimization method, that is, the encoding mode with the smallest rate-distortion cost J=D+λ·R is selected as the final encoding mode, where D represents the current encoding The distortion brought by the mode, R represents the number of coded bits required in the current coding mode, and λ is the Lagrangian multiplier.

Because the depth video is not used for direct viewing, but for synthesizing virtual views for end users to watch. The purpose of finally encoding the depth map is to obtain a certain quality of virtual view. The factors affecting the quality of the virtual view are not only the depth map, but also many other factors, such as the texture video quality used for synthesis, and the rounding operation in the synthesis process will introduce distortion, and only the distortion of the depth map itself is regarded as rate distortion. Distortion measurement during optimization is inappropriate. Therefore, the synthetic view distortion introduced by the current coded depth block is also used as the distortion measure in the rate-distortion optimization process.

In the depth coding process of 3D-HEVC, the view synthesis distortion change SVDC is introduced into the rate-distortion optimization to select the coding mode of the depth map. Due to the change of the distortion mechanism, the Lagrangian multipliers used in the depth map during the rate-distortion optimization process should also be corrected accordingly. In the current 3D-HEVC reference software, the Lagrangian multiplier in the depth map rate-distortion optimization process is corrected with a scaling factor related to the depth map quantization parameter QP.

The Lagrangian multiplier can be expressed as the value of one bit used for encoding. It is generally believed that the view synthesis distortions introduced by texture coding and depth coding are independent of each other. However, the quality of texture video directly affects the quality of synthetic views. When the quality of texture coding is low, even if the number of bits used for depth coding is increased, The improvement to the quality of the synthetic view is also negligible. At this time, the Lagrangian multiplier during depth coding should be increased to avoid unnecessary bit overhead. Theoretically, base viewpoints can be encoded with any combination of QPs. When the texture QP is changed and the depth QP is unchanged for coding, the Lagrangian multiplier of the depth image in the current coding standard is only related to the depth QP, and the influence of the texture quality on it is not considered. Based on the above analysis, it is not accurate and universal to consider the relationship between Lagrangian multipliers and depth map QP unilaterally, and it will reduce the overall coding performance of 3D video when coding texture QP changes.

Table 1 Quantization parameters of basic viewpoint texture and depth

QP _T0 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 QP _D0 51 50 50 50 50 49 48 47 47 46 45 45 44 44 43 43 42 42 41 41 40 39 38 37 36 35 34

Contents of the invention

In order to overcome the deficiencies of the prior art, the present invention provides a Lagrangian multiplier correction method for multi-viewpoint depth video coding in consideration of texture video quality, so as to improve the overall coding performance of 3D video.

The idea of the present invention is: under the current 3D-HEVC coding framework, when the texture QP is changed and the depth QP is unchanged for coding, the Lagrangian multiplier of the depth map is only related to the depth QP. It means that in the case of different texture QP encodings, the same depth QP encoding process does not take into account the impact of texture quality on Lagrangian multipliers. Theoretically, multi-view texture video and depth video can be encoded with any combination of QPs. When the quality of the texture video is good, that is, when the texture QP is small, the quality of the depth map contributes a lot to the quality of the synthetic view. At this time, the coding bits of the depth map can be appropriately increased, which is equivalent to making the Lagrangian multiplier smaller, that is, using a small amount of of depth bits in exchange for a larger synthetic view quality improvement. However, when the quality of the texture video is not good enough, that is, when the texture QP is large, since the quality of the synthesized view is mainly determined by the distorted texture video, no matter how large the depth coding bit is, it cannot bring about a sharp improvement in the quality of the synthesized view, so it should be increased. Grange multipliers to avoid unnecessary bit overhead.

To achieve the above object, the technical scheme of the present invention is as follows:

(1) Before encoding the depth map of the basic viewpoint, the Lagrangian multiplier used for it is corrected:

(1a) According to the quantization parameter Q _t used in the same-viewpoint texture video coding and the quantization parameter Q _d of the depth to be encoded, the correction factor is constructed as follows:

k(Q _t ,Q _d )=2 ^aQt+bQd+c <1>

Among them, a, b, and c are three constant coefficients with different values in the correction factor, a=0.3421, b=-0.2402, c=-4.543;

(1b) Use the correction factor to correct the Lagrangian multipliers used in the depth map encoding rate-distortion optimization, and the corrected Lagrangian multipliers are:

λ′ _depth = k(Q _t , Q _d )·λ _depth <2>

Among them, λ _depth is the Lagrangian multiplier used in the depth map encoding in 3D-HEVC, and its calculation method is:

λ _depth = β·W·2 ^{((Qd-12)/3.0)}

Among them, W is a weighting factor, which is determined by the encoding configuration and the position of the encoded image in the GOP of pictures; β is a scale parameter, and its value depends on whether the current image is used as a reference image. When it is used as a non-reference image, The value is 1.0, and when used as a reference image, the value is 1.0-Clip3( _0.0,0.5,0.05 ·NB ), where NB represents the number of _B -frame reference images in the group of pictures GOP.

(1c) From Equation <1> and Equation <2>, the Lagrange multiplier after correction of the depth map is obtained as:

λ' _depth = β·W·2 ^a'Qt+b'Qd+c'

Among them, a', b', c' are three constant coefficients with different values in the Lagrange multiplier,

(1d) According to the modified Lagrangian multiplier λ′ _depth , the Lagrangian multiplier λ _motion used in motion estimation is obtained:

${\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; ;</span>$

(2) Integrating the revised Lagrangian multiplier λ' _depth into the 3D extended 3D-HEVC reference software of the high-efficiency video coding standard, obtaining the revised 3D extended 3D-HEVC reference software B of the high-efficiency video coding standard;

(3) Encode the 3D video sequence with the modified reference software B.

Compared with the prior art, the present invention has the following advantages:

First, according to the quantization parameter Q _d of the depth image to be coded and the quantization parameter Q _t of the coded texture of the same viewpoint, the present invention constructs a correction factor to correct the Lagrangian multiplier, and use the corrected Lagrangian The multiplier encodes the depth map, overcomes the relationship between the Lagrangian multiplier used in depth encoding and the texture quality of the corresponding viewpoint in the prior art, improves the overall encoding performance of 3D video, and can be used for any texture depth 3D video in QP combination mode is encoded.

Second, encoding different 3D standard test sequences with the modified reference software B, taking into account texture video quality variations, saves on average at the same synthetic view quality compared to the results encoded with the original reference software. 1.3% of the total code rate.

Description of drawings

Figure 1 shows the existing 3D-HEVC coding framework.

Fig. 2 is the realization flowchart of the present invention.

detailed description

The present invention will be described in further detail below in conjunction with accompanying drawings and examples.

With reference to Fig. 2, the Lagrangian multiplier correction method of multi-view depth video coding of the present invention comprises the following steps:

Step 1, determine the relationship between the correction factor of the Lagrangian multiplier of the depth map and the texture quantization parameter Qt and the depth _quantization parameter _Qd .

(1a) Set the correction factor k in the form of 2 ^x , the variable x changes in the range of [-6,-1] at an interval of 0.5, and 11 different correction factors are obtained, and these correction factors are used to modify the high-efficiency video coding standard The Lagrangian multipliers used in the depth coding in the 3D extended 3D-HEVC reference software obtained 11 revised reference software;

(1b) Use the above-mentioned modified reference software to precode 97 frames of two viewpoints of multiple 3D standard test sequences, in which the texture depth QP combinations [Q _t , Q _d ] used are: [23,34], [25,34], [27,34]; [28,39], [30,39], [32,39]; [33,42], [35,42], [37,42]; [38 ,45], [40,45], [42,45];

(1c) Synthesize the multi-viewpoint texture and depth video reconstructed by the decoding end into a virtual viewpoint view between multiple viewpoints using a viewpoint synthesis algorithm, and compare it with the result encoded by the original 3D-HEVC reference software under the same QP combination in the form of BDBR, The k corresponding to the result with the best performance is taken as the optimal k of the QP combination.

The viewpoint synthesis algorithm is based on the depth image rendering DIBR algorithm adopted by the 3D-HEVC standard;

The BDBR form represents the change in the code rate of the video encoded by the modified software relative to the original software under the same objective quality;

(1d) Curve fitting is performed on each texture depth QP combination and its corresponding optimal correction factor k, and the following relationship is obtained:

k=2 ^aQt+bQd+c <1>

Among them, Q _t is the texture quantization parameter, Q _d is the depth quantization parameter, a, b, and c are three constant coefficients with different values in the correction factor, and their values are all obtained from the pre-coding test. The results of different test configuration schemes There will be deviations. In this embodiment, a=0.3421, b=-0.2402, and c=-4.543 are taken.

Step 2, modifying the Lagrangian multipliers used in depth coding.

(2a) Use the correction factor k obtained in step 1 to correct the Lagrangian multipliers used in the depth map coding rate-distortion optimization, and the corrected Lagrangian multipliers are:

λ′ _depth = k·λ _depth <2>

Among them, λ _depth is the Lagrangian multiplier used in the existing 3D-HEVC depth image coding, and its calculation method is:

λ _depth = β·W·2 ^{((Qd-12)/3.0)}

Among them, W is a weighting factor, which is determined by the encoding configuration and the position of the encoded image in the GOP of pictures; β is a scale parameter, and its value depends on whether the current image is used as a reference image. When it is used as a non-reference image, The value is 1.0, when used as a reference image, the value is 1.0-Clip3(0.0,0.5,0.05 N _B ), where N _B represents the number of B frame reference images in the group of pictures GOP;

(2b) From formula <1> and formula <2>, the Lagrangian multiplier after the correction of the depth map is written as the following form:

λ' _depth = β·W·2 ^a'Qt+b'Qd+c'

Among them, a', b', c' are three constant coefficients with different values in the Lagrange multiplier,

(2c) According to the modified Lagrangian multiplier λ′ _depth , the Lagrangian multiplier λ′ _motion used in motion estimation is obtained:

${\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; .</span>$

Step 3: Integrate the modified Lagrange multiplier λ′ _depth into the 3D extended 3D-HEVC reference software HTM13.0 of the high-efficiency video coding standard, and obtain the modified 3D extended 3D-HEVC reference software of the high-efficiency video coding standard software B.

Step 4, use the modified reference software B to encode the 3D video sequence.

Effect of the present invention is further illustrated by following tests:

Test content 1:

Use the revised reference software B to encode the 3D standard test sequence in the 3D-HEVC general test environment CTC, where the texture depth QP combination [Q _t , Q _d ] is [25,34], [30,39], [ 35,42], [40,45]; 3D standard test sequences were encoded with the original reference software HTM13.0 under the same texture depth QP combination.

Comparing the above two encoding results in the form of BDBR, the total bit rate results of encoding texture and depth under the same synthetic view quality are obtained, as shown in Table 2.

The BDBR form indicates the change in the code rate of the result obtained by using the modified software encoding relative to the original software under the synthetic view quality, and the negative sign indicates the code rate savings.

Test content 2:

The 3D standard test sequence is performed under the texture depth QP combination [Q _t , Q _d ]: [23,34], [28,39], [33,42], [38,45] with the modified reference software B Encoding: The 3D standard test sequence was encoded with the original reference software HTM13.0 under the same texture depth QP combination.

Comparing the above two encoding results in the form of BDBR, the change of the total bit rate under the same synthetic view quality is obtained, as shown in Table 2.

Test content 3:

The 3D standard test sequence is performed under the texture depth QP combination [Q _t , Q _d ]: [27,34], [32,39], [37,42], [42,45] with the modified reference software B Encoding: The 3D standard test sequence was encoded with the original reference software HTM13.0 under the same texture depth QP combination.

Comparing the above two encoding results in the form of BDBR, the change of the total bit rate under the same synthetic view quality is obtained, as shown in Table 2.

Test content 4:

The 3D standard test sequence is performed under the texture depth QP combination [Q _t , Q _d ]: [21,34], [26,39], [31,42], [36,45] with the modified reference software B Encoding: The 3D standard test sequence was encoded with the original reference software HTM13.0 under the same texture depth QP combination.

Comparing the above two encoding results in the form of BDBR, the change of the total bit rate under the same synthetic view quality is obtained, as shown in Table 2.

Test content 5:

3D standard test sequence under texture and depth QP combination [Q _t , Q _d ] with revised reference software B: [29,34], [34,39], [39,42], [44,45] Encoding; use the original reference software HTM13.0 to encode the 3D standard test sequence under the same texture depth QP combination.

Comparing the above two encoding results in the form of BDBR, the change of the total bit rate under the same synthetic view quality is obtained, as shown in Table 2.

Table 2 Performance comparison results

It can be seen from Table 2 that for different 3D standard test sequences, under the same synthetic view quality, the average total code rate of test content 1 is basically unchanged, and the results of test content 2 can save 0.6% of the total code rate on average. The results of content 3 can save an average of 0.7% of the total bit rate, the results of test content 4 can save an average of 2.6% of the total bit rate, and the results of test content 5 can save an average of 2.5% of the total bit rate.

The above content is a detailed description of the present invention in conjunction with specific preferred embodiments, but the present invention is not limited to the above embodiments. Within the scope of the knowledge of those skilled in the art, various changes can be made without departing from the idea of the present invention, and all changes should be deemed to belong to the protection scope of the present invention.

Claims (2)

1. A Lagrangian multiplier correction method for multi-view depth video coding, comprising: (1) Before encoding the depth map of multi-viewpoint video, modify the Lagrange multiplier used by it: (1a) According to the quantization parameter Q _t used in the same-viewpoint texture video coding and the quantization parameter Q _d of the depth to be encoded, the correction factor is constructed as follows: k=2 ^aQt+bQd+c <1> Among them, a, b, and c are three constant coefficients with different values in the correction factor, a=0.3421, b=-0.2402, c=-4.543; (1b) Use the correction factor to correct the Lagrangian multipliers used in the depth map encoding rate-distortion optimization, and the corrected Lagrangian multipliers are: λ′ _depth = k·λ _depth <2> Among them, λ _depth is the Lagrangian multiplier used in the existing 3D-HEVC depth image coding, and its calculation method is: λ _depth = β·W·2 ^{((Qd-12)/3.0)} Among them, W is a weighting factor, which is determined by the encoding configuration and the position of the encoded image in the GOP of pictures; β is a scale parameter, and its value depends on whether the current image is used as a reference image. When it is used as a non-reference image, The value is 1.0, when used as a reference image, the value is 1.0-Clip3(0.0,0.5,0.05 N _B ), where N _B represents the number of B frame reference images in the group of pictures GOP; (1c) According to formula <1> and formula <2>, the Lagrangian multiplier after the correction of the depth map is written as the following form: λ' _depth = β·W·2 ^a'Qt+b'Qd+c' Among them, a', b', c' are three constant coefficients with different values in the Lagrangian multiplier, a'=a, c'=c-4; (1d) According to the modified Lagrangian multiplier λ′ _depth , the Lagrangian multiplier λ′ _motion used in motion estimation is obtained: ${\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; ;</span>$ (2) Integrating the revised Lagrangian multiplier λ' _depth into the 3D extended 3D-HEVC reference software of the high-efficiency video coding standard, obtaining the revised 3D extended 3D-HEVC reference software B of the high-efficiency video coding standard; (3) Encode the 3D video sequence with the modified reference software B. 2. The method according to claim 1, wherein in the step (1a), the correction factor is determined by the following steps: (1a1) Setting a plurality of different correction factors, using these correction factors to respectively correct the Lagrangian multipliers used in the depth coding in the 3D extended 3D-HEVC reference software of the High Efficiency Video Coding Standard, to obtain the revised reference software; (1a2) Pre-encode the 3D standard test sequence under different texture depth QP combinations with the above-mentioned modified reference software; (1a3) Synthesize the multi-viewpoint texture and depth video reconstructed by the decoding end into a virtual viewpoint view between multiple viewpoints using a viewpoint synthesis algorithm, and compare it with the original 3D-HEVC reference software encoding results under the same QP combination, and the performance is the best The correction factor k corresponding to the result of is used as the optimal correction factor for this QP combination. (1a4) Carry out curve fitting for each texture depth QP combination and its corresponding optimal correction factor _k , and obtain the relationship between the correction factor and texture quantization parameter Qt and depth quantization parameter _Qd : k=2 ^aQt+bQd+c .