CN103200405A - 3DV video encoding method and encoder - Google Patents
3DV video encoding method and encoder Download PDFInfo
- Publication number
- CN103200405A CN103200405A CN2013101163374A CN201310116337A CN103200405A CN 103200405 A CN103200405 A CN 103200405A CN 2013101163374 A CN2013101163374 A CN 2013101163374A CN 201310116337 A CN201310116337 A CN 201310116337A CN 103200405 A CN103200405 A CN 103200405A
- Authority
- CN
- China
- Prior art keywords
- viewpoint
- map
- code stream
- depth map
- base layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
本发明提出一种3DV视频编码方法及编码器,其中,该方法包括:对最左视点和最右视点彩色图和深度图进行基本层3DV编码操作得到基本层码流;对基本层码流进解码得到最左视点和最右视点彩色图重构和深度图重构;根据最左视点和最右视点彩色图重构和深度图重构进行视点合成,得到至少一个合成中间视点彩色图和深度图;将至少一个中间视点彩色图和深度图,与至少一个合成中间视点彩色图和深度图做比较,计算得到至少一个中间视点彩色图残差和深度图残差;对至少一个中间视点彩色图残差和深度图残差进行加强层3DV编码操作,得到加强层码流;将基本层码流和加强层码流合并为可伸缩码流进行发送。本发明具有编码效率高、编码质量好的优点。
The present invention proposes a 3DV video encoding method and an encoder, wherein the method includes: performing a basic layer 3DV encoding operation on the leftmost and rightmost viewpoint color images and depth maps to obtain a basic layer code stream; Decoding to obtain the leftmost and rightmost viewpoint color image reconstruction and depth image reconstruction; perform viewpoint synthesis according to the leftmost and rightmost viewpoint color image reconstruction and depth image reconstruction, and obtain at least one synthetic intermediate viewpoint color image and depth Figure; at least one intermediate viewpoint color map and depth map are compared with at least one composite intermediate viewpoint color map and depth map, and at least one intermediate viewpoint color map residual and depth map residual are calculated; for at least one intermediate viewpoint color map The residual and the depth map residual are subjected to an enhancement layer 3DV encoding operation to obtain an enhancement layer code stream; the base layer code stream and the enhancement layer code stream are combined into a scalable code stream for transmission. The invention has the advantages of high coding efficiency and good coding quality.
Description
技术领域technical field
本发明属于视频技术领域,具体涉及一种3DV视频编码方法及编码器。The invention belongs to the field of video technology, and in particular relates to a 3DV video encoding method and an encoder.
背景技术Background technique
3DV是目前国际上正在制定的新一代立体视频编码标准。3DV使用多个视点的彩色图加上对应的深度图的视频格式,通过视角合成的办法可以获取其间任意视角的彩色图。这种编码方法在尽量节省码率的情况下,可以获得任意视点的视图。3DV is a new generation of stereoscopic video coding standards being developed internationally. 3DV uses the video format of the color image of multiple viewpoints plus the corresponding depth image, and the color image of any viewing angle can be obtained through the method of viewing angle synthesis. This encoding method can obtain views from any viewpoint while saving the bit rate as much as possible.
在实际应用中,自然场景的深度图获取方式可以分为主动获取和被动获取两种。主动获取方法指的是通过深度相机等硬件设备对场景中的三维深度信息进行直接采集。这种方法可以较快获取深度信息,但是受到采集技术的限制,分辨率一般不高。被动获取方法指的是通过立体匹配等计算方式获得深度图序列,可以得到分辨率较高的稠密深度图序列。但是多个视点的深度图序列数据量非常庞大,为了降低处理数据量,通常都对多视点深度图序列进行下采样,然后对小分辨率的深度图序列进行压缩和传输。待解码后,对重建序列进行上采样,进而恢复成为高分辨率的图像。In practical applications, the depth map acquisition methods of natural scenes can be divided into two types: active acquisition and passive acquisition. The active acquisition method refers to the direct acquisition of 3D depth information in the scene through hardware devices such as depth cameras. This method can obtain depth information quickly, but limited by the acquisition technology, the resolution is generally not high. The passive acquisition method refers to obtaining a sequence of depth maps through calculation methods such as stereo matching, and can obtain a sequence of dense depth maps with high resolution. However, the data volume of the depth map sequence of multiple viewpoints is very large. In order to reduce the amount of processed data, the multi-view depth map sequence is usually down-sampled, and then the small-resolution depth map sequence is compressed and transmitted. After being decoded, the reconstructed sequence is up-sampled to restore a high-resolution image.
可伸缩编码是为了使视频编码适应不同的质量要求的一种视频编码方法。可伸缩编码一般将视频编码分为基本层与加强层,当对视频质量要求不高时只传输基本层的内容,由此可以节约很多的码率,当需要高质量视频的时候在额外传输加强层,使用户在视频质量与带宽的选择上有一个很大的自由度。Scalable coding is a video coding method to adapt video coding to different quality requirements. Scalable coding generally divides video coding into basic layer and enhanced layer. When the video quality is not high, only the content of the basic layer is transmitted, which can save a lot of bit rate. When high-quality video is required, additional transmission is enhanced. Layer, so that users have a great freedom in the choice of video quality and bandwidth.
3DV由于其特别的视频格式,传统的针对彩色图的可伸缩编码方法不太使用。需要一种新的方法来解决3DV视频的可伸缩问题。For 3DV, due to its special video format, traditional scalable coding methods for color images are not used much. A new approach is needed to solve the scalability problem of 3DV video.
发明内容Contents of the invention
本发明旨在至少在一定程度上解决上述技术问题之一或至少提供一种有用的商业选择。为此,本发明的一个目的在于提出一种3DV视频编码方法,通过本发明的3DV视频编码方法,可以解决3DV视频的可伸缩的问题。The present invention aims at solving one of the above technical problems at least to a certain extent or at least providing a useful commercial choice. Therefore, an object of the present invention is to provide a 3DV video coding method, through which the 3DV video coding method of the present invention can solve the scalability problem of 3DV video.
为了实现上述目的,根据本发明实施例的3DV视频编码方法,其中,待编码的所述3DV视频包括最左视点、最右视点和至少一个中间视点彩色图和深度图,包括:S1.对最左视点和最右视点彩色图和深度图进行基本层3DV编码操作,得到基本层码流;S2.对所述基本层码流进行基本层3DV解码操作,得到最左视点和最右视点彩色图重构和深度图重构;S3.按照原始的所述至少一个中间视点的位置,根据所述最左视点和最右视点彩色图重构和深度图重构进行视点合成,得到至少一个合成中间视点彩色图和深度图;S4.将所述至少一个中间视点彩色图和深度图,与所述至少一个合成中间视点彩色图和深度图做比较,计算得到至少一个中间视点彩色图残差和深度图残差;S5.对所述至少一个中间视点彩色图残差和深度图残差进行加强层3DV编码操作,得到加强层码流;S6.将所述基本层码流和所述加强层码流合并为可伸缩码流进行发送。In order to achieve the above object, according to the 3DV video coding method of the embodiment of the present invention, wherein the 3DV video to be coded includes the leftmost viewpoint, the rightmost viewpoint and at least one intermediate viewpoint color map and depth map, including: S1. Perform the base layer 3DV encoding operation on the left viewpoint and the rightmost viewpoint color map and the depth map to obtain the base layer code stream; S2. Perform the base layer 3DV decoding operation on the base layer code stream to obtain the leftmost viewpoint and the rightmost viewpoint color map Reconstruction and depth map reconstruction; S3. According to the original position of the at least one intermediate viewpoint, perform viewpoint synthesis according to the leftmost and rightmost viewpoint color map reconstruction and depth map reconstruction to obtain at least one composite intermediate Viewpoint color map and depth map; S4. Comparing the at least one intermediate viewpoint color map and depth map with the at least one composite intermediate viewpoint color map and depth map, and calculating at least one intermediate viewpoint color map residual and depth Image residual; S5. Perform an enhancement layer 3DV encoding operation on the at least one intermediate viewpoint color image residual and depth image residual to obtain an enhancement layer code stream; S6. Combine the base layer code stream and the enhancement layer code The streams are combined into a scalable code stream for transmission.
在本发明的一个实施例中,所述基本层3DV编码操作为独立编码操作或非独立编码操作。In an embodiment of the present invention, the base layer 3DV encoding operation is an independent encoding operation or a non-independent encoding operation.
在本发明的一个实施例中,所述加强层3DV编码操作为独立编码操作。In one embodiment of the present invention, the enhancement layer 3DV encoding operation is an independent encoding operation.
根据本发明的3DV视频编码方法,可以节约码率,当需要高质量视频的时候在额外传输加强层,使用户在视频质量与带宽的选择上有一个很大的自由度。According to the 3DV video encoding method of the present invention, the code rate can be saved, and when high-quality video is required, an additional enhancement layer is transmitted, so that the user has a large degree of freedom in the selection of video quality and bandwidth.
本发明的另一个目的在于提出一种3DV视频编码器,通过本发明的3DV视频编码器,可以解决3DV视频的可伸缩的问题。Another object of the present invention is to provide a 3DV video encoder, through which the 3DV video encoder can solve the scalability problem of 3DV video.
为了实现上述目的,根据本发明实施例的3DV视频编码器,其中,待编码的所述3DV视频包括最左视点、最右视点和至少一个中间视点彩色图和深度图,包括:基本层3DV编码模块,所述基本层3DV编码模块对最左视点和最右视点彩色图和深度图进行基本层3DV编码操作,得到基本层码流;基本层3DV解码模块,所述基本层3DV解码模块与所述基本层3DV编码模块相连,对所述基本层码流进行基本层3DV解码操作,得到最左视点和最右视点彩色图重构和深度图重构;视点合成模块,所述视点合成模块与所述基本层3DV解码模块相连,按照原始的所述至少一个中间视点的位置,根据所述最左视点和最右视点彩色图重构和深度图重构进行视点合成,得到至少一个合成中间视点彩色图和深度图;比较模块,所述比较模块与所述视点合成模块相连,将所述至少一个中间视点彩色图和深度图,与所述至少一个合成中间视点彩色图和深度图做比较,计算得到至少一个中间视点彩色图残差和深度图残差;加强层3DV编码模块,所述加强层3DV编码模块与所述比较模块相连,对所述至少一个中间视点彩色图残差和深度图残差进行加强层3DV编码操作,得到加强层码流;发送模块,所述发送模块与所述加强层3DV编码模块相连,将所述基本层码流和所述加强层码流合并为可伸缩码流进行发送。In order to achieve the above object, according to the 3DV video encoder of the embodiment of the present invention, the 3DV video to be encoded includes the leftmost viewpoint, the rightmost viewpoint and at least one intermediate viewpoint color map and depth map, including: basic layer 3DV coding module, the base layer 3DV encoding module performs the base layer 3DV encoding operation on the leftmost and rightmost viewpoint color images and depth maps to obtain the base layer code stream; the base layer 3DV decoding module, the base layer 3DV decoding module and the base layer 3DV decoding module The base layer 3DV encoding module is connected, and the base layer 3DV decoding operation is performed on the base layer code stream to obtain the reconstruction of the leftmost viewpoint and the rightmost viewpoint color map and depth map reconstruction; the viewpoint synthesis module, the viewpoint synthesis module and The base layer 3DV decoding modules are connected to perform viewpoint synthesis according to the original position of the at least one intermediate viewpoint, according to the reconstruction of the leftmost viewpoint and the rightmost viewpoint color map and depth map reconstruction, to obtain at least one synthesized intermediate viewpoint A color map and a depth map; a comparison module, the comparison module is connected to the viewpoint synthesis module, and compares the at least one intermediate viewpoint color map and depth map with the at least one synthesized intermediate viewpoint color map and depth map, Calculate and obtain at least one intermediate viewpoint color map residual and depth map residual; an enhancement layer 3DV encoding module, the enhancement layer 3DV encoding module is connected to the comparison module, and calculates the at least one intermediate viewpoint color image residual and depth map Residuals carry out the enhanced layer 3DV encoding operation to obtain the enhanced layer code stream; the sending module, the sending module is connected with the enhanced layer 3DV encoding module, and merges the basic layer code stream and the enhanced layer code stream into a scalable stream to send.
在本发明的一个实施例中,所述基本层3DV编码操作为独立编码操作或非独立编码操作。In an embodiment of the present invention, the base layer 3DV encoding operation is an independent encoding operation or a non-independent encoding operation.
在本发明的一个实施例中,所述加强层3DV编码操作为独立编码操作。In one embodiment of the present invention, the enhancement layer 3DV encoding operation is an independent encoding operation.
根据本发明的3DV视频编码器,可以节约码率,当需要高质量视频的时候在额外传输加强层,使用户在视频质量与带宽的选择上有一个很大的自由度。According to the 3DV video encoder of the present invention, the code rate can be saved, and when high-quality video is required, an additional enhancement layer is transmitted, so that the user has a large degree of freedom in the selection of video quality and bandwidth.
本发明的附加方面和优点将在下面的描述中部分给出,部分将从下面的描述中变得明显,或通过本发明的实践了解到。Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
附图说明Description of drawings
本发明的上述和/或附加的方面和优点从结合下面附图对实施例的描述中将变得明显和容易理解,其中:The above and/or additional aspects and advantages of the present invention will become apparent and comprehensible from the description of the embodiments in conjunction with the following drawings, wherein:
图1是根据本发明实施例的3DV视频编码方法的原理图;Fig. 1 is a schematic diagram of a 3DV video coding method according to an embodiment of the present invention;
图2是根据本发明实施例的3DV视频编码方法的流程图;Fig. 2 is the flowchart of the 3DV video coding method according to the embodiment of the present invention;
图3是根据本发明实施例的3DV视频编码器的结构图。FIG. 3 is a structural diagram of a 3DV video encoder according to an embodiment of the present invention.
具体实施方式Detailed ways
下面详细描述本发明的实施例,所述实施例的示例在附图中示出,其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的,旨在用于解释本发明,而不能理解为对本发明的限制。Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.
在本发明的描述中,需要理解的是,术语“中心”、“纵向”、“横向”、“长度”、“宽度”、“厚度”、“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”“内”、“外”、“顺时针”、“逆时针”等指示的方位或位置关系为基于附图所示的方位或位置关系,仅是为了便于描述本发明和简化描述,而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作,因此不能理解为对本发明的限制。In describing the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", " Orientation or position indicated by "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. The relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, therefore It should not be construed as a limitation of the present invention.
此外,术语“第一”、“第二”仅用于描述目的,而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此,限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本发明的描述中,“多个”的含义是两个或两个以上,除非另有明确具体的限定。In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.
在本发明中,除非另有明确的规定和限定,术语“安装”、“相连”、“连接”、“固定”等术语应做广义理解,例如,可以是固定连接,也可以是可拆卸连接,或一体地连接;可以是机械连接,也可以是电连接;可以是直接相连,也可以通过中间媒介间接相连,可以是两个元件内部的连通。对于本领域的普通技术人员而言,可以根据具体情况理解上述术语在本发明中的具体含义。In the present invention, unless otherwise clearly specified and limited, terms such as "installation", "connection", "connection" and "fixation" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection , or integrally connected; it may be mechanically connected or electrically connected; it may be directly connected or indirectly connected through an intermediary, and it may be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations.
在本发明中,除非另有明确的规定和限定,第一特征在第二特征之“上”或之“下”可以包括第一和第二特征直接接触,也可以包括第一和第二特征不是直接接触而是通过它们之间的另外的特征接触。而且,第一特征在第二特征“之上”、“上方”和“上面”包括第一特征在第二特征正上方和斜上方,或仅仅表示第一特征水平高度高于第二特征。第一特征在第二特征“之下”、“下方”和“下面”包括第一特征在第二特征正下方和斜下方,或仅仅表示第一特征水平高度小于第二特征。In the present invention, unless otherwise clearly specified and limited, a first feature being "on" or "under" a second feature may include direct contact between the first and second features, and may also include the first and second features Not in direct contact but through another characteristic contact between them. Moreover, "above", "above" and "above" the first feature on the second feature include that the first feature is directly above and obliquely above the second feature, or simply means that the first feature is horizontally higher than the second feature. "Below", "beneath" and "under" the first feature to the second feature include that the first feature is directly below and obliquely below the second feature, or simply means that the first feature has a lower level than the second feature.
如图1所示,为根据本发明实施例的3DV视频编码方法的原理图。As shown in FIG. 1 , it is a schematic diagram of a 3DV video coding method according to an embodiment of the present invention.
在图1中,最左视点彩色图为PL,最左视点深度图为DL,最左视点彩色图重构为P′L,最左视点深度图重构为D′L,最右视点彩色图为PR,最右视点深度图为DR,最右视点彩色图重构为P′R,最右视点深度图重构为D′R,中间视点彩色图为PM1、PM2……,中间视点深度图为DM1、DM2……,合成中间视点彩色图为P′M1、P′M2……,合成中间视点深度图为D′M1、D′M2……,中间视点彩色图残差为RPM1、RPM2……,中间视点深度图残差为RDM1、RDM2……。In Fig. 1, the color map of the leftmost viewpoint is P L , the depth map of the leftmost viewpoint is D L , the color map of the leftmost viewpoint is reconstructed as P′ L , the depth map of the leftmost viewpoint is reconstructed as D′ L , and the depth map of the most right viewpoint is reconstructed as D′ L . The color image is P R , the rightmost viewpoint depth image is D R , the rightmost viewpoint color image is reconstructed as P′ R , the rightmost viewpoint depth image is reconstructed as D′ R , the middle viewpoint color image is P M1 , PM2 ... ..., the intermediate viewpoint depth maps are D M1 , D M2 ..., the composite intermediate viewpoint color images are P′ M1 , P′ M2 ..., the composite intermediate viewpoint depth maps are D′ M1 , D′ M2 ..., the intermediate viewpoint color The map residuals are RP M1 , RP M2 . . . , and the mid-view depth map residuals are RD M1 , RD M2 . . .
如图2所示,为根据本发明实施例的3DV视频编码方法的流程图。根据本发明实施例的3DV视频编码方法的步骤包括:As shown in FIG. 2 , it is a flowchart of a 3DV video coding method according to an embodiment of the present invention. The steps of the 3DV video encoding method according to an embodiment of the present invention include:
S1.对最左视点和最右视点彩色图和深度图进行基本层3DV编码操作,得到基本层码流。S1. Perform base layer 3DV encoding operation on the leftmost and rightmost view color images and depth images to obtain base layer code streams.
其中,基本层3DV编码操作可以是独立编码操作或者非独立的编码操作。最常见的编码方法可以参照正在制定的MEPG(Motion Picture Experts Group,运动图像专家组)3DV相关标准。Wherein, the base layer 3DV encoding operation may be an independent encoding operation or a non-independent encoding operation. The most common encoding method can refer to the MEPG (Motion Picture Experts Group, Motion Picture Experts Group) 3DV related standards being developed.
具体的,对最左视点和最右视点彩色图直接进行基本层3DV编码操作,对最左视点和最右视点深度图下采样至彩色图的一半宽度和一半高度进行基本层3DV编码操作,这样可以节约编码时开销以及缩减编码结果。需要说明的是,取值一半仅为示例,也可根据实际情况(特别是硬件条件)取三分之一或四分之一等其他数值。Specifically, the base layer 3DV coding operation is directly performed on the leftmost and rightmost color images, and the base layer 3DV encoding operation is performed on the leftmost and rightmost depth images downsampled to half the width and half height of the color image. It can save encoding overhead and reduce encoding results. It should be noted that half of the value is only an example, and other values such as one-third or one-fourth may also be taken according to actual conditions (especially hardware conditions).
S2.对基本层码流进行基本层3DV解码操作,得到最左视点和最右视点彩色图重构和深度图重构。S2. Perform a base layer 3DV decoding operation on the base layer code stream to obtain the reconstruction of the color map and depth map of the leftmost and rightmost viewpoints.
需要说明的是,此时的基本层3DV解码操作应该与前面步骤S1中的基本层3DV编码操作相对应。It should be noted that the base layer 3DV decoding operation at this time should correspond to the base layer 3DV encoding operation in step S1 above.
S3.按照原始的至少一个中间视点的位置,根据最左视点和最右视点彩色图重构和深度图重构进行视点合成,得到至少一个合成中间视点彩色图和深度图。S3. According to the original position of at least one intermediate viewpoint, perform viewpoint synthesis according to the leftmost and rightmost viewpoint color map reconstruction and depth map reconstruction to obtain at least one synthesized intermediate viewpoint color map and depth map.
其中,最左视点彩色图重构P′L和最左视点深度图重构D′L以及最右视点彩色图重构P′R和最右视点深度图重构D′R用于合成中间视点彩色图P′M1、P′M2……和合成中间视点深度图D′M1、D′M2……。使用现有的视点合成技术,例如MPEG3DV的视点合成参考软件VSRS,合成中间视点彩色图P′M1、P′M2……和合成中间视点深度图D′M1、D′M2……。需要说明的是,新合成的中间视点的位置应该与原始的中间视点位置相对应。Among them, the leftmost viewpoint color map reconstruction P′ L and the leftmost viewpoint depth map reconstruction D′ L , the rightmost viewpoint color map reconstruction P′ R and the rightmost viewpoint depth map reconstruction D′ R are used to synthesize the middle viewpoint Color maps P′ M1 , P′ M2 . . . and synthetic mid-view depth maps D′ M1 , D′ M2 . . . Use the existing view synthesis technology, such as the view synthesis reference software VSRS of MPEG3DV, to synthesize the intermediate viewpoint color maps P′ M1 , P′ M2 . . . and synthesize the intermediate viewpoint depth maps D′ M1 , D′ M2 . . . It should be noted that the position of the newly synthesized intermediate view should correspond to the position of the original intermediate view.
S4.将至少一个中间视点彩色图和深度图,与至少一个合成中间视点彩色图和深度图做比较,计算得到至少一个中间视点彩色图残差和深度图残差。S4. Comparing at least one intermediate viewpoint color map and depth map with at least one synthesized intermediate viewpoint color map and depth map, and calculating at least one intermediate viewpoint color map residual and depth map residual.
由于受到视点合成技术的限制,得到的合成中间视点的质量相对于原始中间视点来说较差。为了提高中间视点的质量,用原始中间视点彩色图PM1、PM2……与原始中间视点深度图DM1,DM2……减去合成中间视点彩色图P′M1、P′M2……与合成中间视点深度图D′M1、D′M2……,得到中间视点彩色图残差RPM1,RPM2……与中间视点深度图残差RDM1,RDM2……,即合成视点与原始视点的差距。Due to the limitations of the view synthesis technique, the quality of the resulting synthesized intermediate view is inferior to that of the original intermediate view. In order to improve the quality of the intermediate viewpoint, the original intermediate viewpoint color maps P M1 , P M2 ... and the original intermediate viewpoint depth maps D M1 , D M2 ... are subtracted from the synthetic intermediate viewpoint color maps P′ M1 , P′ M2 ... and Synthesize intermediate viewpoint depth maps D′ M1 , D′ M2 ...... to obtain intermediate viewpoint color map residuals RP M1 , RP M2 ...... and intermediate viewpoint depth map residuals RD M1 , RD M2 . difference.
可以预见的是,在解码端,通过对加强层码流的解码,可以得到中间视点彩色图残差重构与深度图残差重构,将其加到通过解码基本层码流合成中间视点重构上,可以提高最终得到的中间视点的解码结果的质量。It is foreseeable that at the decoding end, by decoding the code stream of the enhancement layer, the residual reconstruction of the intermediate viewpoint color map and the residual reconstruction of the depth map can be obtained, which are added to the intermediate viewpoint reconstruction by decoding the code stream of the basic layer. Structurally, the quality of the finally obtained intermediate viewpoint decoding result can be improved.
S5.对至少一个中间视点彩色图残差和深度图残差进行加强层3DV编码操作,得到加强层码流。S5. Perform an enhancement layer 3DV encoding operation on at least one intermediate viewpoint color map residual and depth map residual to obtain an enhancement layer code stream.
具体的,将得到的中间视点彩色图残差RPM1、RPM2……与中间视点深度图残差RDM1、RDM2……用现有3DV编码框架独立编码,即不采用其他视点的重构作为参考帧来进行编码,作为加强层。选择残差作为加强层的原因是残差所包含的信息量较小,相对原始中间视点来说可以得到更高的压缩效率。Specifically, the obtained intermediate viewpoint color map residuals RP M1 , RP M2 ... and intermediate viewpoint depth map residuals RD M1 , RD M2 ... are independently coded with the existing 3DV coding framework, that is, reconstruction of other viewpoints is not used Coded as a reference frame, as an enhancement layer. The reason for choosing the residual as the enhancement layer is that the amount of information contained in the residual is small, and higher compression efficiency can be obtained compared with the original intermediate view.
S6.将基本层码流和加强层码流合并为可伸缩码流进行发送。S6. Combine the base layer code stream and the enhancement layer code stream into a scalable code stream and send it.
从本发明的实施例中可以看出,可伸缩码流将视频编码分为基本层码流与加强层码流,当对视频质量要求不高时只传输基本层码流,由此可以节约码率。当需要高质量视频的时候,额外传输独立编码的加强层码流,根据视频质量的需求,可以选择传输全部加强层码流,也可以选择只传输其中的一部分,通过对传输的加强层码流的解码,得到中间视点的彩色图残差重构与深度图残差重构,将其加到通过解码基本层码流合成中间视点上,使用户在视频质量与带宽的选择上有很大的自由度。As can be seen from the embodiments of the present invention, the scalable code stream divides video encoding into a base layer code stream and an enhanced layer code stream, and only the base layer code stream is transmitted when the video quality is not high, thereby saving code. Rate. When high-quality video is required, additionally transmit an independently coded enhancement layer code stream. According to the video quality requirements, you can choose to transmit all the enhancement layer code streams, or you can choose to transmit only a part of them. Through the transmission of the enhancement layer code stream Decoding, get the color image residual reconstruction and depth image residual reconstruction of the intermediate viewpoint, add it to the intermediate viewpoint synthesized by decoding the basic layer code stream, so that the user has a great choice in video quality and bandwidth degrees of freedom.
如图3所示,为根据本发明实施例的3DV视频编码器的结构图,其中,待编码的3DV视频包括最左视点、最右视点和至少一个中间视点彩色图和深度图,3DV视频编码器包括:基本层3DV编码模块100、基本层3DV解码模块200、视点合成模块300、比较模块400、加强层3DV编码模块500和发送模块600。As shown in Figure 3, it is a structural diagram of a 3DV video encoder according to an embodiment of the present invention, wherein the 3DV video to be encoded includes the leftmost viewpoint, the rightmost viewpoint, and at least one intermediate viewpoint color map and depth map, 3DV video coding The device includes: a base layer 3DV encoding module 100 , a base layer 3DV decoding module 200 , a view synthesis module 300 , a
基本层3DV编码模块100对最左视点和最右视点彩色图和深度图进行基本层3DV编码操作,得到基本层码流。The base layer 3DV encoding module 100 performs a base layer 3DV encoding operation on the leftmost and rightmost view color images and depth images to obtain a base layer code stream.
其中,具体的3DV编码操作可以为独立编码操作,也可以为非独立的编码操作。例如,可以参照正在制定的MEPG(Motion Picture Experts Group,运动图像专家组)3DV相关标准。Wherein, the specific 3DV encoding operation may be an independent encoding operation or a non-independent encoding operation. For example, you can refer to the MEPG (Motion Picture Experts Group, Motion Picture Experts Group) 3DV related standards that are being formulated.
具体的,基本层3DV编码模块100对最左视点和最右视点彩色图直接进行基本层3DV编码操作,对最左视点和最右视点深度图下采样至彩色图的一半宽度和一半高度进行基本层3DV编码操作,这样可以节约编码时开销以及缩减编码结果。需要说明的是,取值一半仅为示例,也可根据实际情况(特别是硬件条件)取三分之一或四分之一等其他数值。Specifically, the base layer 3DV encoding module 100 directly performs the base layer 3DV encoding operation on the leftmost and rightmost color images, and downsamples the leftmost and rightmost depth images to half the width and half height of the color image. Layer 3DV encoding operation, which can save encoding overhead and reduce encoding results. It should be noted that half of the value is only an example, and other values such as one-third or one-fourth may also be taken according to actual conditions (especially hardware conditions).
基本层3DV解码模块200与基本层3DV编码模块100相连,对基本层码流进行基本层3DV解码操作,得到最左视点和最右视点彩色图重构和深度图重构。The base layer 3DV decoding module 200 is connected with the base layer 3DV encoding module 100, and performs base layer 3DV decoding operation on the base layer code stream to obtain the reconstruction of the leftmost and rightmost color map and depth map.
需要说明的是,此时的基本层3DV解码操作应该与前面步骤S1中的基本层3DV编码操作相对应。It should be noted that the base layer 3DV decoding operation at this time should correspond to the base layer 3DV encoding operation in step S1 above.
视点合成模块300与基本层3DV解码模块200相连,按照原始的至少一个中间视点的位置,根据最左视点和最右视点彩色图重构和深度图重构进行视点合成,得到至少一个合成中间视点彩色图和深度图。The viewpoint synthesis module 300 is connected to the base layer 3DV decoding module 200, and according to the original position of at least one intermediate viewpoint, the viewpoint is synthesized according to the reconstruction of the leftmost viewpoint and the rightmost viewpoint color image and the reconstruction of the depth map to obtain at least one synthetic intermediate viewpoint Color and depth maps.
其中,基本层3DV解码模块200解码获得的最左视点彩色图重构P′L和最左视点深度图重构D′L以及最右视点彩色图重构P′R和最右视点深度图重构D′R用于合成中间视点彩色图P′M1、P′M2……和合成中间视点深度图D′M1、D′M2……。使用现有的视点合成技术,例如MPEG3DV的视点合成参考软件VSRS,视点合成模块300合成中间视点彩色图P′M1、P′M2……和合成中间视点深度图D′M1、D′M2……。需要说明的是,新合成的中间视点的位置应该与原始的中间视点位置相对应。Among them, the base layer 3DV decoding module 200 decodes the obtained leftmost viewpoint color map reconstruction P′ L and leftmost viewpoint depth map reconstruction D′ L and the rightmost viewpoint color map reconstruction P′ R and rightmost viewpoint depth map reconstruction The structure D′ R is used to synthesize the intermediate viewpoint color maps P′ M1 , P′ M2 . . . and the synthesized intermediate viewpoint depth maps D′ M1 , D′ M2 . . . Using existing view synthesis technology, such as the view synthesis reference software VSRS of MPEG3DV, the view synthesis module 300 synthesizes intermediate viewpoint color maps P′ M1 , P′ M2 . . . and synthesizes intermediate viewpoint depth maps D′ M1 , D′ M2 . . . . It should be noted that the position of the newly synthesized intermediate view should correspond to the position of the original intermediate view.
比较模块400与视点合成模块300相连,将至少一个中间视点彩色图和深度图,与至少一个合成中间视点彩色图和深度图做比较,计算得到至少一个中间视点彩色图残差和深度图残差。The
由于受到视点合成技术的限制,得到的合成中间视点的质量相对于原始中间视点来说较差。为了提高中间视点的质量,用原始中间视点彩色图PM1,PM2……与原始中间视点深度图DM1,DM2……减去合成中间视点彩色图P′M1、P′M2……与合成中间视点深度图D′M1、D′M2……,得到中间视点彩色图残差RPM1,RPM2……与中间视点深度图残差RDM1,RDM2……,即合成视点与原始视点的差距。Due to the limitations of the view synthesis technique, the quality of the resulting synthesized intermediate view is inferior to that of the original intermediate view. In order to improve the quality of the intermediate viewpoint, the original intermediate viewpoint color map P M1 , P M2 ... and the original intermediate viewpoint depth map D M1 , D M2 ... are subtracted from the synthetic intermediate viewpoint color map P′ M1 , P′ M2 ... and Synthesize intermediate viewpoint depth maps D′ M1 , D′ M2 ...... to obtain intermediate viewpoint color map residuals RP M1 , RP M2 ...... and intermediate viewpoint depth map residuals RD M1 , RD M2 . difference.
可以预见的是,在解码端,通过对加强层码流的解码,可以得到中间视点彩色图残差重构与深度图残差重构,将其加到通过解码基本层码流合成中间视点重构上,可以提高最终得到的中间视点的解码结果的质量。It is foreseeable that at the decoding end, by decoding the code stream of the enhancement layer, the residual reconstruction of the intermediate viewpoint color map and the residual reconstruction of the depth map can be obtained, which are added to the intermediate viewpoint reconstruction by decoding the code stream of the basic layer. Structurally, the quality of the finally obtained intermediate viewpoint decoding result can be improved.
加强层3DV编码模块500与比较模块400相连,对至少一个中间视点彩色图残差和深度图残差进行加强层3DV编码操作,得到加强层码流。The enhancement layer
具体的,加强层3DV编码模块500将得到的中间视点彩色图残差RPM1,RPM2……与中间视点深度图残差RDM1,RDM2……用现有3DV编码框架独立编码,即不采用其他视点的重构作为参考帧来进行编码,作为加强层。选择残差作为加强层的原因是残差所包含的信息量较小,相对原始中间视点来说可以得到更高的压缩效率。Specifically, the enhancement layer
发送模块600与加强层3DV编码模块500相连,将基本层码流和加强层码流合并为可伸缩码流进行发送。The sending
从本发明的实施例中可以看出,可伸缩码流将视频编码分为基本层码流与加强层码流,当对视频质量要求不高时只传输基本层码流,由此可以节约码率。当需要高质量视频的时候,额外传输独立编码的加强层码流,根据视频质量的需求,可以选择传输全部加强层码流,也可以选择只传输其中的一部分,通过对传输的加强层码流的解码,得到中间视点的彩色图残差重构与深度图残差重构,将其加到通过解码基本层码流合成中间视点上,使用户在视频质量与带宽的选择上有很大的自由度。As can be seen from the embodiments of the present invention, the scalable code stream divides video encoding into a base layer code stream and an enhanced layer code stream, and only the base layer code stream is transmitted when the video quality is not high, thereby saving code. Rate. When high-quality video is required, additionally transmit an independently coded enhancement layer code stream. According to the video quality requirements, you can choose to transmit all the enhancement layer code streams, or you can choose to transmit only a part of them. Through the transmission of the enhancement layer code stream Decoding, get the color image residual reconstruction and depth image residual reconstruction of the intermediate viewpoint, add it to the intermediate viewpoint synthesized by decoding the basic layer code stream, so that the user has a great choice in video quality and bandwidth degrees of freedom.
需要说明的是,流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为,表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分,并且本发明的优选实施方式的范围包括另外的实现,其中可以不按所示出或讨论的顺序,包括根据所涉及的功能按基本同时的方式或按相反的顺序,来执行功能,这应被本发明的实施例所属技术领域的技术人员所理解。It should be noted that any process or method descriptions described in flowcharts or otherwise described herein can be understood as representing codes including one or more steps of executable instructions for implementing specific logical functions or processes. modules, segments or parts, and the scope of the preferred embodiments of the present invention includes further implementations, which may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved. function, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
在本说明书的描述中,参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中,对上述术语的示意性表述不一定指的是相同的实施例或示例。而且,描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
尽管上面已经示出和描述了本发明的实施例,可以理解的是,上述实施例是示例性的,不能理解为对本发明的限制,本领域的普通技术人员在不脱离本发明的原理和宗旨的情况下在本发明的范围内可以对上述实施例进行变化、修改、替换和变型。Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be construed as limitations to the present invention. Variations, modifications, substitutions, and modifications to the above-described embodiments are possible within the scope of the present invention.
Claims (6)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310116337.4A CN103200405B (en) | 2013-04-03 | 2013-04-03 | A kind of 3DV method for video coding and encoder |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310116337.4A CN103200405B (en) | 2013-04-03 | 2013-04-03 | A kind of 3DV method for video coding and encoder |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103200405A true CN103200405A (en) | 2013-07-10 |
| CN103200405B CN103200405B (en) | 2016-06-01 |
Family
ID=48722746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310116337.4A Expired - Fee Related CN103200405B (en) | 2013-04-03 | 2013-04-03 | A kind of 3DV method for video coding and encoder |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103200405B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105723705A (en) * | 2013-11-20 | 2016-06-29 | 皇家飞利浦有限公司 | Generation of images for an autosteroscopic multi-view display |
| CN105934948A (en) * | 2013-12-13 | 2016-09-07 | 高通股份有限公司 | Signaling of simplified depth coding (SDC) for depth intra-and inter-prediction modes in 3D video coding |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100046612A1 (en) * | 2008-08-25 | 2010-02-25 | Microsoft Corporation | Conversion operations in scalable video encoding and decoding |
| CN102438167A (en) * | 2011-10-21 | 2012-05-02 | 宁波大学 | Three-dimensional video encoding method based on depth image rendering |
| CN102934450A (en) * | 2010-05-13 | 2013-02-13 | 索尼公司 | Image processing device and image processing method |
-
2013
- 2013-04-03 CN CN201310116337.4A patent/CN103200405B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100046612A1 (en) * | 2008-08-25 | 2010-02-25 | Microsoft Corporation | Conversion operations in scalable video encoding and decoding |
| CN102934450A (en) * | 2010-05-13 | 2013-02-13 | 索尼公司 | Image processing device and image processing method |
| CN102438167A (en) * | 2011-10-21 | 2012-05-02 | 宁波大学 | Three-dimensional video encoding method based on depth image rendering |
Non-Patent Citations (1)
| Title |
|---|
| YONGBING ZHANG, XINGYANG JI, HAOQIAN WANG, LEI ZHANG,QIONGHAI: "Content adaptive subsampling for stereo interleaving video coding", 《PROCEEDINGS OF DATA COMPRESSION CONFERENCE》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105723705A (en) * | 2013-11-20 | 2016-06-29 | 皇家飞利浦有限公司 | Generation of images for an autosteroscopic multi-view display |
| CN105723705B (en) * | 2013-11-20 | 2019-07-26 | 皇家飞利浦有限公司 | The generation of image for automatic stereo multi-view display |
| CN105934948A (en) * | 2013-12-13 | 2016-09-07 | 高通股份有限公司 | Signaling of simplified depth coding (SDC) for depth intra-and inter-prediction modes in 3D video coding |
| CN105934948B (en) * | 2013-12-13 | 2019-04-02 | 高通股份有限公司 | Method and apparatus for decoding and encoding video data and computer-readable storage medium |
| US11039129B2 (en) | 2013-12-13 | 2021-06-15 | Qualcomm Incorporated | Signaling of simplified depth coding (SDC) for depth intra- and inter-prediction modes in 3D video coding |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103200405B (en) | 2016-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7492978B2 (en) | Apparatus, method and computer program for video encoding and decoding | |
| US11659151B2 (en) | Apparatus, a method and a computer program for volumetric video | |
| CN110870302B (en) | Apparatus, method and computer program for omnidirectional video | |
| JP5587552B2 (en) | Multi-view video coding using scalable video coding | |
| CN102365869B (en) | Frame packing for video coding | |
| CN103907347B (en) | Multiview Video Encoding and Decoding | |
| CN101715651B (en) | Method and apparatus for generating block-based stereoscopic image format and method and apparatus for reconstructing stereoscopic images from block-based stereoscopic image format | |
| US11051039B2 (en) | Methods for full parallax light field compression | |
| US20140009574A1 (en) | Apparatus, a method and a computer program for video coding and decoding | |
| CN103108187B (en) | The coded method of a kind of 3 D video, coding/decoding method, encoder | |
| CN100563340C (en) | Multichannel video stream encoder and decoder based on deep image rendering | |
| CN116235497A (en) | A method and apparatus for signaling depth of multiplanar image-based volumetric video | |
| CN101466038B (en) | Method for encoding stereo video | |
| US20130222534A1 (en) | Apparatus, a Method and a Computer Program for Video Coding and Decoding | |
| CN102065296B (en) | Three-dimensional video coding method | |
| AU2012391251A1 (en) | Method, apparatus and system for encoding and decoding video | |
| Maugey et al. | Interactive multiview video system with low complexity 2D look around at decoder | |
| CN104010196B (en) | 3D quality scalable video coding method based on HEVC | |
| CN102625097B (en) | Method for intra-frame prediction of three-dimensional video and coding and decoding methods | |
| CN101198061A (en) | Stereoscopic Video Stream Coding Method Based on View Image Mapping | |
| CN104159095A (en) | Code rate control method for multi-view texture video and depth map coding | |
| CN101404766A (en) | Multi-view point video signal encoding method | |
| CN101494778A (en) | Multi-eyepoint collapsible encoding frame | |
| CN103200405B (en) | A kind of 3DV method for video coding and encoder | |
| Maugey et al. | Consistent view synthesis in interactive multiview imaging |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160601 |