HK1216809B - Method and apparatus for advanced residual prediction (arp) for texture coding - Google Patents

Description

Method and apparatus for advanced residual prediction (ARP) for texture decoding

This application claims the benefit of U.S. Provisional Application No. 61/838,208, filed June 21, 2013, and U.S. Provisional Application No. 61/846,036, filed July 14, 2013, the entire contents of each of which are incorporated herein by reference in their entirety.

Technical Field

The present invention relates to video decoding.

Background Art

Digital video capabilities may be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, electronic book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10 Advanced Video Coding (AVC), the High Efficiency Video Coding (HEVC) standard currently under development, and extensions to these standards. By implementing such video coding techniques, video devices may more efficiently transmit, receive, encode, decode, and/or store digital video information.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video frame or a portion of a video frame) may be partitioned into video blocks, which may also be referred to as treeblocks, coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames.

Spatial or temporal prediction produces a prediction block for the block to be coded. Residual data represents the pixel differences between the original block to be coded and the prediction block. Inter-coded blocks are encoded based on a motion vector pointing to a block of reference samples forming the prediction block and residual data indicating the difference between the coded block and the prediction block. Intra-coded blocks are encoded based on an intra-coding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, resulting in residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients, and entropy coding can be applied to achieve even more compression.

Summary of the Invention

In general, this disclosure describes accurate advanced residual prediction (ARP) techniques for texture coding that may provide improved accuracy relative to other ARP techniques. More specifically, this disclosure describes ARP techniques that include identifying a DMV from a current view to a reference view, and determining a residual predictor block for the ARP of a current video block based on the identification of the DMV.

In some examples, the DMV is the DMV of the current video block, and the techniques include determining an inter-view residual predictor block for an inter-view ARP for the current video block. The DMV is used for inter-view prediction of the current video block based on an inter-view reference video block. Techniques for inter-view ARP may also include identifying temporal reference video blocks in the current and reference views based on temporal motion vectors (TMVs) of the inter-view reference video blocks, and determining a residual predictor block based on the difference between the temporal reference video blocks. In these examples, ARP is not limited to temporal ARP for coding temporally predicted video blocks, but may include inter-view ARP for coding inter-view predicted video blocks.

In some examples, the current video block is temporally predicted, and for the temporal ARP of the current video block, the DMV of the reference video block of the current video block replaces the disparity vector derived for the current video block, for example, according to neighboring block-based disparity vector derivation (NBDV). In such examples, the DMV, typically selected through rate-distortion optimization, may be more accurate than the derived disparity vector, which may result in a more accurate temporal ARP for the current video block. In some examples, the current video block is temporally predicted, and for the temporal ARP of the current video block, the disparity vector derived for the current video block, for example, according to neighboring block-based disparity vector derivation (NBDV), is derived via a co-located depth block of the temporal reference video block of the current video block. Such examples may provide a more accurate temporal ARP when the co-located depth block is available during texture coding.

In one example, a method for decoding inter-view advanced residual prediction of video data includes decoding an encoded video bitstream that encodes the video data to identify a disparity motion vector (DMV) and a residual block for a current video block. The current video block is in a current view, and the DMV is used for inter-view prediction of the current video block based on an inter-view reference video block in a reference view and in the same access unit as the current video block. The method further includes identifying a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block; identifying a temporal reference video block in an associated reference picture in the reference view based on the TMV of the inter-view reference video block; and identifying a temporal reference video block in the current view based on the TMV of the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The method further includes determining a residual predictor block for the current video block based on a difference between the temporal reference video block in the current view and the temporal reference video block in the reference view. The method further includes applying the residual predictor block and the identified residual block from the encoded video bitstream to the inter-view reference video block to reconstruct the current video block.

In another example, a method for encoding inter-view advanced residual prediction of video data includes identifying a disparity motion vector (DMV) for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on a reference view and an inter-view reference video block in the same access unit as the current video block. The method further includes identifying a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block; identifying a temporal reference video block in an associated reference picture in the reference view based on the TMV for the inter-view reference video block; and identifying a temporal reference video block in the current view based on the TMV for the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The method further includes determining a residual predictor block for the current video block based on a difference between the temporal reference video block in the current view and the temporal reference video block in the reference view. The method further includes encoding an encoded video bitstream that encodes the video data to identify the DMV and the residual block for the current video block. The residual block identified by the encoded video bitstream comprises the difference between the inter-view reference video block and the residual predictor block for the current video block.

In another example, a device includes a video decoder configured to perform inter-view advanced residual prediction for decoding video data. The video decoder includes a memory configured to store an encoded video bitstream of the encoded video data, and one or more processors. The one or more processors are configured to identify a disparity motion vector (DMV) for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on an inter-view reference video block in a reference view and in the same access unit as the current video block. The one or more processors are further configured to identify a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block, identify a temporal reference video block in an associated reference picture in the reference view based on the TMV of the inter-view reference video block, and identify a temporal reference video block in the current view based on the TMV of the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The one or more processors are further configured to determine a residual predictor block for the current video block based on a difference between a temporal reference video block in the current view and a temporal reference video block in the reference view. The one or more processors are further configured to decode the encoded video bitstream to identify a DMV and a residual block for the current video block. The residual block identified by decoding the encoded video bitstream includes a difference between an inter-view reference video block and the residual predictor block for the current video block.

In another example, a computer-readable storage medium has instructions stored thereon that, when executed, cause one or more processors of a video coder to identify a disparity motion vector (DMV) for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on an inter-view reference video block in a reference view and in the same access unit as the current video block. The instructions further cause the one or more processors to identify a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block, identify a temporal reference video block in an associated reference picture in the reference view based on the TMV for the inter-view reference video block, and identify a temporal reference video block in the current view based on the TMV for the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The instructions further cause the one or more processors to determine a residual predictor block for the current video block based on a difference between the temporal reference video block in the current view and the temporal reference video block in the reference view. The instructions further cause the one or more processors to code the encoded video bitstream to identify a DMV and a residual block for the current video block. The residual block identified by coding the encoded video bitstream includes a difference between an inter-view reference video block and a residual predictor block for the current video block.

In other examples, a method for encoding inter-view advanced residual prediction of video data includes identifying a disparity motion vector (DMV) for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on a reference view and inter-view reference video blocks in the same access unit as the current video block. The method further includes identifying a temporal motion vector (TMV) and an associated reference picture. In some examples, the DMV may be from a first reference picture list for the current video block, and the TMV and associated reference picture may be from a second reference picture list for the current video block. In other examples, the TMV and associated reference picture are derived from spatially or temporally neighboring blocks of the current video block. In either case, the method may further include identifying a temporal reference video block in a reference view based on the TMV, and identifying a temporal reference video block in the current view based on the TMV. The method further includes determining a residual predictor block for the current video block based on a difference between the temporal reference video block in the current view and the temporal reference video block in the reference view. The method further includes encoding an encoded video bitstream that encodes the video data to identify the DMV and the residual block for the current video block. The residual block identified by the encoded video bitstream comprises the difference between the inter-view reference video block and the residual predictor block for the current video block.

In another example, a method for temporally advanced residual prediction (TMV) for coding video data includes identifying a temporal motion vector (TMV) for a current video block, wherein the current video block is in a current view, and wherein the TMV is used for prediction of the current video block based on a temporal reference video block in the current view and in a different access unit than the current video block. The method further includes identifying a disparity motion vector (DMV) for a temporal reference video block used for inter-view prediction of the temporal reference video block. The method further includes determining, based on the DMV, at least one of an inter-view reference video block in a reference view and in the same access unit as the current video block or a temporal reference video block in a reference view and in a different access unit. The method further includes determining a residual predictor block for the current video block based on a difference between the inter-view reference video block in the reference view and in the same access unit as the current video block and a temporal reference video block in a reference view and in a different access unit. The method further includes coding an encoded video bitstream that encodes video data to identify the TMV and the residual block for the current video block. A residual block identified by the encoded video bitstream comprises a difference between a temporal reference video block and a residual predictor block of a current video block. Some examples of this method further include scaling a TMV of the current video block to a target reference picture in a target access unit for advanced residual prediction of the current video block, wherein the scaled TMV identifies a temporal reference video block in the current view. In some examples of this method, the temporal reference video block in the current view identified by the scaled TMV comprises a first temporal reference video block, and the method further includes determining that the first temporal reference video block in the current view identified by the scaled TMV is not associated with a DMV, and scaling to identify a second temporal reference video block in the current view based on the absence of the TMV. In these examples, identifying the DMV comprises identifying a DMV for the second temporal reference video block in the current view identified by the TMV absence scaling. In some examples of this method, the temporal reference video block in the current view and in a different access unit than the current video block comprises a plurality of prediction units, and identifying the DMV for the temporal reference video block comprises identifying a DMV associated with a center position of one of the plurality of PUs containing the temporal reference video block. In some examples of this method, identifying the DMV includes identifying the DMV from a prediction mode other than backward video synthesis prediction (BVSP). In some examples of this method, the inter-view reference video block contains a first motion information set corresponding to a first reference picture list and a second motion information set corresponding to a second reference picture list, and identifying the TMV for the inter-view reference video block includes selecting the TMV from the first motion information set if the first motion information set includes the TMV, and selecting the TMV from the second motion information set if the first motion information set does not include the TMV. In some examples of this method, the first reference picture list includes RefPicList0. In some examples of this method, the order for considering the first and second motion information sets is independent of which of the first and second motion information sets includes the TMV. In some examples of this method, decoding the coded video bitstream includes decoding the coded video bitstream with a video decoder to identify the TMV and a residual block for a current video block, and applying the residual predictor block and the residual block identified from the coded video bitstream to the temporal reference video block to reconstruct the current video block. In some examples of this method, coding the encoded video bitstream includes encoding the encoded video bitstream with a video encoder to indicate a TMV and a residual block for a current video block to a video decoder.

In another example, a method for temporally advanced residual prediction (TMV) for coding video data includes identifying a temporal motion vector (TMV) for a current video block, wherein the current video block is in a current view, and wherein the TMV is used for prediction of the current video block based on a temporal reference video block in the current view and in a different access unit than the current video block. The method further includes deriving a disparity vector (DV) via a co-located depth block of the temporal reference video block. The method further includes determining, based on the DV, at least one of an inter-view reference video block in a reference view and in the same access unit as the current video block or a temporal reference video block in a reference view and in a different access unit. The method further includes determining a residual predictor block for the current video block based on a difference between the inter-view reference video block in the reference view and in the same access unit as the current video block and a temporal reference video block in a reference view and in a different access unit. The method further includes coding an encoded video bitstream that encodes video data to identify the TMV and the residual block for the current video block. A residual block identified by the coded video bitstream comprises a difference between a temporal reference video block and a residual predictor block of a current video block. Some examples of this method further include scaling a temporal reference video block (TMV) of the current video block to a target reference picture in a target access unit for advanced residual prediction of the current video block, wherein the scaled TMV identifies a temporal reference video block in the current view. In some examples of this method, the temporal reference video block in the current view identified by the scaled TMV comprises a first temporal reference video block, and the method further includes determining that the first temporal reference video block in the current view identified by the scaled TMV is not associated with a DMV, and scaling to identify a second temporal reference video block in the current view based on the absence of the TMV. In these examples, identifying the DMV includes identifying the DMV of the second temporal reference video block in the current view identified by the TMV absence scaling. In some examples of this method, deriving the DV includes converting a depth value of a sample within the co-located depth block to the DV. In some examples, the sample is positioned at (W/2, H/2) relative to a top-left sample of the co-located depth block, wherein the co-located depth block has a size of WxH. In some examples of this method, deriving the DV includes determining a representative depth value based on depth values of a plurality of samples within the co-located depth block, and converting the representative depth value into the DV. In some examples, the plurality of samples are four corner samples. In some examples, the plurality of samples are selected based on neighboring samples of the depth block. In some examples, determining the representative depth value based on the depth values of the plurality of samples within the co-located depth block includes determining the representative depth value based on all depth values of the plurality of samples within the co-located depth block. In some examples of this method, decoding the coded video bitstream includes decoding the coded video bitstream with a video decoder to identify a TMV and a residual block for a current video block, and applying the residual predictor block and the residual block identified from the coded video bitstream to a temporal reference video block to reconstruct the current video block. In some examples of this method, decoding the coded video bitstream includes encoding the coded video bitstream with a video encoder to indicate the TMV and the residual block for the current video block to the video decoder.

The details of one or more embodiments of the present invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a block diagram illustrating an example video encoding and decoding system that may utilize the techniques described in this disclosure.

2 is a graphical diagram illustrating an example multi-view encoding or decoding order.

3 is a conceptual diagram illustrating example temporal and inter-view prediction patterns for multi-view video coding.

4 is a conceptual diagram illustrating an example relationship of neighboring blocks and motion information for predicting a current block.

5 is a conceptual diagram illustrating an example of derivation of inter-view predicted motion vector candidates and inter-view disparity motion vector candidates for predicting motion information of a current block.

6 is a conceptual diagram illustrating example spatial neighboring blocks relative to a current video block from which a disparity vector for the current video block may be derived using neighbor-based disparity vector derivation (NBDV).

7 is a conceptual diagram illustrating an example of the location of a depth block from a reference view and using the located depth block in the reference view for backward view synthesis prediction (BVSP).

8 is a conceptual diagram illustrating an example prediction structure for temporal advanced residual prediction (ARP) for temporally predicted video blocks.

9 is a conceptual diagram illustrating an example bi-directional prediction structure for temporal ARP.

10 is a conceptual diagram of an example prediction structure for inter-view ARP of an inter-view predicted video block, according to the techniques described in this disclosure.

11 is a conceptual diagram of an example prediction structure for temporal ARP of temporally predicted video blocks using disparity motion vectors (DMVs), according to the techniques described in this disclosure.

12 is a conceptual diagram illustrating an example technique for identifying a temporal motion vector (TMV) or DMV in or near a video block, according to the techniques described in this disclosure.

13A-13D are conceptual diagrams illustrating example scanning orders for identifying TMVs or DMVs, according to the techniques of this disclosure.

14 is a block diagram illustrating an example video encoder that may implement the techniques described in this disclosure.

15 is a block diagram illustrating an example video decoder that may utilize the techniques described in this disclosure.

16 is a flowchart illustrating an example ARP method for decoding video blocks according to the techniques described in this disclosure.

17 is a flowchart illustrating an example inter-view ARP method for decoding inter-view predicted video blocks according to the techniques described in this disclosure.

18 is a flowchart illustrating an example temporal ARP method for decoding temporally predicted video blocks according to the techniques described in this disclosure.

19 is a flowchart illustrating an example ARP method for encoding video blocks according to the techniques described in this disclosure.

20 is a flowchart illustrating an example inter-view ARP method for encoding an inter-view predicted video block according to the techniques described in this disclosure.

21 is a flowchart illustrating an example temporal ARP method for encoding temporally predicted video blocks according to the techniques described in this disclosure.

22 is a flow diagram illustrating an example method for identifying DMVs for temporal ARPs in accordance with the techniques described in this disclosure.

23 is a flow diagram illustrating an example method for identifying a TMV or DMV for an ARP in accordance with the techniques described in this disclosure.

DETAILED DESCRIPTION

Generally speaking, this disclosure relates to multi-view video coding, in which the coded video data includes two or more views. In some examples, multi-view video coding includes a multi-view plus depth video coding process. In some examples, multi-view coding may include coding of three-dimensional or 3D video and may be referred to as 3D video coding. Some disclosed techniques may also be applicable to video coding other than multi-view or 3D video coding, such as scalable video coding or video coding according to a base profile of a video coding standard, e.g., where the video data does not include multiple views or layers.

This disclosure also relates to prediction of residual signals for video blocks, such as advanced residual prediction (ARP). More specifically, this disclosure describes techniques for more accurate ARP of texture components of multi-view video data in non-base views. Techniques for more accurate ARP may include identifying a disparity motion vector (DMV) from a current view of a current video block to a reference view. A DMV is a motion vector, for example, in a reference view, used for inter-view prediction of video data in the current view based on video data of the current video block or a reference video block. The techniques may further include using the identified DMV to identify a reference video block for the ARP, and determining a residual predictor block for the current video block based on the identified reference video block. The coded residual block for a current block identified in the encoded video bitstream may be the difference between a normal residual block, which is the difference between the current block and the reference video block of the current video block, and the residual predictor block after potential scaling based on a coded weighting factor index. In this disclosure, the term "current" is generally used to identify the view, picture, or block currently being coded. Thus, the current block generally represents a block of video data being coded, as compared to video blocks that have already been coded or as compared to video blocks that remain to be coded.

In some examples, the DMV may be the DMV of the current video block, in which case the video coder may use the DMV to identify a reference block in a reference view. In these examples, the techniques may include determining an inter-view residual predictor block for an inter-view ARP for the current video block based on the identified DMV. In these examples, the ARP is not limited to a temporal ARP for coding a temporally predicted video block, but may include an inter-view ARP for coding an inter-view predicted video block. The inter-view ARP may allow the video coder to more accurately calculate the inter-view residual predictor in different access units to predict the residual of the current video block.

In some examples, the current video block is temporally predicted, and the DMV may be the DMV of a temporal reference block in the same view as the current video block. In such examples, the video coder may use the DMV, rather than the disparity vector (DV) derived for the current video block, to identify one or both of an inter-view reference block for the current video block in a reference view or a temporal reference block in a reference view for a temporal ARP for the current video block. The video coder may use the block identified based on the DMV to more accurately calculate a temporal residual predictor (calculated in the reference view) to predict the residual for the current video block. In such examples, the DMV, typically selected through rate-distortion optimization, may be more accurate than the derived disparity vector, which may result in a more accurate temporal ARP for the current video block.

Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions. The latest joint draft of MVC is described in "Advanced Video Coding for Generic Audiovisual Services" (ITU-T Recommendation H.264), dated March 2010.

Recently, the design of a new video coding standard, High Efficiency Video Coding (HEVC), has been finalized by the Joint Collaboration on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Motion Picture Experts Group (MPEG). The latest HEVC draft specification , hereinafter referred to as HEVC WD10, is available at http://phenix.int-evry.fr/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1003-v34.zip . The full reference for HEVC WD10 is Bross et al., "High Efficiency Video Coding (HEVC) Textual Specification Draft 10 (for FDIS and later calls)," JCTVC-L1003_v34, Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11, 12th Meeting: Geneva, Switzerland, January 14-23, 2013. HEVC WD10 is incorporated herein by reference in its entirety.

A multi - view extension to HEVC, or MV-HEVC, is also being developed by JCT-3V. The latest MV-HEVC Working Draft (WD), hereinafter referred to as MV-HEVC WD3, is available at http://phenix.it-sudparis.eu/jct2/doc_end_user/documents/3_Geneva/wg11/JCT3V-C1004-v4.zip . The full reference for MV-HEVC WD3 is Tech et al., "MV-HEVC Draft Text 3 (ISO/IEC 23008-2:201x/PDAM2)", JCT3V-C1004_d3, Joint Collaborative Group on Development of 3D Video Coding Extensions of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11, 3rd Meeting: Geneva, Switzerland, January 17-23, 2013. The entire text of MV-HEVC WD3 is incorporated herein by reference.

A scalable extension to HEVC, known as SHVC , is also being developed by JCT-VC. A recent working draft (WD) of SHVC, hereinafter referred to as SHVC WD1, is available at http://phenix.int-evry.fr/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1008-v1.zip . The full reference for SHVC WD1 is Chen et al., "SHVC Draft Text 1," JCTVC-L1008, Joint Collaboration Team on Video Coding (JCT-VC), 12th Meeting, Geneva, Switzerland, January 14-23, 2013, of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC29/WG11. SHVC WD1 is incorporated herein by reference in its entirety.

Currently, the Joint Collaboration Team on 3D Video Coding (JCT-3C) of VCEG and MPEG is developing the 3DV standard based on HEVC. Part of the standardization effort includes the standardization of MV-HEVC, and another part includes the standardization of 3D video coding (3DV) based on HEVC (3D-HEVC). For 3D-HEVC, new coding tools for both texture and depth views, including those at the coding unit/prediction unit level, may be included and supported. The latest reference software test model for 3D-HEVC (3D HTM-7.0) can be downloaded from the following link: https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware/tags/HTM-7.0/ .

The most recent reference software description and working draft for 3D-HEVC are fully referenced as follows: Tech et al., "3D-HEVC Test Model 4," JCT3V-D1005_spec_v1, Joint Collaborative Group on 3D Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC29/WG 11, 4th Meeting: Incheon, South Korea, April 20-26, 2013. This reference software description and working draft for 3D-HEVC can be downloaded from the following link: http://phenix.it-sudparis.eu/jct2/doc_end_user/documents/4_Incheon/wg11/JCT3V-D1005-v1.zip . The respective full texts of 3D-HTM-7.0 and 3D-HEVC Test Model 4 are incorporated herein by reference.

The respective entireties of each of the foregoing references are incorporated herein by reference. The techniques described in this disclosure may be implemented by a video coder operating according to, for example, the MV-HEVC or 3D-HEVC extensions of HEVC or the MVC extension of H.264. However, the techniques described in this disclosure are not limited to those standards and may be extended to other video coding standards described herein or not mentioned herein, including standards that provide residual prediction in video coding.

FIG1 is a block diagram illustrating an example video encoding and decoding system according to one or more examples described in this disclosure. For example, system 10 includes a source device 12 and a destination device 14. Source device 12 and destination device 14 are configured to implement the techniques described in this disclosure. In some examples, system 10 may be configured to support encoding, transmission, storage, decoding, and/or presentation of encoded video data, such as video data encoded according to the HEVC standard described in, for example, WD10 and extensions thereof (e.g., MV-HEVC WD3, SHVC WD1, 3D-HEVC Test Model 4, etc.). However, the techniques described in this disclosure may be applicable to other video coding standards or other extensions.

1 , system 10 includes a source device 12 that generates encoded video data that is later decoded by a destination device 14. Source device 12 and destination device 14 may comprise any of a wide variety of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets (e.g., so-called “smart” phones), so-called “smart” tablet computers, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

Destination device 14 may receive the encoded video data to be decoded via link 16. Link 16 may comprise any type of medium or device capable of moving the encoded video data from source device 12 to destination device 14. In one example, link 16 may comprise a communication medium that enables source device 12 to transmit the encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard (e.g., a wireless communication protocol) and transmitted to destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network such as the Internet). The communication medium may include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 12 to destination device 14.

In some examples, the encoded data may be output from output interface 22 to storage device 36. Similarly, the encoded data may be accessed from storage device 34 via input interface 28. Storage device 36 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, storage device 36 may correspond to a file server or another intermediate storage device that can hold the encoded video generated by source device 12. Destination device 14 may access the stored video data from storage device 36 via streaming or downloading. The file server may be any type of server capable of storing encoded video data and transmitting it to destination device 14. Example file servers include a network server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data via any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on a file server. Transmission of the encoded video data from storage device 36 may be a streaming transmission, a download transmission, or a combination of both.

The techniques of this disclosure are, of course, not limited to wireless applications or settings. The techniques may be applied to video coding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, streaming video transmission (e.g., via the Internet), encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. In some cases, output interface 22 may include a modulator/demodulator (modem) and/or a transmitter. In source device 12, video source 18 may include a source such as a video capture device (e.g., a video camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and/or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As one example, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, the techniques described in this disclosure may be applicable to video coding generally and may be applied to wireless and/or wired applications.

The captured video, pre-captured video, or computer-generated video may be encoded by video encoder 12. The encoded video data may be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data may also (or instead) be stored on storage device 36 for later access by destination device 14 or other devices for decoding and/or playback.

Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. In some cases, input interface 28 may include a receiver and/or a modem. Input interface 28 of destination device 14 receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 36 may include a variety of syntax elements generated by video encoder 20 for use by a video decoder, such as video decoder 30, in decoding the video data. Such syntax elements may be included with the encoded video data transmitted on a communication medium, stored on a storage medium, or stored in a file server.

Display device 32 may be integrated with destination device 14 or external to the destination device. In some examples, destination device 14 may include an integrated display device and also be configured to interface with an external display device. In other examples, destination device 14 may be a display device. Generally, display device 32 displays decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

Video encoder 20 and video decoder 30 may operate according to a video compression standard, such as the HEVC standard currently under development and extensions of the HEVC standard such as MV-HEVC, SHVC, and 3D-HEVC. However, the techniques of this disclosure are not limited to any particular coding standard.

Although not shown in FIG1 , in some aspects, video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams. In some examples, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP), if applicable.

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, a device may store instructions for the software in a suitable, non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in the respective device.

In general, video encoder 20 and video decoder 30 may each operate in accordance with HEVC WD10, MV-HEVC WD3, SHVC WD1, and/or 3D-HEVC Test Model 4 (as described above), or other similar standards or extensions in which the techniques described in this disclosure may be useful. The HEVC standard specifies several additional capabilities of video coding devices relative to existing devices, for example, according to ITU-T H.264/AVC. For example, while H.264 provides nine intra-prediction coding modes, the HEVC standard may provide up to thirty-three intra-prediction coding modes.

In general, a video frame or picture can be divided into a sequence of treeblocks or largest coding units (LCUs) that include both luma and chroma samples. Treeblocks in the HEVC coding process have a similar purpose to macroblocks in the H.264 standard. A slice includes several consecutive treeblocks in coding order. A video frame or picture can be partitioned into one or more slices. Each treeblock can be split into coding units (CUs) according to a quadtree. For example, a treeblock that is the root node of a quadtree can be split into four child nodes, and each child node can be a parent node and split into another four child nodes. The final unsplit child node (which is a leaf node of the quadtree) includes a coding node, i.e., a coded video block. Syntax data associated with the coded bitstream can define the maximum number of times a treeblock can be split, and can also define the minimum size of a coding node.

A CU includes a coding node and prediction units (PUs) and transform units (TUs) associated with the coding node. The size of the CU corresponds to the size of the coding node and must be square in shape. The size of a CU can vary from 8x8 pixels to a treeblock size of 64x64 pixels or larger. Each CU can contain one or more PUs and one or more TUs. For example, the syntax data associated with a CU can describe the partitioning of the CU into one or more PUs. The partitioning mode can distinguish between whether the CU is skipped or encoded in direct mode, encoded in intra-frame prediction mode, or encoded in inter-frame prediction mode. The PU can be partitioned into non-square shapes. For example, the syntax data associated with a CU can also describe the partitioning of the CU into one or more TUs according to a quadtree. The TU can be square or non-square in shape.

The HEVC standard allows for per-TU transforms, which can be different for different CUs. The size of a TU is typically set based on the size of the PU within a given CU defined for the partitioned LCU, but this may not always be the case. A TU is typically the same size as or smaller than a PU. In some instances, the residual samples corresponding to a CU may be subdivided into smaller units using a quadtree structure called a "residual quadtree" (RQT). The leaf nodes of the RQT may be referred to as transform units (TUs). Pixel difference values associated with a TU may be transformed to produce transform coefficients, which may be quantized.

In general, a PU includes data related to the prediction process. For example, when the PU is intra-mode encoded, the PU may include data describing the intra-prediction mode of the PU. As another example, when the PU is inter-mode encoded, the PU may include data defining a motion vector for the PU. The data defining the motion vector for the PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., quarter-pixel precision or eighth-pixel precision), the reference picture to which the motion vector points, and/or the reference picture list for the motion vector (e.g., RefPicList0 (L0) or RefPicList1 (L1)).

In general, TUs are used for transform and quantization processes. A given CU having one or more PUs may also include one or more transform units (TUs). After prediction, video encoder 20 may calculate residual values corresponding to the PUs. The residual values include pixel difference values, which may be transformed into transform coefficients, quantized, and scanned using the TUs to produce serialized transform coefficients for entropy coding. This disclosure generally uses the term "video block" to refer to a coding node of a CU. In some specific cases, this disclosure may also use the term "video block" to refer to a treeblock that includes a coding node as well as a PU and a TU, i.e., an LCU or a CU.

For example, for video coding according to the HEVC standard, a video frame can be divided into coding units (CUs), prediction units (PUs), and transform units (TUs). A CU generally refers to an image region that serves as a basic unit to which various coding tools are applied to achieve video compression. A CU typically has a square geometry and can be considered similar to a so-called "macroblock" under other video coding standards, such as ITU-T H.264.

To achieve better coding efficiency, a CU may have a variable size depending on the video data it contains. That is, a CU may be partitioned or "split" into smaller blocks or sub-CUs, each of which may also be referred to as a CU. In addition, each CU that is not split into sub-CUs may be further partitioned into one or more PUs and TUs for the purposes of prediction and transformation of the CU, respectively.

A PU can be considered similar to a so-called partition of a block under other video coding standards, such as H.264. A PU is the basis for performing prediction for a block to generate "residual" coefficients. The residual coefficients of a CU represent the difference between the video data of the CU and the predicted data for the CU determined using one or more PUs of the CU. Specifically, the one or more PUs specify how the CU is partitioned for prediction purposes and which prediction mode to use to predict the video data within each partition of the CU.

One or more TUs of a CU specify a partition of blocks of residual coefficients for the CU, based on which transforms are applied to the blocks to produce blocks of residual transform coefficients for the CU. The one or more TUs may also be associated with the type of transform applied. The transform converts the residual coefficients from the pixel or spatial domain to a transform domain, such as the frequency domain. Additionally, the one or more TUs may specify parameters based on which quantization is applied to the resulting blocks of residual transform coefficients to produce blocks of quantized residual transform coefficients. The residual transform coefficients may be quantized to potentially reduce the amount of data used to represent the coefficients.

A CU typically includes one luma component, denoted as Y, and two chroma components, denoted as U and V. In other words, a given CU that is not further split into sub-CUs may include Y, U, and V components, each of which may be further partitioned into one or more PUs and TUs for the purposes of prediction and transformation of the CU, as previously described. For example, depending on the video sampling format, the sizes of the U and V components may be the same or different from the size of the Y component in terms of the number of samples. Thus, the techniques described above with respect to prediction, transformation, and quantization may be performed on each of the Y, U, and V components of a given CU.

To encode a CU, one or more predictors for the CU are first derived based on the CU's one or more PUs. A predictor is a reference block containing predicted data for the CU and is derived based on the CU's corresponding PU, as previously described. For example, the PU indicates the partition of the CU for which the predicted data will be determined, and the prediction mode used to determine the predicted data. The predictor can be derived via intra (I) prediction (i.e., spatial prediction) or inter (P or B) prediction (i.e., temporal prediction) modes. Thus, some CUs may be intra-coded (I) using spatial prediction relative to neighboring reference blocks or CUs in the same frame, while other CUs may be inter-coded (P or B) relative to reference blocks or CUs in other frames.

Upon identifying the one or more predictors based on the one or more PUs of a CU, a difference is calculated between the original video data of the CU corresponding to the one or more PUs and the predicted data of the CU contained in the one or more predictors. This difference, also referred to as a prediction residual, comprises residual coefficients and refers to the pixel differences between the portion of the CU specified by the one or more PUs and the one or more predictors, as previously described. The residual coefficients are typically arranged in a two-dimensional (2-D) array corresponding to the one or more PUs of a CU.

To achieve further compression, the prediction residuals are typically transformed, for example, using a discrete cosine transform (DCT), an integer transform, a Karhunen-Lavi (K-L) transform, or another transform. The transform converts the prediction residuals (i.e., residual coefficients) in the spatial domain into residual transform coefficients in a transform domain (e.g., the frequency domain), as also described previously. The transform coefficients are also typically arranged into a 2-D array corresponding to the one or more TUs of the CU. For further compression, the residual transform coefficients may be quantized to potentially reduce the amount of data used to represent the coefficients, as also described previously.

To achieve even further compression, an entropy coder then encodes the resulting residual transform coefficients using context-adaptive binary arithmetic coding (CABAC), context-adaptive variable length coding (CAVLC), probability interval partitioning entropy coding (PIPE), or another entropy coding method. Entropy coding can achieve this further compression by reducing or removing statistical redundancy inherent in the video data of the CU represented by the coefficient relative to other CUs.

A video sequence typically includes a series of video frames or pictures. A group of pictures (GOP) generally includes one or more of a series of video pictures. A GOP may include syntax data in a header of the GOP, a header of one or more of the pictures, or elsewhere, that describes the number of pictures included in the GOP. Each slice of a picture may include slice syntax data that describes the encoding mode for the corresponding slice. Video encoder 20 typically operates on video blocks within individual video slices to encode the video data. A video block may correspond to a decoding node within a CU. A video block may have a fixed or variable size and may have different sizes according to a specified decoding standard.

As an example, HEVC supports prediction for various PU sizes. Assuming a particular CU is 2Nx2N, HEVC supports intra prediction for PU sizes of 2Nx2N or NxN, and inter prediction for symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, or NxN. HEVC also supports asymmetric partitioning for inter prediction for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N. In asymmetric partitioning, one direction of the CU is not partitioned, while the other direction is partitioned into 25% and 75%. The portion of the CU corresponding to the 25% partition is indicated by an "n" followed by an indication of "above," "below," "left," or "right." Thus, for example, "2NxnU" refers to a 2Nx2N CU partitioned horizontally with a 2Nx0.5N PU at the top and a 2Nx1.5N PU at the bottom.

In this disclosure, "NxN" and "N by N" may be used interchangeably to refer to the pixel dimensions of a video block in terms of vertical and horizontal dimensions, e.g., 16x16 pixels or 16 by 16 pixels. Generally, a 16x16 block will have 16 pixels in the vertical direction (y=16) and 16 pixels in the horizontal direction (x=16). Similarly, an NxN block generally has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. The pixels in a block may be arranged in rows and columns. Furthermore, a block need not necessarily have the same number of pixels in the horizontal direction as in the vertical direction. For example, a block may comprise NxM pixels, where M is not necessarily equal to N.

After intra-predictive or inter-predictive coding using the PUs of a CU, video encoder 20 may calculate residual data for the TUs of the CU. A PU may include pixel data in the spatial domain (also referred to as the pixel domain), and a TU may include coefficients in the transform domain after applying a transform to the residual video data, such as a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform. The residual data may correspond to pixel differences between pixels of an unencoded picture and prediction values corresponding to the PU. Video encoder 20 may form a TU including the residual data for the CU, and then transform the TU to produce transform coefficients for the CU.

After performing any transforms used to generate transform coefficients, video encoder 20 may perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing the transform coefficients to potentially reduce the amount of data used to represent the coefficients, thereby providing further compression. The quantization process may reduce the bit depth associated with some or all of the transform coefficients. For example, an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

In some examples, video encoder 20 may utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector that can be entropy encoded. In other examples, video encoder 20 may perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 20 may entropy encode the one-dimensional vector, for example, according to context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method. Video encoder 20 may also entropy encode syntax elements associated with the encoded video data for use by video decoder 30 when decoding the video data.

To perform CABAC, video encoder 20 may assign a context within a context model to a symbol to be transmitted. For example, the context may relate to whether the symbol's neighboring values are non-zero. To perform CAVLC, video encoder 20 may select a variable length code for the symbol to be transmitted. Codewords in VLC may be constructed such that relatively shorter codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, using VLC may achieve bit savings compared to, for example, using codewords of equal length for each symbol to be transmitted. Probability determinations may be based on the context assigned to the symbol.

Video encoder 20 may further send syntax data, such as block-based syntax data, frame-based syntax data, and GOP-based syntax data, e.g., in a frame header, block header, slice header, or GOP header, to video decoder 30. The GOP syntax data may describe the number of frames in the respective GOP, and the frame syntax data may indicate the encoding/prediction mode used to encode the corresponding frame.

Furthermore, video encoder 20 may decode or reconstruct the encoded picture, e.g., by inverse quantizing and inverse transforming the residual data, and combining the residual data with the prediction data. In this manner, video encoder 20 may emulate the decoding process performed by video decoder 30. Both video encoder 20 and video decoder 30 will therefore be able to access substantially the same decoded or reconstructed picture for use in inter-picture prediction.

In general, video decoder 30 may perform a decoding process that is the inverse of the encoding process performed by the video encoder. For example, video decoder 30 may perform entropy decoding using the inverse of the entropy encoding technique used by the video encoder to entropy encode quantized video data. Video decoder 30 may further inverse quantize the video data using the inverse of the quantization technique employed by video encoder 20, and may perform the inverse of the transform used by video encoder 20 to produce quantized transform coefficients. Video decoder 30 may then apply the resulting residual block to a neighboring reference block (intra-frame prediction) or a reference block from another picture (inter-frame prediction) to produce a video block for final display. Video decoder 30 may be configured, instructed, or directed to perform the inverse of various processes performed by video encoder 20 based on syntax elements provided by video encoder 20 and the encoded video data in the bitstream received by video decoder 30. As used herein, the term "video coder" may refer to a video encoder, such as video encoder 20, or a video decoder, such as video decoder 30. Furthermore, the terms "video coding" or "coding" may refer to either or both encoding, such as by a video encoder, or decoding, such as by a video decoder.

In some examples, video encoder 20 and video decoder 30 ( FIG. 1 ) may use techniques for multi-view video coding, e.g., coding of video data including two or more views. In such examples, video encoder 20 may encode a bitstream of encoded video data including two or more views, and video decoder 30 may decode the encoded video data to provide the two or more views, e.g., to display device 32. In some examples, video decoder 30 may provide multiple views of the video data to enable display device 32 to display 3D video. In some examples, video encoder 20 and video decoder 30 may conform to the 3D-HEVC extension of the HEVC standard, e.g., where multi-view coding or multi-view plus depth coding processes are used. Multi-view or 3D video coding may involve coding of two or more texture views and/or views including texture and depth components. In some examples, video data encoded by video encoder 20 and decoded by video decoder 30 includes, or data from which two or more pictures for any given temporal instance (i.e., within an "access unit") may be derived.

In some examples, a device (such as video source 18) may generate the two or more pictures by, for example, capturing a common scene using two or more spatially offset cameras or other video capture devices. Two pictures of the same scene captured simultaneously or nearly simultaneously from slightly different horizontal positions may be used to produce a three-dimensional effect. In some examples, video source 18 (or another component of source device 12) may use depth information or disparity information to generate a second (or other additional) picture of a second (or other additional) view at a given time instance from a first picture of a first view at the given time instance. In this case, the view within the access unit may include a texture component corresponding to the first view and a depth component that may be used with the texture component to generate the second view. The depth or disparity information may be determined by the video capture device that captured the first view, for example, based on camera parameters or other known information about the configuration of the video capture device and the capture of video data for the first view. The depth or disparity information may additionally or alternatively be calculated, for example, by video source 18 or another component of source device 12 from the camera parameters and/or video data in the first view.

To present 3D video, display device 32 may simultaneously or nearly simultaneously display two pictures associated with different views of a common scene, which were captured simultaneously or nearly simultaneously. In some examples, the user of destination device 14 may wear active glasses to quickly and alternately cover the left and right lenses, and display device 32 may quickly switch between the left and right views in sync with the active glasses. In other examples, display device 32 may display both views simultaneously, and the user may wear passive glasses (e.g., with polarized lenses) that filter the views so that the appropriate view enters the user's eyes. In other examples, display device 32 may include a naked-eye stereoscopic display that does not require glasses for the user to perceive a 3D effect.

Multi-view video coding refers to a method of coding multiple views. In the case of 3D video coding, the multiple views may correspond to a left-eye view and a right-eye view, for example. Each of the multiple views includes multiple pictures. The viewer's perception of a 3D scene is due to the horizontal disparity between objects in pictures of different views.

The disparity vector (DV) of a current block in a current picture is a vector that points to a corresponding block in a corresponding picture in a different view than the current picture. Thus, using the DV, a video coder can locate a block in the corresponding picture that corresponds to the current block in the current picture. In this case, the corresponding picture is a picture that is the same temporal instance as the current picture but in a different view. The corresponding block in the corresponding picture and the current block in the current picture may contain similar video content; however, there is at least horizontal disparity between the position of the current block in the current picture and the position of the corresponding block in the corresponding picture. The DV of the current block provides a measure of this horizontal disparity between the block in the corresponding picture and the current block in the current picture.

In some cases, there may also be vertical disparity between the position of a block in the corresponding picture and the position of the current block in the current picture. The DV of the current block may also provide a measure of this vertical disparity between the block in the corresponding picture and the current block in the current picture. The DV contains two components (x and y), but in many cases the vertical component will be zero. The current picture of the current view and the corresponding picture of the different view may be displayed at the same time, that is, the current picture and the corresponding picture are pictures of the same time instance.

In video coding, there are generally two types of prediction, commonly referred to as intra-prediction and inter-prediction. In intra-prediction, the video coder predicts a video block in a picture based on already coded blocks in the same picture. In inter-prediction, the video coder predicts a video block in a picture based on already coded blocks in a different picture (i.e., a reference picture). As used in this disclosure, a reference picture generally refers to any picture containing samples that can be used for inter-prediction in the decoding process of subsequent pictures in decoding order. When coding multi-view content relative to the current picture, such as in 3D-HEVC, the reference pictures may belong to the same temporal instance but in different views, or may be in the same view but in different temporal instances. In the case of multi-view coding, such as in 3D-HEVC, inter-picture prediction can include predicting the current video block (e.g., the current coding node of a CU) from another video block in a temporally different picture (i.e., from a different access unit than the current picture), as well as predicting from a different picture in the same access unit as the current picture but associated with a different view than the current picture.

In the latter case of inter-frame prediction, it may be referred to as inter-view coding or inter-view prediction. In multi-view coding, inter-view prediction is performed among pictures captured in different views of the same access unit (i.e., with the same temporal instance) to remove correlation between views. When coding pictures of non-base views, such as dependent views, pictures from the same access unit but different views (e.g., from a reference view, such as a base view) may be added to a reference picture list. Inter-view reference pictures may be placed in any position of a reference picture list, just as with any inter-predicted (e.g., temporal or inter-view) reference picture.

The blocks of the reference picture used to predict the blocks of the current picture are identified by motion vectors. In multi-view coding, there are at least two types of motion vectors. A temporal motion vector (TMV) is a motion vector that points to a block in a temporal reference picture that is in the same view as the block being coded but in a different temporal instance or access unit than the block being coded, and the corresponding inter-frame prediction is called motion-compensated prediction (MCP). Another type of motion vector is a disparity motion vector (DMV), which points to a block in a picture that is in the same access unit as the current picture but belongs to a different view. With DMV, the corresponding inter-frame prediction is called disparity-compensated prediction (DCP) or inter-view prediction.

FIG2 is a graphical diagram illustrating an example multi-view encoding or decoding order. The decoding order arrangement illustrated in FIG2 may be referred to as time-first coding. In general, for each access unit (i.e., having the same time instance), a multi-view or 3D video sequence may include two or more pictures for each of two or more views, respectively. In FIG2 , S0 to S7 each refer to a different view of the multi-view video. T0 to T8 each represent an output time instance. An access unit may include coded pictures for all views for one output time instance. For example, a first access unit includes all views S0 to S7 (i.e., pictures 0 to 7) for time instance T0, a second access unit includes all views S0 to S7 (i.e., pictures 8 to 15) for time instance T1, and so on. In this example, pictures 0 to 7 are at the same time instance (i.e., time instance T0), and pictures 8 to 15 are at the same time instance (i.e., time instance T1). Pictures with the same temporal instance are typically displayed simultaneously, and horizontal disparity, and possibly some vertical disparity, between objects within the pictures of the same temporal instance causes the viewer to perceive the image as comprising a 3D volume.

In FIG2 , each of the views includes a picture set. For example, view S0 includes picture sets 0, 8, 16, 24, 32, 40, 48, 56, and 64, view S1 includes picture sets 1, 9, 17, 25, 33, 41, 49, 57, and 65, and so on. Each set includes two pictures: one picture is called a texture view component, and the other picture is called a depth view component. Texture view components and depth view components within a picture set of a view can be considered to correspond to each other. For example, a texture view component within a picture set of a view can be considered to correspond to a depth view component within the picture set of the view, and vice versa (i.e., a depth view component corresponds to its texture view component in the set, and vice versa). As used in this disclosure, corresponding texture view components and depth view components can be considered to be part of the same view of a single access unit.

The texture view component contains the actual image content being displayed. For example, the texture view component may include luma (Y) and chroma (Cb and Cr) components. The depth view component may indicate the relative depth of pixels in its corresponding texture view component. As an example, the depth view component may be similar to a grayscale image containing only luma values. In other words, the depth view component may not convey any image content, but rather provides a measure of the relative depth of pixels in the texture view component.

For example, a pixel value corresponding to a pure white pixel in a depth view component may indicate that its corresponding pixel in the corresponding texture view component is closer from the viewer's perspective, and a pixel value corresponding to a pure black pixel in a depth view component may indicate that its corresponding pixel in the corresponding texture view component is farther away from the viewer's perspective. Pixel values corresponding to various shades of gray between black and white indicate different depth levels. For example, a dark gray pixel in a depth view component indicates that its corresponding pixel in the texture view component is farther away than a light gray pixel in the depth view component. Because only one pixel value, similar to grayscale, is required to identify pixel depth, a depth view component may include only one pixel value. Therefore, no values, similar to those for chroma components, are required.

The use of only luma values (eg, intensity values) to identify depth for depth view components is provided for purposes of illustration and should not be considered limiting. In other examples, any technique may be utilized to indicate the relative depth of pixels in a texture view component.

According to multi-view coding, texture view components are inter-predicted from texture view components in the same view or from texture view components in one or more different views. Texture view components may be coded in blocks of video data, which are referred to as "video blocks" and are typically referred to as macroblocks in the context of H.264, or as treeblocks or coding units (CUs) in the context of HEVC.

Pictures of any similar temporal instance may contain similar content. However, the video content of different pictures in similar temporal instances may be slightly shifted relative to each other in the horizontal direction. For example, if a block is located at (x, y) in picture 0 of view S0, then the block located at (x+x', y) in picture 1 of view S1 contains similar video content as the block located at (x, y) in picture 0 of view S0. In this example, the block located at (x, y) in picture 0 of view S0 and the block located at (x+x', y) in picture 1 of view S1 are considered corresponding blocks. In some examples, the DV of the block located at (x+x', y) in picture 1 of view S1 refers to the position of its corresponding block. For example, the DV of the block located at (x+x', y) is (-x', 0).

In some examples, video encoder 20 or video decoder 30 may utilize the DV of a block in a picture of a first view to identify a corresponding block in a picture of a second view. Video encoder 20 and video decoder 20 may utilize the DV, for example, when performing inter-view prediction. Video encoder 20 and video decoder 30 may perform inter-view prediction, for example, by using information of a reference block of a reference picture in a reference view determined using the DV of the current block.

FIG3 is a conceptual diagram illustrating example temporal and inter-view prediction patterns for multi-view video coding. Similar to the example of FIG2 , in the example of FIG3 , eight views (with view IDs “S0” through “S7”) are illustrated, and twelve temporal locations or access units (“T0” through “T11”) are illustrated for each view. That is, each row in FIG3 corresponds to a view, while each column indicates a temporal location or access unit. An object (which may be a picture or an example video block in a different picture) is indicated at the intersection of each row and each column in FIG3 . The H.264/AVC standard with the MVC extension may use the term frame to refer to a portion of a video, while the HEVC standard may use the term picture to refer to a portion of a video. This disclosure may use the terms picture and frame interchangeably.

In FIG3 , view S0 can be considered a base view, and views S1 through S7 can be considered dependent views. The base view includes pictures that are not inter-predicted. Pictures in the base view can be inter-predicted relative to other pictures in the same view. For example, no pictures in view S0 can be inter-predicted relative to pictures in any of views S1 through S7, but some of the pictures in view S0 can be inter-predicted relative to other pictures in view S0.

Additionally, access units T0 and T8 are random access units, or random access points, of the video sequence of the example prediction structure of FIG. As illustrated by the blocks labeled "I" in the example prediction structure of FIG. 3 , at each random access point (T0 and T8), the video blocks of the base view picture (S0) are intra-picture predicted. Video blocks of other non-base view pictures in the random access point, or base and non-base view pictures in non-random access points, may be inter-picture predicted via temporal inter-frame prediction or inter-view prediction, as illustrated by the various blocks labeled "I," "B," "P," or "b" in the example prediction structure of FIG. Predictions in the example prediction structure of FIG. 3 are indicated by arrows, where the picture to which the arrow points uses the picture from which the arrow originates for prediction reference.

Dependent views include pictures that are inter-view predicted. For example, each of views S1 to S7 includes at least one picture that is inter-predicted relative to a picture in another view. Pictures in a dependent view may be inter-predicted relative to pictures in a base view, or may be inter-predicted relative to pictures in other dependent views. In the example of FIG3 , uppercase "B" and lowercase "b" are used to indicate different hierarchical relationships between pictures, rather than different coding methods. Generally speaking, uppercase "B" pictures are higher in the prediction hierarchy than lowercase "b" pictures.

A video stream that includes both a base view and one or more dependent views may be decodable by different types of video decoders. For example, a basic type of video decoder may be configured to decode only the base view. Additionally, another type of video decoder may be configured to decode each of views S0 to S7. A decoder configured to decode both the base view and the dependent views may be referred to as a decoder that supports multi-view coding.

The pictures (or other objects) in FIG3 are illustrated using shaded blocks containing letters that indicate whether the corresponding picture is intra-coded (that is, an I-picture), inter-coded in one direction (that is, as a P-picture), or inter-coded in multiple directions (that is, as a B-picture). In general, prediction is indicated by arrows, where the picture pointed to by the arrow uses the picture from which the arrow originates for prediction reference. For example, the P-picture of view S2 at temporal location T0 is predicted from the I-picture of view S0 at temporal location T0.

As with single-view video coding, pictures of a multi-view video coding video sequence can be predictively encoded relative to pictures at different temporal locations. For example, a B-picture of view S0 at temporal location T1 has an arrow pointing to it from an I-picture of view S0 at temporal location T0, indicating that the B-picture is predicted from the I-picture. However, in the context of multi-view video coding, pictures can also be inter-view predicted. That is, a view component (e.g., a texture view component) can use view components in other views for reference purposes. For example, in multi-view coding, inter-view prediction is implemented as if a view component in another view is an inter-prediction reference. Potential inter-view references can be signaled and modified by the reference picture list construction process, which enables flexible ordering of inter-prediction or inter-view prediction references.

FIG3 provides various examples of inter-view prediction. In the example of FIG3, pictures of view S1 are illustrated as being predicted from pictures at different temporal locations of view S1, as well as being inter-view predicted from pictures of views S0 and S2 at the same temporal location. For example, the B-picture of view S1 at temporal location T1 is predicted from each of the B-pictures of view S1 at temporal locations T0 and T2, as well as the B-pictures of views S0 and S2 at temporal location T1.

FIG3 also illustrates a variation in the prediction hierarchy using different levels of shading, where frames with greater amounts of shading (i.e., relatively darker) are higher in the prediction hierarchy than those with less shading (i.e., relatively lighter). For example, all I pictures in FIG3 are illustrated as having full shading, while P pictures have slightly lighter shading, and B pictures (and lowercase b pictures) have various levels of shading relative to each other, but are always lighter than P and I pictures.

In general, the prediction hierarchy may be related to the view order index, as pictures higher in the prediction hierarchy should be decoded before pictures lower in the hierarchy. Pictures higher in the hierarchy can be used as reference pictures during decoding of pictures lower in the hierarchy. The view order index is an index that indicates the decoding order of view components in an access unit. The decoding of view components may follow the ascending order of the view order index. If all views are presented, the view order index set may comprise a contiguous, ordered set from zero to the total number of views minus one.

For some pictures at equal levels of the hierarchy, the decoding order relative to each other may not matter. For example, the I-picture of view S0 at temporal location T0 can be used as a reference picture for the P-picture of view S2 at temporal location T0, which in turn can be used as a reference picture for the P-picture of view S4 at temporal location T0. Therefore, the I-picture of view S0 at temporal location T0 should be decoded before the P-picture of view S2 at temporal location T0, which in turn should be decoded before the P-picture of view S4 at temporal location T0. However, between views S1 and S3, the decoding order does not matter because views S1 and S3 do not depend on each other for prediction. Instead, views S1 and S3 are predicted only from other views higher in the prediction hierarchy. Furthermore, view S1 can be decoded before view S4 as long as view S1 is decoded after views S0 and S2.

As described above, in 3D-HEVC, video encoder 20 and video decoder 30 may inter-predict a current block in a current picture of a first view with reference to a reference block in a reference picture of a second view. This inter-prediction is referred to as inter-view prediction. The temporal instances of the current picture and the reference picture may be the same in the respective views. In such examples, video encoder 20 or video decoder 30 performs inter-view prediction across pictures in the same access unit, where the pictures in the same access unit are at the same temporal instance.

To perform inter-view prediction on a current block, video encoder 20 or video decoder 30 constructs a reference picture list that identifies reference pictures that can be used for inter-frame prediction, including pictures that can be used for inter-view prediction. Inter-frame prediction refers to predicting a current block in a current picture relative to a reference block in a reference picture. Inter-view prediction is a subset of inter-frame prediction because, in inter-view prediction, the reference picture is in a view different from that of the current picture. Therefore, for inter-view prediction, video encoder 20 and video decoder 30 add a reference picture from another view to one or both of the constructed reference picture lists. The reference picture from another view can be identified at any position within the constructed reference picture lists. As used in this disclosure, when video encoder 20 is performing inter-frame prediction (e.g., inter-frame prediction) on a block, video encoder 20 may be considered to be performing inter-frame prediction encoding on the block. When video decoder 30 is performing inter-frame prediction (e.g., inter-frame prediction) on a block, video decoder 30 may be considered to be performing inter-frame prediction decoding on the block. In inter-view prediction, the DMV of the current video block identifies the position of a block in a reference picture in a view (except for the view where the picture includes the video block to be predicted to be used as a reference block for inter-prediction of the current block), and a reference index into one or both of the constructed reference picture lists identifies a reference picture in the other view.

This disclosure describes techniques for performing ARP, which include identifying a DMV for a current video block or a reference video block, and determining a residual predictor block for the current video block based on the identified DMV. The DMV for the current video block or a reference video block in the same view as the current video block can be considered as a DMV from the current view of the current video block to a reference view used for inter-view prediction of video data in the current view based on video data in the reference view. The techniques of this disclosure may be implemented by one or both of video encoder 20 and video decoder 30. These techniques may be used, for example, in conjunction with HEVC-based multi-view video coding and/or HEVC-based 3D video coding.

As discussed above, the data defining the TMV or DMV for a block of video data may include the horizontal and vertical components of a vector and the resolution of the vector. Motion information for a video block may include a motion vector, as well as a prediction direction and a reference picture index value. Additionally, the motion information for a current video block may be predicted from the motion information of neighboring video blocks, which may also be referred to as reference video blocks. Reference video blocks may be spatial neighbors within the same picture, temporal neighbors within different pictures of the same view but within different access units, or video blocks within different pictures of different views but within the same access unit. In the case of motion information from a reference block in a different view, the motion vector may be a TMV derived from a reference block in an inter-view reference picture (i.e., a reference picture in the same access unit as the current picture but from a different view) or a DMV derived from a DV.

Typically, for motion information prediction, a list of candidate motion information from various reference blocks is formed in a defined manner, for example, such that the motion information from the various reference blocks is considered for inclusion in the list in a defined order. After forming the candidate list, video encoder 20 may evaluate each candidate to determine which provides the best rate and distortion characteristics that best matches a given rate and distortion profile selected for encoding the video. Video encoder 20 may perform a rate-distortion optimization (RDO) procedure with respect to each of the candidates, thereby selecting one of the motion information candidates with the best RDO result. Alternatively, video encoder 20 may select one of the candidates stored in the list that best approximates the motion information determined for the current video block.

In any case, video encoder 20 may specify a selected one of the candidates in the candidate list of motion information using an index that identifies the selected candidate. Video encoder 20 may signal this index in the encoded bitstream for use by video decoder 30. For coding efficiency, the candidates may be ordered in the list so that the candidate motion information most likely to be selected for coding the current video block is associated first or otherwise with the lowest magnitude index value.

Techniques for motion information prediction may include merge mode, skip mode, and advanced motion vector prediction (AMVP) mode. Generally speaking, according to merge mode and/or skip mode, the current video block inherits motion information, such as a motion vector, prediction direction, and reference picture index, from another previously coded neighboring block, such as a spatially neighboring block in the same picture or a block in a temporal or inter-view reference picture. When implementing merge/skip mode, video encoder 20 constructs a list of merge candidates as motion information for a reference block in a defined manner, selects one of the merge candidates, and signals a candidate list index identifying the selected merge candidate to video decoder 30 in the bitstream.

When implementing merge/skip mode, video decoder 30 receives this candidate list index, reconstructs the merge candidate list according to a defined method, and selects one of the merge candidates in the candidate list indicated by the index. Video decoder 30 can then materialize the selected one of the merge candidates as the motion vector of the current PU at the same resolution and pointing to the same reference picture as the motion vector of the selected one of the merge candidates. Therefore, at the decoder side, once the candidate list index is decoded, all motion information of the corresponding block of the selected candidate, such as the motion vector, prediction direction, and reference picture index, can be inherited. Merge mode and skip mode improve bitstream efficiency by allowing video encoder 20 to signal an index into the merge candidate list instead of all motion information for inter-frame prediction of the current video block.

When implementing AMVP, video encoder 20 constructs a list of candidate motion vector predictors (MVPs) in a defined manner, selects one of the candidate MVPs, and signals a candidate list index identifying the selected MVP in the bitstream to video decoder 30. Similar to merge mode, when implementing AMVP, video decoder 30 reconstructs a list of candidate MVPs in a defined manner, decodes the candidate list index from the encoder, and selects and instantiates one of the MVPs based on the candidate list index.

However, in contrast to merge/skip mode, when implementing AMVP, video encoder 20 also signals a reference picture index and prediction direction, thus specifying the reference picture to which the MVP, as specified by the candidate list index, points. Furthermore, video encoder 20 determines a motion vector difference (MVD) for the current block, where the MVD is the difference between the MVP and the actual motion vector that would have been used for the current block. With AMVP, in addition to the reference picture index, reference picture direction, and candidate list index, video encoder 20 also signals the MVD for the current block in the bitstream. Due to the signaling of the reference picture index and prediction vector difference for a given block, AMVP may not be as efficient as merge/skip mode, but may provide improved fidelity of the coded video data.

FIG4 shows an example of a current video block 47, five spatially neighboring blocks (41, 42, 43, 44, and 45) from another picture but in the same view as the current picture, and a temporal reference block 46. Temporal reference block 46 may, for example, be a co-located block in a picture of a different temporal instance but in the same view as current video block 47. In some examples, current video block 47 and reference video blocks 41-46 may be defined as generally in the currently developing HEVC standard. Reference video blocks 41-46 are labeled A0, A1, B0, B1, B2, and T according to the currently developing HEVC standard. Video encoder 20 and video decoder 30 may predict motion information (including TMV) for current video block 47 based on the motion information of reference video blocks 41-46 according to a motion information prediction mode such as merge/skip mode or AMVP mode. As described in more detail below, the TMV of a video block may be used with the DMV to implement advanced residual prediction according to the techniques of this disclosure.

4 , video blocks 42, 44, 43, 41, and 45 may be respectively to the left, above, above right, below left, and above left relative to current video block 47. However, the number and positions of neighboring blocks 41-45 relative to current video block 47 illustrated in FIG4 are merely examples. In other locations, motion information of a different number of neighboring blocks and/or blocks at different positions may be considered for inclusion in the motion information prediction candidate list for current video block 47.

The spatial relationship of each of spatially neighboring blocks 42, 44, 43, 41, and 45 to current video block 47 can be described as follows. Luma position (xP, yP) specifies the top-left luma sample of the current block relative to the top-left sample of the current picture; the variables nPSW and nPSH refer to the width and height of the current block for luma. The top-left luma sample of spatially neighboring block 42 is xP-1, yP+nPSH-1. The top-left luma sample of spatially neighboring block 44 is xP+nPSW-1, yP-1. The top-left luma sample of spatially neighboring block 43 is xP+nPSW, yP-1. The top-left luma sample of spatially neighboring block 41 is xP-1, yP+nPSH. The top-left luma sample of spatially neighboring block 45 is xP-1, yP-1. Although described relative to luma position, the current and reference blocks may include chroma components.

Each of spatially neighboring blocks 41-45 may provide a spatial motion information candidate for predicting motion information (e.g., TMV) for current video block 47. A video coder, such as video encoder 20 ( FIG. 1 ) or video decoder 30 ( FIG. 1 ), may consider the motion information of spatially neighboring reference blocks in a predetermined order (e.g., scan order). For example, in the case of 3D-HEVC, a video decoder may consider the motion information of reference blocks included in a merge candidate list for merge mode in the following order: 42, 44, 43, 41, and 45. In the illustrated example, spatially neighboring blocks 41-45 are to the left and/or above current video block 47. This arrangement is typical because most video coders code video blocks in raster scan order, starting from the top left of a picture. Therefore, in such examples, spatially neighboring blocks 41-45 will typically be coded before current video block 47. However, in other examples, such as when a video coder codes video blocks in a different order, spatially neighboring blocks 41 - 45 may be located to the right of and/or below current video block 47 .

Temporal reference block 46 is located in a temporal reference picture that is coded before (but not necessarily immediately before in coding order) the current picture of current video block 47. Additionally, the reference picture of block 46 does not necessarily precede the picture of current video block 47 in display order. Reference video block 46 may generally be co-located in the reference picture relative to the position of current video block 47 in the current picture. In some examples, reference video block 46 is located to the right and below the position of current video block 47 in the current picture, or overlaps the center position of current video block 47 in the current picture.

5 is a conceptual diagram illustrating an example of deriving inter-view predicted motion vector candidates (IPMVC) and inter-view disparity motion vector candidates (IDMVC) for predicting motion information of a current video block 50, e.g., according to merge/skip mode or AMVP mode. When inter-view prediction is enabled, video encoder 20 and/or video decoder 30 may add a new motion vector candidate, IPMVC or IDMVC, to the motion information candidate list for the current video block 50. IPMVC may predict the TMV of the current video block 50, which video encoder 20 and/or video decoder 30 may use for the ARP of the current video block 50 or another video block, according to the techniques of this disclosure, as described in more detail below. IDMVC may predict the DMV of the current video block 50, which video encoder 20 and/or video decoder 30 may use for the ARP of the current video block 50 or another video block, according to the techniques of this disclosure, as described in more detail below.

5 , current block 50 is in current view Vm. Video encoder 20 and/or video decoder 30 may use disparity vector (DV) 51 to locate corresponding or reference block 52 in reference view V0. The video coder may determine DV 51 based on camera parameters or according to any of the techniques described herein. For example, the video coder may determine DV 51 for current video block 50 based on DVs of neighboring blocks, e.g., using neighbor-based disparity vector derivation (NBDV).

If reference block 52 is not intra-coded and not inter-view predicted, and its reference picture (e.g., reference picture 58 or reference picture 60) has a picture order count (POC) value equal to the POC value of an entry in the same reference picture list of current video block 50, then video encoder 20 and/or video decoder 30 may derive its motion information (prediction direction, reference picture, and motion vector) after converting the POC-based reference index to an IPMVC for current video block 50. In the example of FIG5 , reference video block 52 is associated with a TMV 54 specified in a first reference picture list (RefPicList0) pointing to a first reference picture 58 in reference view V0 and a TMV 56 specified in a second reference picture list (RefPicList1) pointing to a second picture 60 in reference view V0. That current video block 50 inherits TMVs 54 and 56 is illustrated by the dashed arrows in FIG5 . Based on the motion information of reference video block 52, the video coder derives the IPMVC for current video block 50 as at least one of a TMV 62 specified in a first reference picture list (RefPicList0) pointing to a first reference picture 66 in current view Vm (e.g., having the same POC as reference picture 58 in the first reference picture list) and a TMV 64 specified in a second reference picture list (RefPicList1) pointing to a second picture 68 in current view Vm (e.g., having the same POC as reference picture 60). Video encoder 20 and/or video decoder 30 may use TMV 62 and/or TMV 64 for the ARP of current video block 50 or another video block in accordance with the techniques of this disclosure, as described in more detail below.

The video encoder 20 and/or video decoder 30 may also convert the DV 51 to an IDMVC for the current video block 50 and add the IDMVC to the motion information candidate list for the current video block 50 in a different location than the IPMVC. Each of the IPMVC or IDMVC may be referred to as an "inter-view candidate" in this context. In merge/skip mode, the video coder inserts the IPMVC (if available) preceding all spatial and temporal merge candidates into the merge candidate list. In merge/skip mode, the video coder inserts the IDMVC (block 41 of FIG. 4 ) preceding the spatial merge candidate derived from A0. The conversion of the DV 51 to the IDMVC may be viewed as a conversion of the DV 51 to the DMV for the current video block 50. The video encoder 20 and/or video decoder 30 may use the DMV for the ARP of the current video block 50 or another video block according to the techniques of this disclosure, as described in more detail below.

In some cases, the video coder may derive the DV for the current video block. For example, as described above with reference to FIG5, video encoder 20 and/or video decoder 30 may derive DV 51 for current video block 50. In some examples, the video coder may derive the DV for the current video block using neighboring block-based disparity vector (NBDV) derivation.

The proposal for 3D-HEVC uses a texture-first coding order for all views. In other words, for each of the multiple views in the bitstream, the texture component is coded, e.g., encoded or decoded, before any depth component of the view. In some cases, such as for inter-view prediction, a DV is required to code a video block in the texture component of a view in a particular access unit. However, in texture-first coding, the corresponding depth component of the current video block is not available for determining the DV for the current video block. NBDV can be employed by the video coder and is proposed for 3D-HEVC to derive the DV for the current video block in such cases. In the current 3D-HEVC design, the DV derived from NBDV can be further improved by retrieving depth data from the depth map of the reference view pointed to by the DV from the NBDV process.

DV is used to estimate the displacement between two views. Because neighboring blocks share almost the same motion/disparity information in video decoding, the current video block can use the motion vector information in the neighboring blocks as a good predictor of its motion/disparity information. Following this idea, NBDV uses neighboring disparity information to estimate DV in different views.

According to NBDV, the video coder identifies several spatial and temporal neighboring blocks. Two sets of neighboring blocks are used. One set comes from spatial neighboring blocks and the other comes from temporal neighboring blocks. The video coder then checks each of the spatial and temporal neighboring blocks in a predefined order determined by the relative priority between the current block and the candidate (neighboring) block. When the video coder identifies a DMV in the motion information of a candidate (i.e., a motion vector pointing from a neighboring candidate block to an inter-view reference picture (in the same access unit but in a different view)), the video coder converts the DMV to a DV and returns the associated view order index. For example, the video coder may set the horizontal component of the DV of the current block equal to the horizontal component of the DMV and may set the vertical component of the DV to 0.

3D-HEVC initially adopted the NBDV method proposed by Zhang et al. in "3D-CE5.h: Disparity Vector Generation Results" (Joint Collaboration Group on Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11, 1st Meeting: Stockholm, Sweden, July 16-20, 2012, document JCT3V-A0097 (MPEG No. m26052, hereinafter referred to as "JCT3V-A0097"). JCT3V-A0097 can be downloaded from the following link:

http://phenix.int-evry.fr/jct2/doc_end_user/current_document.php?id=89. The entire contents of JCT3V-A0097 are incorporated herein by reference.

In some designs of 3D-HEVC, when the video coder performs the NBDV process, the video coder checks the disparity motion vectors in temporally neighboring blocks, the disparity motion vectors in spatially neighboring blocks, and then checks the implicit disparity vector (IDV) in order. The IDV may be the disparity vector of a spatially or temporally neighboring PU coded using inter-view prediction. The IDV may also be referred to as a derived disparity vector. The IDV may be generated when the PU employs inter-view prediction, i.e., a candidate for AMVP or merge mode is derived from a reference block in another view with the aid of a disparity vector. This disparity vector is referred to as the IDV. The IDV may be stored in the PU for the purpose of DV derivation. For example, even though a block is coded using motion prediction, the derived DV of the block is not discarded for the purpose of coding the following video block. Therefore, when the video coder identifies a DMV or IDV, the video coder may return the identified DMV or IDV.

The simplified NBDV in Sung et al., "3D-CE5.h: Simplified Disparity Vector Derivation for HEVC-Based 3D Video Coding" (Joint Collaboration Group on Video Coding Extensions Development, 1st Meeting, ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11: Stockholm, Sweden, July 16-20, 2012, document JCT3V-A0126 (MPEG No. m26079, hereinafter "JCT3V-A0126")) includes an implicit disparity vector (IDV). JCT3V-A0126 can be downloaded from the following link:

http://phenix.int-evry.fr/jct2/doc_end_user/current_document.php?id=142. The entire contents of JCT3V-A0126 are incorporated herein by reference.

Further development of NBDV for 3D-HEVC occurred in Kang et al., “3D-CE5.h: Improvements for disparity vector derivation” (2nd Meeting of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11 Joint Collaboration Group on Video Coding Extension Development: Shanghai, China, October 13-19, 2012, document JCT3V-B0047 (MPEG No. m26736, hereinafter “JCT3V-B0047”). JCT3V-B0047 can be downloaded from the following link:

http://phenix.it-sudparis.eu/jct2/doc_end_user/current_document.php?id=236. The entire contents of JCT3V-B0047 are incorporated herein by reference. In JCT3V-B0047, NBDV for 3D-HEVC is further simplified by removing the IDV stored in the decoded picture buffer, but coding gain is also improved with random access point (RAP) picture selection. The video coder can convert the returned disparity motion vector or IDV into a disparity vector and use the disparity vector for inter-view motion prediction and inter-view residual prediction. Random access refers to decoding of the bitstream starting from a coded picture that is not the first coded picture in the bitstream. Random access is achieved by inserting random access pictures or random access points into the bitstream at regular intervals. Example types of random access pictures include instantaneous decoder refresh (IDR) pictures, clean random access (CRA) pictures, and broken link access (BLA) pictures. Thus, IDR pictures, CRA pictures, and BLA pictures are collectively referred to as RAP pictures.In some examples, a RAP picture may have a NAL unit type equal to BLA_W_LP, BLA_W_RADL, BLA_N_LP, IDR_W_RADL, IDR_N_LP, RSV_IRAP_VCL22, RSV_IRAP_VCL23, or CRA_NUT.

Techniques for CU-based DV derivation for 3D-HEVC are proposed in Kang et al., “CE2.h: CU-based Disparity Vector Derivation in D-HEVC” (4th Meeting of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11 Joint Collaborative Group on Video Coding Extension Development: Incheon, South Korea, April 20-26, 2013, document JCT3V-D0181 (MPEG No. m29012, hereinafter “JCT3V-D0181”). JCT3V-D0181 can be downloaded from the following link:

http://phenix.it-sudparis.eu/jct3v/doc_end_user/current_document.php?id=866. The entire contents of JCT3V-D0181 are incorporated herein by reference.

When the video coder identifies a DMV or IDV, the video coder can terminate the checking process. Therefore, once the video coder finds the DV for the current block, the video coder can terminate the NBDV process. When the video coder is unable to determine the DV for the current block by performing the NBDV process (i.e., when there is no DMV or IDV found during the NBDV process), the NBDV is marked as unavailable for use. In other words, the NBDV process can be considered to return an unusable disparity vector.

If the video coder is unable to derive the DV of the current block by performing the NBDV process (i.e., if no disparity vector is found), the video coder may use 0DV as the DV of the current PU. 0DV is a DV having both horizontal and vertical components equal to 0. Therefore, even when the NBDV process returns a result that is not available for use, other decoding processes of the video coder that require DV may use a 0 disparity vector for the current block. In some examples, if the video coder is unable to derive the DV of the current block by performing the NBDV process, the video coder may disable inter-view residual prediction for the current block. However, regardless of whether the video coder is able to derive the DV of the current block by performing the NBDV process, the video coder may use inter-view prediction for the current block. That is, if no DV is found after checking all predefined neighboring blocks, a 0 disparity vector may be used for inter-view prediction, while inter-view residual prediction may be disabled for the corresponding CU.

FIG6 is a conceptual diagram illustrating example spatial neighboring blocks relative to current video block 90 from which the DV of the current video block can be derived using NBDV. The five spatial neighboring blocks illustrated in FIG6 are lower left block 96, left block 95, upper right block 92, upper block 93, and upper left block 94 relative to the current video block. The spatial neighboring blocks may be the lower left, left, upper right, upper, and upper left blocks of the CU covering the current video block. It should be noted that these spatial neighboring blocks of NBDV may be the same spatial neighboring blocks used by the video coder for motion information prediction of the current video block, for example, according to the MERGE/AMVP mode in HEVC. In such cases, additional memory access by the video coder for the NBDV may not be required because the motion information of the spatial neighboring blocks is already taken into account for motion information prediction of the current video block.

To check temporally neighboring blocks, the video coder constructs a candidate picture list. In some examples, the video coder may process up to two reference pictures from the current view, i.e., the same view as the current video block, as candidate pictures. The video coder may first insert the co-located reference picture into the candidate picture list, followed by the rest of the candidate pictures in ascending order of reference picture index. When a reference picture with the same reference index in two reference picture lists is available, the video coder may insert one of the reference picture lists that is the same as the co-located picture before another reference picture from the other reference picture list. In some examples, the video coder may identify three candidate regions for deriving temporally neighboring blocks from each of the candidate pictures in the candidate picture list. The three candidate regions may be defined as follows:

●CPU: The co-located area of the current PU or the current CU.

● CLCU: Largest coding unit (LCU) covering the co-located region of the current block.

BR: The lower right 4x4 block of the CPU.

If the PU covering the candidate area specifies a DMV, the video coder may determine the DV for the current video unit based on the disparity motion vector of the PU.

As discussed above, in addition to the DMVs derived from spatial and temporal neighboring blocks, the video coder can also check for IDVs. In the proposed NBDV process of 3D-HTM 7.0, the video coder sequentially checks for DMVs in temporal neighboring blocks, then DMVs in spatial neighboring blocks, and then IDVs. Once a DMV or IDV is found, the process terminates.

When the video coder checks a neighboring PU (i.e., a spatial or temporal neighboring PU), the video coder may first check whether the neighboring PU has a disparity motion vector. If none of the neighboring PUs has a disparity motion vector, the video coder may determine whether any of the spatial neighboring PUs has an IDV. If one of the spatial neighboring PUs has an IDV and the IDV is coded as a merge/skip mode, the video coder may terminate the checking process and use the IDV as the final disparity vector for the current PU.

As noted above, a video coder may apply an NBDV process to derive the DV for a current block (e.g., a CU, PU, etc.). The disparity vector of the current block may indicate a position in a reference picture (i.e., a reference component) in a reference view. In some 3D-HEVC designs, the video coder is allowed to access depth information of the reference view. In some such 3D-HEVC designs, when the video coder uses the NBDV process to derive the DV for the current block, the video coder may apply a refinement process to further refine the disparity vector for the current block. The video coder may refine the DV for the current block based on a depth map of a reference picture. The video coder may use a similar refinement process to refine the DMV for backward view synthesis prediction. In this way, depth may be used to refine either the DV or the DMV for backward view synthesis prediction. This refinement process may be referred to herein as NBDV refinement ("NBDV-R"), an NBDV refinement process, or depth-directed NBDV (Do-NBDV).

When the NBDV process returns a usable disparity vector (e.g., when the NBDV process returns a variable indicating that the NBDV process is able to derive the disparity vector of the current block based on the disparity motion vector or IDV of the neighboring block), the video coder may further refine the disparity vector by retrieving depth data from a depth map of the reference picture. In some examples, the refinement process includes the following two steps:

1) Locate the corresponding depth block by the derived DV in a previously coded reference depth view, such as the base view; the size of the corresponding depth block is the same as that of the current PU.

2) Select one depth value from the four corner pixels of the corresponding depth block and convert it to the horizontal component of the improved DV. The vertical component of DV remains unchanged.

The improved DV can be used for inter-view prediction of the current video block, while the unimproved DV can be used for inter-view residual prediction of the current video block. In addition, the improved DV is stored as the motion vector of one PU (if it is coded using backward view synthesis prediction (BVSP) mode), which is described in more detail below. In the proposed NBDV process of 3D-HTM 7.0, the depth view component of the base view will always be accessed, regardless of the value of the view order index derived from the NBDV process.

Tian et al., “CE1.h: View synthesis prediction using neighboring blocks” (3rd meeting of the Joint Collaborative Team on Video Coding Extensions Development of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11: Geneva, Switzerland, January 23, 2013, document JCT3V-C0152 (MPEG No. m27909, hereinafter referred to as “JCT3V-C0152”)) proposed a backward view synthesis prediction (BVSP) method. JCT3V-C0152 can be downloaded from the following link:

http://phenix.int-evry.fr/jct3v/doc_end_user/current_document.php?id=594 . The entire contents of JCT3V -C0152 are incorporated herein by reference.

JCT3V-C0152 was adopted at the 3rd JCT-3V meeting. The basic concept of this BSVP is the same as the block-based VSP in 3D-AVC. Both techniques use backward warping and block-based VSP to avoid transmitting motion vector differences and use more accurate motion vectors. Implementation details vary depending on the platform. In the following paragraphs, we also use the term BVSP to refer to either or both the backward warping VSP method in 3D-HEVC and the block-based VSP in 3D-AVC.

In 3D-HTM, texture-first decoding is applied in a common test condition. Therefore, when decoding a non-base texture view, the corresponding non-base depth view is not available. Therefore, depth information is estimated and used to perform BVSP.

Generally speaking, when a video coder performs BVSP to synthesize a reference texture picture, the video coder processes a block (e.g., a video unit) in a dependent texture picture. The dependent texture picture and the synthesized texture picture are in the same access unit but in different views. When the video coder processes a block of the dependent texture picture (i.e., the current block), the video coder may perform an NBDV process to identify the depth of the current block. That is, to estimate the depth information of the block, the video coder may first derive the depth of the current block from neighboring blocks.

Furthermore, when the video coder performs BVSP to synthesize a reference texture picture, the video coder may use the DV of the current block to identify a reference block in a reference depth picture. In other words, the video coder may then use the derived DV to obtain a depth block from the reference view. For example, the DV identified by the NBDV process may be represented as (dvx, dvy), and the current block position may be represented as (blockx, blocky). Furthermore, in this example, the video coder may extract a depth block at (blockx+dvx, blocky+dvy) in the depth image of the reference view. In this example, the obtained depth block has the same size as the current PU. The dependent texture picture and the reference depth picture are in the same access unit but in different views. The video coder may then perform a backward warping process to determine sample values of the synthesized picture based on sample values of the current block and sample values of the identified reference block of the reference picture. In other words, in this example, the video coder may use the extracted depth block to perform backward warping of the current PU.

As noted above, when the video coder performs BVSP, the video coder may perform an NBDV process to identify the DV for the current block. Furthermore, when the video coder performs BVSP, the video coder may use an improvement process similar to the improvement process described elsewhere in this disclosure to improve the DMV derived using the NBDV process. When the video coder performs a DV improvement process, the video coder may improve the DV based on depth values in a depth map in a reference view. In other words, depth may be used to improve either the DV or the DMV for BVSP. If the improved DV is coded using BVSP mode, the improved DV may be stored as a motion vector for a PU.

In some versions of 3D-HEVC, texture-first coding is applied. In texture-first coding, the video coder codes (e.g., encodes or decodes) a texture view component and then codes the corresponding depth view component (i.e., a depth view component with the same POC value and view identifier as the texture view component). Therefore, the non-base view depth view component is not available for coding the corresponding non-base view texture view component. In other words, when the video coder codes the non-base texture view component, the corresponding non-base depth view component is not available for use. Therefore, depth information can be estimated and used to perform BVSP.

FIG7 is a conceptual diagram illustrating depth block derivation from a reference view to perform BVSP prediction. In the example of FIG7 , a video coder is coding a current texture picture 70. Current texture picture 70 is labeled a "dependent texture picture" because it is dependent on a synthesized reference texture picture 72. In other words, the video coder may need to synthesize reference texture picture 72 in order to decode current texture picture 70. Reference texture picture 72 and current texture picture 70 are in the same access unit but in different views.

To synthesize reference texture picture 72, the video coder may process a block (i.e., a video unit) of current texture picture 70. In the example of FIG. 7 , the video coder is processing current block 74. While the video coder is processing current block 74, the video coder may perform an NBDV process to derive a DV for current block 74. For example, in the example of FIG. 7 , the video coder identifies DV 76 of block 78 adjacent to current video block 74. Identification of DV 76 is shown as step 1 of FIG. 7 . Furthermore, in the example of FIG. 7 , the video coder determines DV 78 for current block 74 based on DV 76. For example, DV 78 may be a copy of DV 76. Copying DV 76 is shown as step 2 of FIG.

The video coder may identify a reference video block 80 in a reference depth picture 82 based on DV 78 of the current block 74. Reference depth picture 82, the current texture picture 70, and the reference texture picture 72 may each be in the same access unit. Reference depth picture 82 and reference texture picture 72 may be in the same view. The video coder may determine texture sample values of the reference texture picture 72 based on the texture sample values of the current block 74 and the depth sample values of the reference depth block 80. The process of determining texture sample values may be referred to as backward warping. Backward warping is shown as step 3 of FIG. In this manner, FIG. 7 illustrates the three steps of locating a depth block from a reference view and subsequently using it for BVSP prediction.

The introduced BVSP mode is treated as a special inter-coded mode, and a flag indicating the use of BVSP mode should be maintained for each PU. Instead of signaling the flag in the bitstream, a new merge candidate for merge mode (BVSP merge candidate) is added to the merge candidate list, and the flag depends on whether the decoded merge candidate index corresponds to a BVSP merge candidate. The BVSP merge candidate is defined as follows:

1. Reference picture index of each reference picture list: -1

2. Motion vector for each reference picture list: improved disparity vector

The insertion position of BVSP merge candidates depends on the spatial neighboring blocks:

1. If any of the five spatial neighboring blocks is coded using BVSP mode, i.e., the maintained flag of the neighboring block is equal to 1, then the video coder considers the BVSP merge candidate as the corresponding spatial merge candidate and inserts the BVSP candidate into the merge candidate list. In one example, the video coder inserts the BVSP merge candidate into the merge candidate list only once.

2. Otherwise (none of the five spatial neighboring blocks are coded using the BVSP mode), the video coder may insert (single-board) the BVSP merge candidate into the merge candidate list just before the temporal merge candidate.

In some examples, during the combined bi-predictive merging candidate derivation process, the video coder should check additional conditions to avoid including BVSP merging candidates.

For each BVSP-coded PU whose size is represented by NxM, the video coder may further partition the PU into several sub-regions having a size equal to KxK (where K may be equal to 4). For each sub-region, the video coder may derive a separate DMV and may predict each sub-region from one block in the inter-view reference picture located by the derived DMV. In other words, the size of the motion compensation unit for the BVSP-coded PU may be set to KxK. In the common test condition, K is set to 4.

For each sub-region (4x4 block) within a PU coded in BVSP mode, the video coder may locate the corresponding 4x4 depth block in the reference depth view with the improved DV described above. The video coder may select the maximum value of the sixteen depth pixels in the corresponding depth block. The video coder may convert the maximum value to the horizontal component of the DMV and may set the vertical component of the DMV to 0.

FIG8 is a conceptual diagram illustrating an example prediction structure for a currently proposed temporal advanced residual prediction (ARP) for temporally predicted video blocks. As proposed in Zhang et al., "CE4: Advanced Residual Prediction for Multiview Coding," ITU-T SG16 WP 3 and ISO/IEC JTC1/SC 29/WG 11 Joint Collaborative Team on Video Coding Extension Development, 4th Meeting: Incheon, South Korea, April 20-26, 2013, document JCT3V-D0177 (MPEG No. m29008, hereinafter "JCT3V-D0177"), ARP applied to CUs with a partition mode equal to Part_2Nx2N was adopted in the 4th JCT3V meeting. JCT3V-D0177 can be downloaded from the following link:

http://phenix.it-sudparis.eu/jct3v/doc_end_user/current_document.php?id=862 . The entire contents of JCT3V -D0177 are incorporated herein by reference.

As shown in FIG. 8 , the video coder calls upon or identifies the following blocks in the prediction of the residual of the current video block 100 in the current picture 102 of the current (eg, dependent) view Vm.

1. Current video block 100 (in view V _m ): Curr

2. An inter-view reference video block 106 in the inter-view reference picture 108 of the reference/base view (V ₀ in FIG8 ): Base. The video coder derives the inter-view reference video block 106 (Curr) based on the DV 104 of the current video block 100. The video coder may determine the DV 104 using NBDV, as described above.

3. A temporal reference video block 112 in a temporal reference picture 114 in the same view (V _m ) as the current video block 100 (Curr): CurrTRef. The video coder derives the temporal reference video block 112 based on the TMV 110 of the current video block 100. The video coder may determine the TMV 100 using any of the techniques described herein.

4. A temporal reference video block 116 in a temporal reference picture 118 in a reference view (i.e., the same view as the inter-view reference video block 106 (Base)): BaseTRef. The video coder derives the temporal reference video block 116 in the reference view using the TMV 110 of the current video block 100 (Curr). A vector 120 of TMV+DV may identify the temporal reference video block 116 (BaseTRef) relative to the current video block 100 (Curr).

When the video encoder performs temporal inter-frame prediction on the current video block 100 based on the temporal reference video block 112 identified by the video decoder using TMV 110, the video encoder determines the pixel-by-pixel difference between the current video block 100 and the temporal reference video block 112 as a residual block. Without ARP, the video decoder will transform, quantize, and entropy encode the residual block. The video decoder will entropy decode the encoded video bitstream, perform inverse quantization and transform to derive the residual block, and apply the residual block to the reconstruction of the reference video block 112 to reconstruct the current video block 100.

Using ARP, the video coder determines a residual predictor block that predicts the value of the residual block (i.e., predicts the difference between the current video block 100 (Curr) and the temporal reference video block 112 (CurrTRef)). The video encoder may then only need to encode the difference between the residual block and the residual predictor block, thereby reducing the amount of information included in the encoded video bitstream used to encode the current video block 100. In the temporal ARP example of FIG8, a predictor for the residual of the current video block 100 is determined based on blocks in the reference/base view ( _V0 ) that correspond to the current video block 100 (Curr) and the temporal reference video block 112 (CurrTRef) and are identified by DV 104. The difference between these corresponding blocks in the reference view can be a good predictor for the residual, i.e., the difference between the current video block 100 (Curr) and the temporal reference video block 112 (CurrTRef). Specifically, the video coder identifies an inter-view reference video block 106 (Base) and a temporal reference video block 116 (BaseTRef) in a reference view and determines a residual predictor block based on the difference (BaseTRef-Base) between the inter-view reference video block 106 and the temporal reference video block 116, wherein a subtraction operation is applied to each pixel of the represented pixel array. In some examples, the video coder may apply a weighting factor w to the residual predictor. In such examples, the final predictor for the current block, i.e., the sum of the reference block and the residual predictor block, may be expressed as: CurrTRef+w*(BaseTRef-Base).

FIG9 is a conceptual diagram illustrating an example bidirectional prediction structure for temporal ARP of current video block 120 in current view (V _m ). The above description and FIG8 illustrate unidirectional prediction. When extending ARP to the case of bidirectional prediction, the video coder may apply the above techniques to one or both of the reference picture lists to identify the residual predictor block for current video block 120. Specifically, the video coder may examine one or both of the reference lists for current video block 100 to determine whether one of them contains a TMV that can be used for temporal ARP. In the example illustrated by FIG9 , current video block 120 is associated with TMV 130 pointing to a first temporal reference picture 134 in a first reference picture list (RefPicList0), and TMV 132 pointing to a second temporal reference picture 136 in a second reference picture list (RefPicList1).

In some examples, the video coder checks the reference picture lists according to a checking order to determine whether one of them includes a TMV that can be used for temporal ARP, and if the first list includes such a TMV, then the second list need not be checked according to the checking order. In some examples, the video coder checks two reference picture lists and, if both lists include a TMV, determines which TMV to use based on, for example, a comparison of a residual predictor generated using the TMV relative to a residual of the current video block. Notably, according to current proposals for ARP, when the current block uses an inter-view reference picture (in a different view) for one reference picture list, the residual prediction process is disabled.

9 , the video coder may use, for example, DV 124 identified for current video block 120 according to NBDV to identify a corresponding inter-view reference video block 126 (Base) in an inter-view reference picture 128 in a different reference view (V ₀ ) than current picture 122 but in the same access unit. The video coder may also use TMVs 130 and 132 for current video block 120 to identify a temporal reference block (BaseTRef) for inter-view reference video block 126 (Base) in respective temporal reference pictures of reference views in two reference picture lists (e.g., RefPicList0 and RefPicList1). In the example of FIG9 , the video coder identifies, based on TMVs 130 and 132 for current video block 120, a temporal reference video block (BaseTRef) 140 in a temporal reference picture 142 in a first reference picture list (e.g., RefPicList0) and a temporal reference video block (BaseTRef) 144 in a temporal reference picture 146 in a second reference picture list (e.g., RefPicList1).

The use of TMVs 130 and 132 for current video block 120 in the reference view is illustrated by the dashed arrows in FIG9 . In FIG9 , temporal reference video blocks 140 and 144 in the reference view are referred to as motion compensated reference blocks due to their identification based on TMVs 130 and 132. The video coder may determine a residual predictor block for current video block 120 based on a difference between temporal reference video block 140 and inter-view reference video block 126 or based on a difference between temporal reference video block 144 and inter-view reference video block 126.

The main procedure of the proposed temporal ARP on the decoder side can be described (see FIG9 ) as follows:

1. The video decoder obtains DV 124 as specified in current 3D-HEVC, for example, using NBDV pointing to the target reference view (V ₀ ). Then, in a picture 128 of a reference view within the same access unit, the video decoder identifies the corresponding inter-view reference video block 126 (Base) through DV 124 .

2. The video decoder reuses the motion information (e.g., TMV 130, 132) of the current video block 120 to derive motion information for the corresponding inter-view reference video block 126. The video decoder may apply motion compensation for the corresponding inter-view reference video block 126 based on the TMV 130, 132 of the current video block 120 and the derived reference pictures 142, 146 in the reference view of the reference video block 126 to identify a motion-compensated temporal reference video block 140, 144 (BaseTRef) and determine a residual predictor block by determining BaseTRef-Base. The relationship between the current block, the corresponding block (Base), and the motion compensated block (BaseTRef) is shown in Figures 8 and 9. In some examples, a reference picture in the reference view ( _V0 ) having the same POC (Picture Order Count) value as the reference picture of the current view ( _Vm ) is selected as the reference picture for the corresponding block.

3. The video decoder may apply the weighting factor w to the residual predictor block to obtain a weighted residual predictor block, and add the values of the weighted residual block to the predicted samples to reconstruct the current video block 100.

In the proposal for ARP, three weighting factors, 0, 0.5, and 1, can be used. The one that produces the minimum rate-distortion cost for the current CU is selected as the final weighting factor, and the corresponding weighting factor index (0, 1, and 2, which correspond to weighting factors of 0, 1, and 0.5, respectively) is transmitted in the bitstream at the CU level. All PU predictions in a CU share the same weighting factor. When the weighting factor is equal to 0, ARP is not used for the current CU.

Aspects of ARP for 3D-HEVC are described in Zhang et al., “3D-CE4: Advanced Residual Prediction for Multi-view Coding” (3rd Meeting of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11 Joint Collaborative Team on Development of Extensions for Video Coding: Geneva, Switzerland, January 17-23, 2013, document JCT3V-C0049 (MPEG No. m27784, hereinafter “JCT3V-C0049”). JCT3V-C0049 can be downloaded from the following link:

http://phenix.int-evry.fr/jct3v/doc_end_user/current_document.php?id=487 . The entire contents of JCT3V -C0049 are incorporated herein by reference.

In JCT3V-C0049, the reference pictures of different PUs coded with non-zero weighting factors may be different between different PUs (or current video blocks). Therefore, different pictures from the reference view may need to be accessed to generate motion-compensated blocks (BaseTRef) of corresponding inter-view reference video blocks (e.g., inter-view reference video blocks 106 and 126 in Figures 8 and 9) in the reference view (Base), such as temporal reference video blocks 116, 140, and 144 in Figures 8 and 9.

JCT3V-D0177 proposes to further simplify ARP by performing reference picture selection via motion vector scaling. For example, it is proposed that when the weighting factor is not equal to 0, the video coder scales the motion vector of the current PU towards a fixed picture before performing motion compensation for the residual generation process. In JCT3V-D0177, a fixed picture is defined as the first reference picture of each reference picture list (if they are from the same view). When the decoded motion vector does not point to a fixed picture, it is first scaled by the video coder and then used by the video coder to identify the CurrTRef and BaseTRef of the current video block. This reference picture used for ARP may be referred to as a target ARP reference picture. There may be two target ARP reference pictures corresponding to RefPicList0 and RefPicList1, respectively, which may be denoted as L0 target ARP reference picture and L1 target ARP reference picture, respectively.

According to JCT3V-C0049, the video coder applies a bilinear filter during the interpolation process of the corresponding block (Base) and its prediction block (BaseTRef), but applies a conventional 8/4 tap filter for the interpolation process of the current video block (Curr) (e.g., PU) and the prediction block (CurrTRef) of the current video block. JCT3V-D0177 proposes that the video coder always adopt a bilinear filter for such interpolation processes, regardless of whether the block is in the base view or the non-base view when ARP is applied.

In addition, according to existing proposals for ARP, the reference view of the ARP is identified by the view order index returned from the NBDV process. As described above, the video coder can use the NBDV process to determine the DV, such as DV 104 or 124, for identifying the corresponding inter-view reference video block (Base) (e.g., inter-view reference video blocks 106 and 126 in Figures 8 and 9). According to existing proposals for ARP, when the reference picture of a video block (PU) in a reference picture list is from a view different from the target reference view of the ARP (as identified by the view order index returned from the NBDV process), ARP is disabled for this reference picture list.

There may be problems associated with existing proposals for ARP in 3D-HEVC. For example, according to existing proposals, ARP only predicts the residual generated from temporal prediction when the current motion vector of the current video block refers to a reference picture in the same view. Therefore, ARP is not applicable when the current motion vector of the current video block refers to an inter-view reference picture, but syntax elements related to ARP are still transmitted.

As another example, a derived DV (e.g., as derived according to an NBDV process) may be less accurate than an explicit DMV, which is typically selected through rate-distortion optimization (RDO). Furthermore, as a decoding process, motion prediction (including inter-view prediction) occurs after DV generation, and ARP occurs after motion prediction. Therefore, when ARP is performed by a video coder, a more accurate TMV or DMV may be available that can be used to identify different blocks not considered in the current ARP. Nevertheless, as described above with reference to FIGs. 8 and 9 , existing proposals for temporal ARP use DVs derived via NBDV to identify corresponding inter-view reference video blocks.

This disclosure provides techniques that can address issues associated with existing proposals for ARP, including those discussed above, and thereby improve the coding efficiency of ARP. For example, a video coder (e.g., video encoder 20 and/or video decoder 30) implementing the techniques of this disclosure to code a current video block using ARP may identify a DMV from the current view to a reference view of the current video block and, based on the identification of the DMV, determine a residual predictor block for the current video block. In some examples, the DMV is a DMV used for inter-view prediction of the current video block, and the video coder may perform inter-view ARP for encoding the current video block. In other examples, the DMV is a DMV of a temporal reference video block in the same view as the current video block. In such examples, the DMV can be used in the temporal ARP of the current video block, replacing the DV derived for the current video block by NBDV.

FIG10 is a conceptual diagram of an example prediction structure for inter-view ARP of an inter-view predicted video block according to the techniques described in this disclosure. According to the example technique illustrated in FIG10 , a video coder (e.g., video encoder 20 and/or video decoder 30) can use inter-view residuals calculated in different access units to predict a residual of a current block that is inter-view predicted. Compared to proposals for ARP in which ARP is not performed when the motion vector of the current block is a DMV and is performed only when the motion vector of the current video block is a TMV, the example technique of FIG10 uses DMVs to perform ARP.

10 may be performed by a video coder, such as video encoder 20 or video decoder 30, when the motion vector of a current video block 150 (Curr) in a current picture 152 is a DMV 154, and an inter-view reference video block 156 (Base) in an inter-view reference picture 158 in a reference view (V0) contains at least one TMV 160. In some examples, DMV 154 may be a DV that is converted to a DMV to serve as an IDMVC for motion information prediction of the current video block 150.

The video coder uses DMV 154 of current video block 150 to identify inter-view reference video block 156 (Base) in inter-view reference picture 158. The video coder uses TMV 160 of inter-view reference video block 156 and an associated reference picture (e.g., temporal reference picture 164 in reference view (V ₀ )) along with the DMV to identify temporal reference video block 162 (BaseTRef) in temporal reference picture 164 in reference view (V ₀ ). Identification of temporal reference video block 162 (BaseTRef) based on TMV 160 and DMV 154 is represented by dashed vector 170 (TMV+DMV). The video coder also uses TMV 160 to identify temporal reference block 166 (CurrTRef) in temporal reference picture 168 in current view (V _m ). Temporal reference video block 162 (BaseTRef) in reference view (V ₀ ) and temporal reference video block 166 (CurrTRef) in current view (V _m ) may be in the same access unit, i.e., temporal reference picture 164 in reference view (V ₀ ) and temporal reference picture 168 in current view (V _m ) may be in the same access unit.

A video coder (e.g., video encoder 20 and/or video decoder 30) may then calculate an inter-view residual predictor block in a different access unit from current video block 150 based on the pixel-by-pixel difference between these latter two blocks (i.e., the difference between temporal reference video block 166 in the current view and temporal reference video block 164 in the reference view, or CurrTRef - BaseTRef). The difference signal (denoted as the inter-view residual predictor) may be used to predict the residual for current video block 150. The prediction signal for current video block 150 may be the sum of the inter-view predictor (i.e., inter-view reference video block 156 (Base)) and the predicted inter-view residual in the different access unit determined based on the difference between temporal reference video block 166 in the current view and temporal reference video block 164 in the reference view. In some examples, a weighting factor w is applied to the predicted inter-view residual in the different access units. In such examples, the prediction signal for current video block 150 may be: Base + w * (CurrTRef - BaseTRef).

In some examples, the video coder may determine a target reference picture in a target access unit for inter-view ARP, e.g., similar to the determination of a target reference picture for temporal ARP, as discussed above. In some examples, as discussed above with reference to JCT3V-D0177, the target reference picture for each reference picture list is the first reference picture in the reference picture list. In other examples, the target reference picture (e.g., target POC) of one or both reference picture lists may be signaled from video encoder 20 to video decoder 30, e.g., on a PU, CU, slice, picture, or other basis. In other examples, the target reference picture for each reference picture list is the temporal reference picture in the reference picture list having the smallest POC difference and a smaller reference picture index than the current block. In other examples, the target reference picture of both reference picture lists is the same.

If the picture containing the temporal reference video block in the reference view indicated by TMV 160 is in a different access unit (temporal instance) than the target ARP reference picture, the video coder may scale TMV 160 to the target reference picture (e.g., target reference picture 164) to identify a temporal reference video block 162 (BaseTRef) in the reference view for inter-view ARP. In these examples, the video coder locates temporal reference video block 162 in the access unit containing the target ARP reference picture. The video coder may scale TMV 160 by POC scaling. Furthermore, the scaled TMV is used to identify a temporal reference video block (CurrTRef) 166 in the current view that is located in the target ARP reference picture.

In some examples, the video coder scales TMV 160 to an LX (X is 0 or 1) target reference picture, where LX corresponds to the RefPicListX of the PU that includes the TMV. In some examples, the video coder may scale TMVs from either or both of RefPicList0 or RefPicList1 to an L0 or L1 target reference picture, respectively. In some examples, the video coder scales TMV 160 to an LX target reference picture, where X satisfies the condition that DMV 154 of current video block 150 (e.g., current PU) corresponds to RefPicListX.

Similarly, in some examples, the video coder scales DMV 154 to the target reference view of the ARP before identifying inter-view reference video blocks 156 in reference pictures 158 in the target reference view. The video coder may scale DMV 154 by view order difference scaling. The target reference view may be predetermined and known by video encoder 20 and video decoder 30, or may be signaled from video encoder 20 to video decoder 30, for example, on a PU, CU, slice, picture, or other basis.

In some examples of inter-view ARP, a video coder (e.g., video encoder 20 and/or video decoder 30) may derive a prediction signal for current block 150 using the same prediction structure illustrated in FIG. 10 and identified reference video blocks 156, 164, and 168, but determine a residual predictor block based on the difference between reference blocks 156 and 162 in the reference view rather than between reference blocks 162 and 166 in different access units. In such examples, the video coder may apply a weighting factor to the other sample array (e.g., the difference between reference blocks 156 and 162 in the reference view) and accordingly derive a prediction signal for current video block 150 as follows: CurrTRef+w*(Base-BaseTRef). In some examples of inter-view ARP, the video coder may use various interpolation filters, including bilinear filters, to derive reference video blocks 156, 162, and 166 when they are aligned with fractional pixel positions.

Although FIG10 illustrates an inter-view ARP example in which the TMV and associated reference pictures of an inter-view reference video block are used to identify temporal reference video blocks in the current and reference views, in other examples, other TMVs and associated reference pictures may be used to identify temporal reference video blocks in the current and reference views. For example, if the DMV of the current video block is from the first reference picture list of the current video block (e.g., RefPicList0 or RefPicList1), the video coder may use the TMV and associated reference pictures from the second reference picture list of the current video block (e.g., the other of RefPicList0 or RefPicList1). In such examples, the video coder may identify the temporal reference video block in the current view in the reference picture associated with the TMV, or scale the TMV to the target access unit and target reference picture of the ARP to identify the temporal reference video block in the current view. In such examples, the video coder may identify the temporal reference video block in the reference view in a reference picture in the same access unit as the reference picture in which the temporal reference video block in the current view is located. In other examples, instead of the TMV of an inter-view reference video block or the TMV of another reference picture list of the current video block, the video coder may similarly use the TMV and associated reference pictures derived from motion information of spatially or temporally neighboring video blocks of the current video block to identify temporal reference video blocks in the current and reference views of the ARP.

FIG10 illustrates an example of inter-view ARP according to this disclosure. As discussed above, according to existing proposals for temporal ARP, the DV of the current video block, such as that derived from NBDV, is used to identify inter-view reference video blocks in a reference view. According to the techniques of this disclosure, the accuracy of the temporal residual predictor calculated in the reference view of temporal ARP can be increased by replacing the DV with the DMV of the temporal reference block in the current view (CurrTRef) if it contains at least one DMV.

FIG11 is a conceptual diagram of an example prediction structure for a temporal ARP of a temporally predicted current video block 180 in a current picture 182 using a DMV 190 for a temporal reference block in a current view (CurrTRef) according to the techniques described in this disclosure. According to the example of FIG11 , a video coder (e.g., video encoder 20 and/or video decoder 30) temporally predicts current video block 180 using TMV 184 identifying a temporal reference video block 186 in a temporal reference picture 188. The video coder determines whether temporal reference video block 186 contains at least one DMV, such as DMV 190, for inter-view prediction of temporal reference video block 186. In some examples, DMV 190 may be an IDMVC for motion information prediction of temporal reference video block 186.

The video coder may use DMV 190 instead of the DV of current video block 180 to identify either or both of the following: inter-view reference video block 196 (Base) within reference picture 198 in reference view (V0) or temporal reference video block 194 (BaseTRef) in temporal reference picture 194 in reference view (V0). Identification of temporal reference video block 194 based on TMV 184 and DMV 190 is illustrated by vector 200, which is labeled TMV+DMV. In some examples, when the video coder uses the DMV to replace the DV of the NBDV from the temporal ARP, the video coder may also replace the view order index returned from the NBDV process with the view order index associated with the selected DMV. Additionally, in some examples, the video coder may not select the DMV associated with the temporal reference video block 186 of the temporal ARP of the current video block 180 if the DMV derived using BVSP mode would replace the DV from the NBDV. The video coder may determine a temporal residual predictor block for current video block 180 using identified reference video blocks 186 , 192 , and 196 , as described above with reference to blocks 106 , 112 , and 116 in FIG. 8 .

In some examples, if decoded TMV 184 of current video block 180 points to a reference picture in a different access unit (temporal instance) than the target ARP reference picture, the video coder may scale TMV 184 to target ARP reference picture 188 and locate temporal reference video block 186 (CurrTRef) in the target ARP reference picture via scaled TMV 184, e.g., using POC scaling. In such examples, the video coder may derive DMV 190 from temporal reference video block 186 (CurrTRef) as identified by scaled TMV 184. In some examples, when the video coder scales TMV 184 to identify temporal reference video block 186 (CurrTRef) in picture 188 that belongs to the same access unit as the target ARP picture, another temporal reference video block identified by TMV 184 without scaling, namely, CurrTempRef, may be identified. In such examples, the video coder may use the DMV from this temporal reference video block (CurrTempRef), if available, to replace the DV of the temporal ARP of current video block 180. In some examples, the video coder recognizes and uses CurrTempRef only when there is no DMV associated with temporal reference video block 186 (CurrTRef). In some examples, other DMVs of the coded block may be used to replace the DV from the NBDV.

FIG12 is a conceptual diagram illustrating an example technique for identifying a TMV or DMV in or near a video block according to the techniques described in this disclosure. As discussed above with respect to FIG10 and 11, a video coder (e.g., video encoder 20 and/or video decoder 30) identifies TMVs and DMVs in accordance with the techniques described in this disclosure to implement inter-view ARP and temporal ARP. In some examples, the video coder identifies TMVs and DMVs in or near a current video block or an inter-view or temporal reference video block, which may be a region within a reference picture having the same width x height size as the current video block (e.g., the current PU).

FIG12 illustrates a width x height block 210. Block 210 may be a current video block or an inter-view or temporal reference video block, which may be a region within a reference picture having the same size as the current video block. FIG12 also illustrates a block 212 adjacent to or including the center of block 210, and a block 214 adjacent to or including the lower right portion of block 210.

In some examples, for temporal or inter-view ARP, the video coder considers (e.g., only considers) the motion vector (e.g., TMV or DMV) and associated reference index associated with the PU or other block containing the center position of the block (e.g., block 212 within block 210). In some examples, the video coder considers (e.g., only considers) the motion information (including motion vectors and reference indexes) of two blocks containing the lower right (with coordination relative to the upper left pixel of (width, height)) and the center (with coordination relative to the upper left pixel of (width/2, height/2)) pixels of block 210. Referring to FIG. 12 , blocks 214 and 212 may be examples of blocks containing the lower right and center pixels of block 210, respectively. Blocks 212 and 214 may be N×N, which may be the maximum block granularity that can contain up to one motion vector corresponding to each reference picture list, e.g., N×N may be 4×4. Blocks 212 and 214 may be checked for the DMV or TMV for the ARP in any order.

In some examples, assuming that the current video block (e.g., the current PU) has coordinates (x, y) and the vector used to identify the reference video block (v[0], v[1]) is from a motion vector (TMV or DMV), the video coder may convert the DMV to v[i]=(mv[i]+2)>>2, where i is equal to 0 or 1, respectively, or v[i]=mv[i]>>2. In such examples, the video coder may identify blocks 212 and 214 as a block (e.g., a 4x4 block) covering pixels with coordinates (x+v[0]+width/2, y+v[1]+height/2) and a block covering pixels with coordinates (x+v[0]+width, y+v[1]+height), respectively. In some examples, the video coder may identify one or both of the center block 212 and the bottom right block 214 by coordinates shifted by (-1, -1), which correspond to (x+v[0]+width/2-1, y+v[1]+height/2-1) and (x+v[0]+width-1, y+v[1]+height-1), respectively.

In some examples, the video coder may check blocks 212 and 214 for available TMVs or DMVs according to the order of checking. In some examples, the video coder may first check center block 212 and use the DMV or TMV associated with the center block of the ARP if such a motion vector is available. In these examples, if such a motion vector is not available from center block 212, the video coder may check bottom-right block 214 for the TMV or DMV of the ARP.

In some examples, the video coder may check the reference picture lists for the appropriate motion vector for the ARP in order to find blocks 212, 214. For example, the video coder may check RefPicList0 and use the DMV or TMV associated with RefPicList0 if such a motion vector is available. In these examples, if such a motion vector is not available from RefPicList0, the video coder may check RefPicList1 for the TMV or DMV for the ARP.

In some examples, the video coder may consider (e.g., only consider) motion vectors associated with PUs containing the center and one or more of the four corner positions of the block. The video coder may consider PUs in order based on priority, and once a motion vector is found, other PUs may not be considered. The priorities of these different positions may be defined in one example as: center, top left, top right, bottom left, and bottom right of the block.

In some examples, the video coder may be configured to consider all motion information associated with a block. In some examples, once a TMV or DMV for ARP is found within block 210, the video coder may not check for additional motion vectors. The priority of checking PUs within block 210 for motion vectors may be, for example, a raster scan order or a spiral scan. An example of a spiral scan order for scanning a block (e.g., a 4x4 block) for motion vectors is depicted in Figures 13A-13D.

In some examples, when checking block 210 for TMVs for ARP, the video coder may only consider TMVs that point to reference pictures in the same access unit as the target ARP reference picture. In some examples, when checking reference block 210 for DMVs for ARP, the video coder may only consider DMVs that point to inter-view reference pictures in the same view as the view indicated by the DMV or DV of the current video block. In some examples, the video coder first extends block 210 to the block associated with the PU and looks for a TMV or DMV within the extended block that will become the TMV or DMV for ARP. In some examples, if no TMV or DMV is found in block 210, the video coder performs ARP using a zero motion vector, or does not perform ARP. In some examples, when the video coder uses a 0 motion vector to identify two temporal reference video blocks in the current and reference views, the video coder may use a target reference picture of RefPicListX, where X may be 0 or 1 and indicates which list to call for inter-view prediction, e.g., which list includes the DMV.

As discussed above, for example, with reference to FIG12 , a video coder can identify TMVs or DMVs in a given block (e.g., a 4×4 block within block 210) that contains only up to two sets of motion information. One set of motion information corresponds to a first reference picture list for the given block, such as reference picture list 0 (RefPicList0), and the other set corresponds to a second reference picture list for the given block, such as reference picture list 1 (RefPicList1). Each set of motion information includes a motion vector and a reference index.

In some examples, the video coder considers only the motion information corresponding to RefPicList0 for identifying the TMV or DMV of the ARP for the current video block. In other examples, the video coder considers only the motion information corresponding to RefPicList1 for identifying the TMV or DMV of the ARP for the current video block. In other examples, the video coder first considers the motion information corresponding to RefPicListX. If the motion information corresponding to RefPicListX does not include a suitable TMV or DMV for the ARP, the video coder then considers the motion information corresponding to RefPicListY (where Y is equal to 1-X).

In some examples, X is equal to 0. In some examples, X is equal to 1. In some examples, X is equal to Z, where Z corresponds to the reference picture list that contains the motion vector (TMV or DMV) of the current video block. For example, if the motion vector belonging to the current video block (e.g., the current PU) corresponds to RefPicList0, then Z is 0. If the motion vector belonging to the current video block (e.g., the current PU) corresponds to RefPicList1, then Z is 1. In some examples, the video coder corresponds only to the motion information of RefPicListZ.

Another example technique for improving the accuracy of temporal ARP (e.g., the accuracy of the temporal residual calculated in the reference view) by replacing the DV of the current video block includes replacing the DV derived, for example, from NBDV, with the DV derived via the co-located depth block of the temporal reference block of the current block (CurrTRef). A video coder (e.g., video encoder 20 and/or video decoder 30) may derive the DV via the co-located depth block of the temporal reference block of the current block (CurrTRef) using techniques similar to or the same as those used to derive the DV of the current video block for BVSP (as described above with respect to FIG. 7).

The video coder may use a DV derived from a co-located depth block of a temporal reference block of a current block (CurrTRef) instead of a current DV derived from, for example, NBDV, to derive any one or both of the reference blocks in the reference view for ARP. For example, the video coder may use the DV derived from a co-located depth block of the temporal reference block of the current block (CurrTRef) to identify one or both of an inter-view reference block of the current block in a reference view (Base) or a temporal reference block in a reference view (BaseTRef). The video coder may identify the temporal reference block in the reference view (BaseTRef) by adding the TMV of the current block to the DV derived from the co-located depth block of the temporal reference block of the current block (CurrTRef).

As discussed above, in some examples, if the decoded TMV of the current block points to a reference picture in a different access unit (temporal instance) than the target ARP reference picture, the video coder may scale the TMV to the target ARP reference picture and locate CurrTRef via the scaled TMV. In these examples, the video coder derives the DV from the co-located depth block of the temporal reference block of the current block (CurrTRef), as identified by the scaled TMV. Additionally, as discussed above, in some examples, when the TMV is scaled to identify CurrTRef in a picture belonging to the same access unit as the target ARP picture, the video coder may identify another temporal reference block identified by the TMV without scaling (i.e., CurrTempRef may be identified), and the DV derived from the co-located depth block of CurrTempRef (if available) may be used to replace the DV. In some examples, the video coder need only identify and use CurrTempRef when it cannot be derived via the co-located depth block of the temporal reference block of the current block (CurrTRef).

A video coder (e.g., video encoder 20 and/or video decoder 30) may derive DV from the co-located depth block of the temporal reference block of the current block (CurrTRef) in any of a variety of ways. In some examples, the video coder directly uses only one sample within the co-located depth block and converts the associated depth value into DV for the temporal ARP. In some examples, the single sample of the co-located depth block used to derive the DV for the temporal ARP is a pixel located at the center of the co-located depth block, e.g., at (W/2, H/2) relative to the top-left sample of a depth block of size WxH.

In some examples, the video coder uses several selective samples within the co-located depth block to determine a representative depth value, for example, via a mathematical function. In one example, the video coder selects four corner depth samples. In another example, the video coder selects depth samples within the co-located depth block based on neighboring depth samples of the depth block. For example, when the neighboring depth samples show a horizontal edge, the video coder may select only two corner pixels at the first row. In some examples, all depth samples within the co-located depth block may be used to determine a representative depth value via a mathematical function. In some examples, the video coder may determine a representative depth value based on selected (or all) depth values from the co-located depth block by, for example, determining the maximum, average, or median of the selected depth values or applying some other function to the selected depth values.

In some examples, the video coder may apply the temporal ARP techniques described above involving DV derived via a co-located depth block of a temporal reference block of a current block (CurrTRef) when decoding of a texture view independent of an associated depth view is not required. When decoding of a texture view independent of an associated depth view is required, the video coder may apply other ARP techniques described herein, such as those described with respect to FIGs. 10 and 11.

When both temporal and inter-view ARP are enabled, the weighting factor signaling condition for ARP may be changed from checking whether all reference pictures are inter-view reference pictures to simply checking whether the current picture is a random access picture (IRAP, with NAL unit types 15 to 22 (inclusive): BLA_W_LP, BLA_W_RADL, BLA_N_LP, IDR_W_RADL, IDR_N_LP, or CRA_NUT). Therefore, in some examples, the video encoder (e.g., video encoder 20) signals the weighting factor if the current CU is an inter-coded CU that does not belong to an IRAP picture. In such examples, the video encoder never transmits the weighting factor when the picture is a random access picture. In other examples, video encoder 20 additionally signals that the weighting factor is for an inter-coded CU that belongs to an IRAP picture if at least one of its reference pictures (which may be only inter-view reference pictures) has an inter-view reference picture in any of its reference picture lists. In such examples, the video coder may perform ARP for inter-view residual prediction for pictures within an access unit.

For the example where a video encoder (e.g., video encoder 20) signals a weighting factor when the current CU is an inter-coded CU that does not belong to an IRAP picture, the syntax table for coding_unit changes as highlighted below. Additions relative to 3D-HEVC Test Model 4 are underlined , and deletions are shown in text.

In addition, remove the variable TempRefPicInListsFlag and the related export process of TempRefPicInListsFlag, as shown below:

H.8.3.7 Derivation process of alternative target reference index for TMVP in merge mode

●This process is called when the current slice is a P or B slice.

The variables AltRefIdxL0 and AltRefIdxL1 are set equal to -1; and the following applies for X in the range of 0 to 1, inclusive:

○ When X is equal to 0 or the current slice is a B slice, the following applies:

■zeroIdxLtFlag＝RefPicListX[0] is a short-term reference picture? 0:1

■For (i=1; i<=num_ref_idx_lX_active_minus1&&AltRefIdxLX==-1; i++)

●if ((zeroIdxLtFlag&&RefPicListX[i] is a short-term reference picture)||

○(!zeroIdxLtFlag&&RefPicListX[i] is a long-term reference picture))

○AltRefIdxLX＝i

■

Existing proposals for temporal ARP disable ARP when NBDV does not return a usable DV for the current video block. However, as discussed above, the present invention provides techniques for ARP that do not rely on the DV derived from NBDV. Therefore, in some examples according to the present invention, instead of always disabling ARP when NBDV does not return a usable DV, the video decoder may enable ARP in at least some situations where NBDV does not return a usable DV. For example, the video decoder (e.g., video encoder 20 and/or video decoder 30) may enable temporal ARP when a temporal reference video block (CurrTRef) covers at least one DMV. As another example, the video decoder may enable temporal ARP when a temporal reference video block (CurrTRef) covers at least one DMV and the corresponding block is not coded in BVSP mode. In such examples, the video decoder may apply temporal ARP using DMVs instead of DVs, such as described above with respect to FIG. 11. As another example, the video coder may enable inter-view ARP if the current reference picture is an inter-view reference picture, e.g., as described above with respect to FIG 10. One or more constraints may be given at the video decoder such that when NBDV does not return a usable DV and one or more of the above conditions are not true, the weighting factor w for ARP will be set to 0.

FIG14 is a block diagram illustrating an example video encoder 20 that may implement the techniques described in this disclosure. The video encoder 20 may perform intra- and inter-coding on video blocks within a video slice. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture. Inter-coding relies on temporal or inter-view prediction to reduce or remove redundancy in video within adjacent frames or pictures of a video sequence. Intra-mode (I-mode) may refer to any of several spatially-based compression modes. Inter-modes, such as unidirectional prediction (P-mode) or bidirectional prediction (B-mode), may include any of several temporally-based compression modes.

14 , video encoder 20 includes a partitioning unit 235, a prediction processing unit 241, a reference picture memory 264, a summer 250, a transform processing unit 252, a quantization processing unit 254, and an entropy encoding unit 256. Prediction processing unit 241 includes a motion estimation unit 242, a motion compensation unit 244, an advanced residual prediction (ARP) unit 254, and an intra-prediction processing unit 246. For video block reconstruction, video encoder 20 also includes an inverse quantization processing unit 258, an inverse transform processing unit 260, and a summer 262. A deblocking filter (not shown in FIG. 14 ) may also be included to filter block boundaries to remove blocking artifacts from the reconstructed video. If necessary, the deblocking filter will typically filter the output of summer 262. In addition to the deblocking filter, additional loop filters (in-loop or post-loop) may also be used.

In various examples, units of video encoder 20 may be tasked with performing the techniques of this disclosure. Also, in some examples, the techniques of this disclosure may be divided among one or more of the units of video encoder 20. For example, ARP unit 245 may perform the techniques of this disclosure alone or in conjunction with other units of the video encoder, such as motion estimation unit 242 and motion compensation unit 244.

14 , video encoder 20 receives video data, and partitioning unit 235 partitions the data into video blocks. This partitioning may also include partitioning into slices, tiles, or other larger units, as well as video block partitioning according to a quadtree structure of LCUs and CUs, for example. Video encoder 20 generally illustrates components that encode video blocks within a video slice to be encoded. The slice may be divided into multiple video blocks (and possibly into sets of video blocks called tiles).

Prediction processing unit 241 may select one of a plurality of possible coding modes for the current video block, e.g., one of a plurality of intra-coding modes or one of a plurality of inter-coding modes, based on the error results (e.g., coding rate and distortion level). Prediction processing unit 241 may provide the resulting intra-coded or inter-coded block to summer 250 to generate residual block data and to summer 262 to reconstruct the encoded block for use as a reference picture.

Intra-prediction unit 246 within prediction processing unit 241 performs intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression. Motion estimation unit 242 and motion compensation unit 244 within prediction processing unit 241 perform inter-predictive coding of the current video block relative to one or more predictive blocks in one or more reference pictures to provide, for example, temporal compression.

Motion estimation unit 242 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. Motion estimation unit 242 and motion compensation unit 244 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation performed by motion estimation unit 242 is the process of generating motion vectors, which estimate the motion of video blocks. For example, a motion vector may indicate the displacement of a PU of a video block within a current video frame or picture relative to a predictive block within a reference picture.

A predictive block is a block that is found to closely match the PU of the video block to be coded in terms of pixel difference, which may be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. In some examples, video encoder 20 may calculate values for sub-integer pixel positions of a reference picture stored in reference picture memory 264. For example, video encoder 20 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Thus, motion estimation unit 242 may perform a motion search relative to full-pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

Motion estimation unit 242 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0 or RefPicList0) or a second reference picture list (List 1 or RefPicList1), each of which identifies one or more reference pictures stored in reference picture memory 264. Motion estimation unit 242 may send the calculated motion vector to entropy encoding unit 256 and motion compensation unit 246.

Motion compensation performed by motion compensation unit 244 may involve extracting or generating a predictive block based on a motion vector determined by motion estimation (possibly performing interpolation with sub-pixel precision). Upon receiving the motion vector for the PU of the current video block, motion compensation unit 244 may locate the predictive block to which the motion vector points in one of the reference picture lists. Video encoder 20 forms a residual video block by subtracting the pixel values of the predictive block from the pixel values of the current video block being coded, thereby forming pixel difference values. The pixel difference values form residual data for the block and may include both luma and chroma difference components. Summer 250 represents the one or more components that perform this subtraction operation. Motion compensation unit 244 may also generate syntax elements associated with the video block and video slice for use by video decoder 30 when decoding the video block of the video slice.

As an alternative to the inter-frame prediction performed by the motion estimation unit 242 and the motion compensation unit 244 as described above, the intra-frame prediction unit 246 can perform intra-frame prediction on the current block. Specifically, the intra-frame prediction processing unit 246 can determine the intra-frame prediction mode used to encode the current block. In some examples, the intra-frame prediction processing unit 246 can encode the current video block using various intra-frame prediction modes, for example during separate encoding passes, and the intra-frame prediction module 246 (or the prediction processing unit 241 in some examples) can select an appropriate intra-frame prediction mode to use from a test mode. For example, the intra-frame prediction processing unit 246 can use rate-distortion analysis to calculate rate-distortion values for various tested intra-frame prediction modes and select the intra-frame prediction mode with the best rate-distortion characteristics from the tested modes. Rate-distortion analysis generally determines the amount of distortion (or error) between the encoded block and the original unencoded block encoded to produce the encoded block, as well as the bit rate (i.e., number of bits) used to produce the encoded block. Intra-prediction processing unit 246 may calculate ratios from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

In any case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 246 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 256. Entropy encoding unit 256 may encode the information indicating the selected intra-prediction mode according to the techniques of this disclosure. Video encoder 20 may include configuration data in the transmitted bitstream that may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also called codeword mapping tables), definitions of contexts for encoding various blocks, and indications of the most probable intra-prediction mode, intra-prediction mode index tables, and modified intra-prediction mode index tables to use for each of the contexts.

After prediction processing unit 241 generates a prediction block for the current video block via inter-prediction or intra-prediction, video encoder 20 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be contained in one or more TUs and applied to transform processing unit 252. Transform processing unit 252 transforms the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform. Transform processing unit 252 may convert the residual video data from the pixel domain to a transform domain, such as the frequency domain.

Transform processing unit 252 may send the resulting transform coefficients to quantization processing unit 254. Quantization processing unit 254 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the transform coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization processing unit 254 may then perform a scan of the matrix containing the quantized transform coefficients. Alternatively, entropy encoding unit 256 may perform the scan.

After quantization, entropy encoding unit 256 performs entropy encoding on the quantized transform coefficients. For example, entropy encoding unit 256 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method or technique. Following entropy encoding by entropy encoding unit 256, the encoded video bitstream may be transmitted to video decoder 30 or archived for later transmission or retrieval by video decoder 30. Entropy encoding unit 256 may also entropy encode the motion vectors and other syntax elements for the current video slice being coded.

Inverse quantization processing unit 258 and inverse transform processing unit 260 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, for later use as a reference block of a reference picture. Motion compensation unit 244 may calculate a reference block by adding the residual block to a predictive block of one of the reference pictures in one of the reference picture lists. Motion compensation unit 244 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for motion estimation. Summer 262 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 244 to generate a reference block for storage in reference picture memory 264. The reference block may be used by motion estimation unit 242 and motion compensation unit 244 as a reference block for inter-frame prediction of a block in a subsequent video frame or picture.

Video encoder 20 (e.g., ARP unit 245 of video encoder 20) may perform any of the ARP techniques, such as the inter-view or temporal ARP techniques described herein. For example, if prediction processing unit 241 and/or motion estimation unit 242 performs inter-view prediction on a current video block (e.g., predicts the current video block based on a reference block from a reference picture in a different reference view than the current video block using a DMV), ARP unit 245 may identify a DMV associated with the current video block for inter-view prediction of the current video block. In some examples, the DMV may be a DV that is converted to an IDMVC for motion information prediction of the current video block.

Based on the DMV, ARP unit 245, alone or in conjunction with motion compensation unit 244, may also identify an inter-view reference video block (Base) and the TMV of the inter-view reference video block, which may have been previously determined by motion estimation unit 242 during prediction of the inter-view reference video block (Base). Based on the TMV, ARP unit 245, alone or in conjunction with motion compensation unit 244, may identify a temporal reference video block in a reference view (BaseTRef) and a temporal reference video block in a current view (CurrTRef). ARP unit 245 may determine an inter-view residual predictor for the current video block based on the difference (CurrTRef-BaseTRef) between the temporal reference video block in the current view and the temporal reference video block in the reference view. ARP unit 245 may apply a weighting factor w to the difference (CurrTRef-BaseTRef) and may determine the inter-view predictor block for the current video block as Base+w*(CurrTRef-BaseTRef), as described herein, for example, with respect to FIG.

As another example, if prediction processing unit 241 and/or motion estimation unit 242 temporally predicts the current video block (e.g., predicts the current video block based on a reference block from a reference picture in a different access unit but the same view as the current video block using a TMV), ARP unit 245 may identify the TMV. Based on the TMV, ARP unit 245, alone or in conjunction with motion compensation unit 244, may also identify a temporal reference video block (CurrTRef) and a DMV for the temporal reference video block, which may have been previously determined by motion estimation unit 242 during prediction of the temporal reference video block (CurrTRef). Based on the DMV, ARP unit 245, alone or in conjunction with motion compensation unit 244, may identify a temporal reference video block in a reference view (BaseTRef) and an inter-view reference video block in a reference view (Base). ARP unit 245 may determine a temporal residual predictor for the current video block based on the difference between the reference video blocks in the reference views (Base-BaseTRef). ARP unit 245 may apply a weighting factor w to the difference (Base-BaseTRef) and may determine the temporal predictor block for the current video block as CurrTRef+w*(Base-BaseTRef), as described herein, for example, with respect to FIG. 11 .

In any of the above examples, ARP unit 245, motion compensation unit 244, and/or prediction processing unit 241, or any component of video encoder 20, may provide the inter-view predictor block to summer 250, which determines the residual to be encoded in the encoded video bitstream for the current video block. Additionally, ARP unit 245 may scale the TMV and DMV, or perform any of the functions described herein for ARP in accordance with the techniques of this disclosure.

In this manner, video encoder 20 may be configured to implement the example ARP techniques of this disclosure to encode video blocks. For example, video encoder 20 may be an example of a video encoder configured to perform a method for encoding inter-view advanced residual prediction of video data, the method comprising identifying a DMV for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on an inter-view reference video block in a reference view and in the same access unit as the current video block. The method further comprises identifying a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block; identifying a temporal reference video block in an associated reference picture in the reference view based on the TMV for the inter-view reference video block; and identifying a temporal reference video block in the current view based on the TMV for the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The method further includes determining a residual predictor block for the current video block based on a difference between a temporal reference video block in the current view and a temporal reference video block in a reference view; and encoding an encoded video bitstream that encodes video data to identify a DMV and a residual block for the current video block, wherein the residual block identified by the encoded video bitstream comprises a difference between the inter-view reference video block and the residual predictor block for the current video block.

Video encoder 20 may also be an example of a video coder that includes a memory configured to store an encoded video bitstream of encoded video data and one or more processors. The one or more processors of a video coder, such as video encoder 20, may be configured to identify a DMV for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on inter-view reference video blocks in a reference view and in the same access unit as the current video block. The one or more processors are further configured to identify a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block, identify a temporal reference video block in an associated reference picture in the reference view based on the TMV for the inter-view reference video block, and identify a temporal reference video block in the current view based on the TMV for the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The one or more processors are further configured to: determine a residual predictor block for the current video block based on a difference between a temporal reference video block in the current view and a temporal reference video block in a reference view; and decode the encoded video bitstream to identify a DMV and a residual block for the current video block, wherein the residual block identified by decoding the encoded video bitstream comprises a difference between the inter-view reference video block and the residual predictor block for the current video block.

FIG15 is a block diagram illustrating an example video decoder 30 that may implement the techniques described in this disclosure. In the example of FIG15 , video decoder 30 includes an entropy decoding unit 280, a prediction processing unit 281, an inverse quantization processing unit 286, an inverse transform unit 288, a summer 290, and a reference picture memory 292. Prediction processing unit 281 includes a motion compensation unit 282, an ARP unit 283, and an intra-prediction unit 284. In some examples, video decoder 30 may perform a decoding pass that is generally reciprocal to the encoding pass described with respect to video encoder 20 of FIG14 .

In various examples, units of video decoder 30 may be tasked with performing the techniques of this disclosure. Also, in some examples, the techniques of this disclosure may be divided among one or more of the units of video decoder 30. For example, ARP unit 283 may perform the techniques of this disclosure alone or in conjunction with other units of the video encoder, such as motion compensation unit 282.

During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video slice and associated syntax elements from video encoder 20. Entropy decoding unit 280 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. Entropy decoding unit 280 forwards the motion vectors and other syntax elements to prediction processing unit 281. Video decoder 30 may receive syntax elements at the video slice level and/or the video block level.

When the video slice is coded as an intra-coded (I) slice, intra-prediction unit 284 of prediction processing unit 281 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When the video frame is coded as an inter-coded (i.e., B or P) slice, motion compensation unit 282 of prediction processing unit 281 generates a predictive block for the video block of the current video slice based on the motion vector and other syntax elements received from entropy decoding unit 280. The predictive block may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct reference frame lists RefPicList0 and RefPicList1 using a default construction technique or any other technique based on the reference pictures stored in reference picture memory 292.

Motion compensation unit 282 determines prediction information for a video block of the current video slice by parsing motion vectors and other syntax elements, and uses the prediction information to generate a predictive block for the current video block being decoded. For example, motion compensation unit 282 uses some of the received syntax elements to determine the prediction mode (e.g., intra-prediction or inter-prediction) used to code the video block of the video slice, the inter-prediction slice type (e.g., B-slice or P-slice), construction information for one or more of the slice's reference picture lists, motion vectors for each inter-coded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information used to decode the video blocks in the current video slice.

Motion compensation unit 282 may also perform interpolation based on interpolation filters. Motion compensation unit 282 may calculate interpolated values for sub-integer pixels of a reference block using interpolation filters used by video encoder 20 during encoding of the video block. In this case, motion compensation unit 282 may determine the interpolation filters used by video encoder 20 from received syntax information elements and use the interpolation filters to produce a predictive block.

Inverse quantization processing unit 286 inverse quantizes, or dequantizes, the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 280. The inverse quantization process may include using quantization parameters calculated by video encoder 20 for each video block in a video slice to determine the degree of quantization and, likewise, the degree of inverse quantization that should be applied. Inverse transform processing unit 288 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients to produce residual blocks in the pixel domain.

After motion compensation unit 282 generates a predictive block for the current video block based on the motion vector and other syntax elements, video decoder 30 forms a decoded video block by summing the residual block from inverse transform processing unit 288 with the corresponding predictive block generated by motion compensation unit 282. Summer 290 represents one or more components that can perform this summing operation. Optionally, a deblocking filter may also be applied to the decoded block to remove blocking artifacts. Other loop filters (either in the coding loop or after the coding loop) may also be used to smooth pixel transitions or otherwise improve video quality. The decoded video blocks in a given frame or picture are then stored in reference picture memory 292, which stores reference pictures for subsequent motion compensation. Reference picture memory 292 also stores the decoded video for later presentation on a display device (e.g., display device 32 of FIG. 1 ).

Video decoder 30 (e.g., ARP unit 283 of video decoder 30) may perform any of the ARP techniques, such as the inter-view or temporal ARP techniques described herein. For example, if prediction processing unit 281 and/or motion compensation unit 282 uses DMVs for inter-view prediction of a current video block based on syntax elements recovered from the encoded video bitstream by entropy decoding unit 280, ARP unit 283 may identify the DMV associated with the current video block for inter-view prediction of the current video block. In some examples, the DMV may be a DV that is converted to IDMVC for motion information prediction of the current video block.

Based on the DMV, ARP unit 283, alone or in conjunction with motion compensation unit 282, may also identify an inter-view reference video block (Base) and the TMV of the inter-view reference video block, which may have been previously determined by motion estimation unit 282 during prediction of the inter-view reference video block (Base). Based on the TMV, ARP unit 283, alone or in conjunction with motion compensation unit 282, may identify a temporal reference video block in a reference view (BaseTRef) and a temporal reference video block in a current view (CurrTRef). ARP unit 283 may determine an inter-view residual predictor for the current video block based on the difference (CurrTRef-BaseTRef) between the temporal reference video block in the current view and the temporal reference video block in the reference view. ARP unit 283 may apply a weighting factor w to the difference (CurrTRef-BaseTRef) and may determine the inter-view predictor block for the current video block as Base+w*(CurrTRef-BaseTRef), as described herein, for example, with respect to FIG.

As another example, if prediction processing unit 281 and/or motion compensation unit 282 uses a TMV to temporally predict the current video block based on syntax elements recovered from the encoded video bitstream by entropy decoding unit 280, ARP unit 283 may identify the TMV. Based on the TMV, ARP unit 283, alone or in conjunction with motion compensation unit 282, may also identify a temporal reference video block (CurrTRef) and a DMV for the temporal reference video block, which may have been previously determined by motion estimation unit 282 during prediction of the temporal reference video block (CurrTRef). Based on the DMV, ARP unit 283, alone or in conjunction with motion compensation unit 282, may identify a temporal reference video block in a reference view (BaseTRef) and an inter-view reference video block in a reference view (Base). ARP unit 283 may determine a temporal residual predictor for the current video block based on the difference between the reference video blocks in the reference views (Base-BaseTRef). ARP unit 283 may apply a weighting factor w to the difference (Base-BaseTRef) and may determine the temporal predictor block for the current video block as CurrTRef+w*(Base-BaseTRef), as described herein, for example, with respect to FIG. 11 .

In any of the above examples, ARP unit 283, motion compensation unit 282, and/or prediction processing unit 281, or any component of video decoder 30, may provide the inter-view predictor block to summer 290, which sums the inter-view predictor block with the decoded residual received from inverse transform processing unit 288 to reconstruct the current video block. Additionally, ARP unit 283 may scale the TMV and DMV, or perform any of the functions described herein for ARP according to the techniques of this disclosure.

In this manner, video decoder 30 may be configured to implement the example ARP techniques of this disclosure to decode video blocks. For example, video decoder 30 may be an example of a video decoder configured to perform a method for inter-view advanced residual prediction of video data, the method comprising decoding an encoded video bitstream that encodes video data to identify a disparity motion vector (DMV) and a residual block for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on an inter-view reference video block in a reference view and in the same access unit as the current video block. The method further comprises identifying a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block; identifying a temporal reference video block in an associated reference picture in the reference view based on the TMV for the inter-view reference video block; and identifying a temporal reference video block in the current view based on the TMV for the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The method further includes determining a residual predictor block for the current video block based on a difference between a temporal reference video block in the current view and a temporal reference video block in a reference view; and applying the residual predictor block and a residual block identified from the encoded video bitstream to the inter-view reference video block to reconstruct the current video block.

Video decoder 30 may also be an example of a video coder that includes a memory configured to store an encoded video bitstream of encoded video data and one or more processors. The one or more processors of a video coder, such as video decoder 30, may be configured to identify a DMV for a current video block, wherein the current video block is in a current view, and wherein the DMV is used for inter-view prediction of the current video block based on inter-view reference video blocks in a reference view and in the same access unit as the current video block. The one or more processors are further configured to identify a temporal motion vector (TMV) and an associated reference picture for the inter-view reference video block, identify a temporal reference video block in an associated reference picture in the reference view based on the TMV for the inter-view reference video block, and identify a temporal reference video block in the current view based on the TMV for the inter-view reference video block in the reference view. The temporal reference video block in the current view and the temporal reference video block in the reference view are located in the same access unit. The one or more processors are further configured to: determine a residual predictor block for the current video block based on a difference between a temporal reference video block in the current view and a temporal reference video block in a reference view; and decode the encoded video bitstream to identify a DMV and a residual block for the current video block, wherein the residual block identified by decoding the encoded video bitstream comprises a difference between the inter-view reference video block and the residual predictor block for the current video block.

16 is a flowchart illustrating an example ARP method for decoding a video block according to the techniques described in this disclosure. The example method of FIG. 16 may be performed by a video decoder, such as video decoder 30, which may include ARP unit 283.

16 , video decoder 30 decodes the encoded video bitstream to identify a reference video block and a residual block for a current video block ( 300 ). For example, motion compensation unit 282 may identify the reference video block based on a motion vector indicated by syntax decoded by entropy decoding unit 280, and inverse transform processing unit 288 may provide the decoded residual block to summer 290. Video decoder 30 (e.g., ARP unit 283) identifies a DMV from the current view to the reference view for the current video block ( 302 ).

Video decoder 30 (e.g., ARP unit 283) may then determine a residual predictor block for decoding the current video block based on the DMV (304). For example, if the current video block is inter-view predicted, video decoder 30 may determine the inter-view residual predictor block based on the DMV of the current video block using, for example, inter-view ARP techniques as described with respect to FIG. 10. If the current video block is temporally predicted, video decoder 30 may determine the temporal residual predictor block based on the DMV of a temporal reference video block using, for example, temporal ARP techniques as described with respect to FIG. 11. Video decoder 30 (e.g., ARP unit 283 and/or summer 290) may apply the residual predictor block and the decoded residual block to the reference video block to reconstruct the current video block (306).

17 is a flowchart illustrating an example inter-view ARP method for decoding an inter-view predicted video block according to the techniques described in this disclosure. The example method of FIG. 17 may be performed by a video decoder, such as video decoder 30, which may include ARP unit 283.

According to the example method of FIG17 , video decoder 30 decodes the encoded video bitstream to identify a DMV and a residual block for inter-view prediction of a current video block (310). Video decoder 30 (e.g., ARP unit 283) identifies an inter-view reference video block (Base) based on the DMV (312). Video decoder 30 (e.g., ARP unit 283) also identifies a TMV and an associated reference picture for the inter-view reference video block (Base) (314).

The video decoder 30 (e.g., ARP unit 283) may then identify temporal reference video blocks in the current and reference views (CurrTRef and BaseTRef, respectively) based on the TMV (316), for example, using the techniques described above with respect to FIG. 10. The video decoder 30 (e.g., ARP unit 283) may then determine an inter-view residual predictor block for the current video block based on the difference between these temporal reference video blocks (CurrTRef - BaseTRef) (318). The video decoder (e.g., ARP unit 283 and/or summer 290) may apply the inter-view residual predictor block and the decoded residual block to the inter-view reference video block (Base) to reconstruct the current video block (Curr) (320).

18 is a flowchart illustrating an example temporal ARP method for decoding temporally predicted video blocks according to the techniques described in this disclosure. The example method of FIG. 18 may be performed by a video decoder, such as video decoder 30, which may include ARP unit 283.

18 , video decoder 30 decodes the encoded video bitstream to identify a temporal reference video block (CurrTRef) in the current view and a residual block for reconstructing the current video block (330). Video decoder 30 (e.g., motion compensation unit 282) may identify the temporal reference video block (CurrTRef) in the current view using the TMV associated with the current video block as determined from the decoded video bitstream. Video decoder 30 (e.g., ARP unit 283) may identify the DMV of the temporal reference video block (CurrTRef), which in turn may identify the temporal reference video block in the reference view (BaseTRef) (332).

Video decoder 30 (e.g., ARP unit 283) may also identify inter-view reference video blocks in the reference view (Base) based on the DMV of the temporal reference video block (CurrTRef) in the current view (334). Video decoder 30 (e.g., ARP unit 283) may then determine a temporal residual predictor block for the current video block based on the difference (Base - BaseTRef) between these reference video blocks in the reference view (336). Video decoder (e.g., ARP unit 283 and/or summer 290) may apply the temporal residual predictor block and the decoded residual block to the temporal reference video block (CurrTRef) to reconstruct the current video block (Curr) (338).

19 is a flowchart illustrating an example ARP method for encoding a video block according to the techniques described in this disclosure. The example method of FIG. 19 may be performed by a video encoder, such as video encoder 20, which may include ARP unit 245.

According to the example method of FIG. 19 , video encoder 20 (e.g., ARP unit 245) identifies the DMVs of the current view to the reference view for the current video block (340). Video encoder 20 (e.g., ARP unit 245) may then determine a residual predictor block for encoding the current video block based on the DMVs (342). For example, if the current video block is inter-view predicted, video encoder 20 may determine the inter-view residual predictor block based on the DMVs of the current view using, for example, the inter-view ARP techniques described with respect to FIG. 10. If the current video block is temporally predicted, video encoder 20 may determine the temporal residual predictor block based on the DMVs of temporal reference video blocks in the current view using, for example, the temporal ARP techniques described with respect to FIG. 11. In either case, video encoder 20 (e.g., ARP unit 245 and summer 250) may determine a residual block for the current video block based on the difference between the current video block and the predictor block for the current video block, which may be the sum of the reference video block and the residual predictor block for the current video block (344). Video encoder 20 may encode the video bitstream to identify this residual block and the reference video block (346).

FIG20 is a flowchart illustrating an example inter-view ARP method for encoding an inter-view predicted video block according to the techniques described in this disclosure. The example method of FIG20 may be performed by a video encoder, such as video encoder 20, which may include ARP unit 245.

According to the example method of FIG20 , video encoder 20 (e.g., ARP unit 245) identifies a DMV for a current video block (Curr) to an inter-view reference video block (Base) (350). Video encoder 20 (e.g., ARP unit 245) also identifies a TMV and an associated reference picture (Base) for the inter-view reference video block (352). Video encoder 20 (e.g., ARP unit 245) may then identify temporal reference video blocks in the current and reference views (CurrTRef and BaseTRef, respectively) based on the TMV (354), for example, using the techniques described above with respect to FIG10 .

Video encoder 30 (e.g., ARP unit 245) may then determine an inter-view residual predictor block for the current video block based on the difference between these temporal reference video blocks (CurrTRef - BaseTRef) (318). Video encoder 20 (e.g., ARP unit 245 and summer 250) may determine a residual block for the current video block based on the difference between the current video block and the predictor block for the current video block, which may be the sum of the inter-view reference video block (Base) and the residual predictor block for the current video block (358). Video encoder 20 may encode the video bitstream to identify this residual block and the inter-view reference video block (360).

21 is a flowchart illustrating an example temporal ARP method for encoding temporally predicted video blocks according to the techniques described in this disclosure. The example method of FIG. 21 may be performed by a video encoder, such as video encoder 20, which may include ARP unit 245.

According to the example method of FIG21 , video encoder 20 (e.g., ARP unit 245) identifies a temporal reference video block (CurrTRef) in a current view, e.g., using a TMV associated with a current video block. Video encoder 20 (e.g., ARP unit 245) may then identify a DMV for the temporal reference video block (CurrTRef), which in turn may identify a temporal reference video block in a reference view (BaseTRef) (370). Based on the DMV for the temporal reference video block (CurrTRef) in the current view, video encoder 20 (e.g., ARP unit 245) may also identify an inter-view reference video block in a reference view (Base) (372).

Video encoder 20 (e.g., ARP unit 245) may then determine a temporal residual predictor block for the current video block based on the difference (Base-BaseTRef) between these reference video blocks in the reference view (374). Video encoder 20 (e.g., ARP unit 245 and summer 250) may determine a residual block for the current video block, which may be the sum of the temporal reference video block (CurrTRef) and the residual predictor block for the current video block, based on the difference between the current video block and the predictor block for the current video block (376). Video encoder 20 may encode the video bitstream to identify this residual block and the inter-view reference video block (378).

FIG22 is a flowchart illustrating an example method for identifying a DMV for a temporal ARP according to the techniques described in this disclosure. The example method of FIG22 may be performed by a video coder, such as video encoder 20 and/or video decoder 30, which may include an ARP unit 245, 283.

According to the example method of FIG. 22 , a video coder identifies a temporal reference video block in a current view (CurrTRef) based on a scaled TMV (380). The video coder then determines whether the identified temporal reference video block is associated with a DMV (382). If the temporal reference video block is associated with a DMV, the video coder identifies an inter-view reference video block based on the DMV (388). If the temporal reference video block is not associated with a DMV, the video coder identifies another temporal reference video block in the current view based on the TMV without scaling (384), and identifies an inter-view reference video block based on the DMV of the temporal reference video block in the current view identified based on the TMV without scaling (388).

FIG23 is a flowchart illustrating an example method for identifying a DMV or TMV for an ARP according to the techniques described in this disclosure. The example method of FIG23 may be performed by a video coder, such as video encoder 20 and/or video decoder 30, which may include an ARP unit 245, 283.

According to the example method of FIG. 23 , the video coder first checks RefPicList0 for the DMV or TMV required for ARP (390). If RefPicList0 includes the DMV or TMV, the video coder identifies the reference video block based on the DMV or TMV (396). If RefPicList0 does not include the DMV or TMV, the video coder checks RefPicList1 for the DMV or TMV (394) and may identify the reference video block based on the DMV or TMV from RefPicList1 (396). If the reference picture list includes neither the DMV nor the TMV, the video coder may use a 0 motion vector or not perform ARP, as an example. In some examples where the video coder uses a 0 motion vector for ARP, the video coder may apply a 0 motion vector to the reference picture list (direction) called for inter-view prediction using the DMV.

In some examples, one or more aspects of the techniques described in this disclosure may be performed by an intermediate network device, such as a media-aware network element (MANE), a stream adaptation processor, a splicing processor, or an editing processor. For example, such an intermediate device may be configured to generate or receive any of the various signaling methods described in this disclosure.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted on a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium, or a communication medium that includes any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-transitory, tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and/or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather are directed to non-transitory, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by distinct hardware units. Indeed, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and/or firmware, or provided by a collection of interoperating hardware units, including one or more processors as described above.

Various examples have been described. These and other examples are within the scope of the following claims.

Claims (30)

1. A method for predicting inter-view residuals in decoding HEVC video data, the method comprising: Decode the encoded video bitstream to identify the disparity motion vector (DMV) and residual block of the current video block, wherein the current video block is in the current view, and wherein the DMV is used for inter-view prediction of the current video block and the inter-view prediction of the current video block is based on an inter-view reference video block in a reference view and in the same access unit as the current video block. Identify the temporal motion vector (TMV) and associated reference images of the inter-view reference video blocks; The TMV identifies time-referenced video blocks in the associated reference images of the reference view based on the inter-view reference video blocks; The TMV identifies the time reference video block in the current view based on the inter-view reference video block in the reference view, wherein the time reference video block in the current view and the time reference video block in the reference view are located in the same access unit; The residual predictor block of the current video block is determined based on the difference between the time reference video block in the current view and the time reference video block in the reference view; and The residual predictor block and the residual block identified from the encoded video bitstream are applied to the reference video block to reconstruct the current video block. 2. The method of claim 1, wherein determining the residual predictor block of the current video block based on the difference between the time reference video block in the current view and the time reference video block in the reference view comprises applying a weighting factor to the difference between the time reference video block in the current view and the time reference video block in the reference view. 3. The method of claim 1, further comprising scaling the TMV of the inter-view reference video block to a target reference image in the target access unit for residual prediction of the current video block, wherein at least one of the following: The TMV identification of the time reference video block in the reference view based on the inter-view reference video block includes identifying the time reference video block in the reference view based on the scaled TMV; and The TMV identification of the time reference video block in the current view based on the inter-view reference video block includes the identification of the time reference video block in the reference view based on the scaled TMV. 4. The method according to claim 3, further comprising: Select a list of reference images; and Select the target reference image from the list of selected reference images. The list of reference images includes one of the following: Select the list of reference images associated with the TMV; or Select the list of reference images associated with the DMV. 5. The method of claim 1, wherein the inter-view reference video block comprises a plurality of prediction units, and the TMV for identifying the inter-view reference video block comprises identifying a TMV associated with one of the plurality of prediction units containing the center position of the inter-view reference video block. 6. The method of claim 5, further comprising determining that the prediction unit containing the center position of the inter-view reference video block does not have a TMV, wherein identifying the TMV of the inter-view reference video block includes identifying a 0 motion vector as the TMV of the inter-view reference video block. 7. The method of claim 6, wherein identifying the associated reference image of the inter-view reference video block includes identifying a reference image in the reference view that is in the same access unit as the target reference image used in residual prediction. 8. The method according to claim 1, wherein the inter-view reference video block contains a first motion information set corresponding to a first reference image list and a second motion information set corresponding to a second reference image list, and identifying the TMV of the inter-view reference video block includes: If the first motion information set contains a TMV, then the TMV is selected from the first motion information set; and If the first set of motion information does not contain a TMV, the TMV is selected from the second set of motion information. 9. The method of claim 8, wherein the first reference image list includes RefPicList0. 10. The method of claim 8, wherein the order in which the first and second motion information sets are considered is independent of which of the first and second motion information sets contains the DMV. 11. The method of claim 1, further comprising: Apply view order difference scaling to scale the identified DMV to a target reference view for residual prediction of the current video block; and The reference video blocks between views are identified based on the scaled DMV. 12. A method for predicting inter-view residuals in HEVC video data, the method comprising: Identify the disparity motion vector (DMV) of the current video block, wherein the current video block is in the current view, and wherein the DMV is used for inter-view prediction of the current video block and the inter-view prediction of the current video block is based on a reference view and an inter-view reference video block in the same access unit as the current video block. Identify the temporal motion vector (TMV) and associated reference images of the inter-view reference video blocks; The TMV identifies time-referenced video blocks in the associated reference images of the reference view based on the inter-view reference video blocks; The TMV identifies the time reference video block in the current view based on the inter-view reference video block in the reference view, wherein the time reference video block in the current view and the time reference video block in the reference view are located in the same access unit; The residual predictor block of the current video block is determined based on the difference between the time reference video block in the current view and the time reference video block in the reference view; and The encoded video bitstream is encoded to identify the DMV and residual blocks of the current video block, wherein the residual blocks identified by the encoded video bitstream include the difference between the inter-view reference video block and the residual predictor block of the current video block. 13. The method of claim 12, wherein determining the residual predictor block of the current video block based on the difference between the time reference video block in the current view and the time reference video block in the reference view comprises applying a weighting factor to the difference between the time reference video block in the current view and the time reference video block in the reference view. 14. The method of claim 12, further comprising scaling the TMV of the inter-view reference video block to a target reference image in the target access unit for residual prediction of the current video block, wherein at least one of the following: The TMV identification of the time reference video block in the reference view based on the inter-view reference video block includes identifying the time reference video block in the reference view based on the scaled TMV; and The TMV identification of the time reference video block in the current view based on the inter-view reference video block includes the identification of the time reference video block in the reference view based on the scaled TMV. 15. The method of claim 14, further comprising: Select a list of reference images; and Select the target reference image from the list of selected reference images. The list of reference images includes one of the following: Select the list of reference images associated with the TMV; or Select the list of reference images associated with the DMV. 16. The method of claim 12, wherein the inter-view reference video block comprises a plurality of prediction units, and the TMV for identifying the inter-view reference video block comprises identifying a TMV associated with one of the plurality of prediction units containing the center position of the inter-view reference video block. 17. The method of claim 12, wherein the inter-view reference video block contains a first set of motion information corresponding to a first list of reference images and a second set of motion information corresponding to a second list of reference images, and identifying the TMV of the inter-view reference video block includes: If the first motion information set contains a TMV, then the TMV is selected from the first motion information set; and If the first set of motion information does not contain a TMV, the TMV is selected from the second set of motion information. 18. The method of claim 17, wherein the first reference image list includes RefPicList0. 19. The method of claim 17, wherein the order in which the first and second motion information sets are considered is independent of which of the first and second motion information sets contains the DMV. 20. The method of claim 12, further comprising: Apply view order difference scaling to scale the identified DMV to a target reference view for residual prediction of the current video block; and The reference video blocks between views are identified based on the scaled DMV. 21. An apparatus comprising a video decoder configured to perform inter-view residual prediction for decoding HEVC video data, the video decoder comprising: A memory configured to store an encoded video bitstream that encodes the video data; and One or more processors, configured to: Identify the disparity motion vector (DMV) of the current video block, wherein the current video block is in the current view, and wherein the DMV is used for inter-view prediction of the current video block and the inter-view prediction of the current video block is based on an inter-view reference video block in a reference view and in the same access unit as the current video block. Identify the temporal motion vector (TMV) and associated reference images of the inter-view reference video blocks; The TMV identifies time-referenced video blocks in the associated reference images of the reference view based on the inter-view reference video blocks; The TMV identifies the time reference video block in the current view based on the inter-view reference video block in the reference view, wherein the time reference video block in the current view and the time reference video block in the reference view are located in the same access unit; The residual predictor block of the current video block is determined based on the difference between the time reference video block in the current view and the time reference video block in the reference view; and The encoded video bitstream is decoded to identify the DMV and residual blocks of the current video block, wherein the residual blocks identified by decoding the encoded video bitstream include the difference between the inter-view reference video block and the residual predictor block of the current video block. 22. The apparatus of claim 21, wherein the one or more processors are further configured to scale the TMV of the inter-view reference video block to a target reference image in the target access unit for residual prediction of the current video block, and at least one of the following: The TMV identification of the time reference video block in the reference view based on the inter-view reference video block includes identifying the time reference video block in the reference view based on the scaled TMV; and The TMV identification of the time reference video block in the current view based on the inter-view reference video block includes the identification of the time reference video block in the reference view based on the scaled TMV. 23. The device of claim 22, wherein the one or more processors are further configured to: Select a list of reference images; and Select the target reference image from the list of selected reference images. Wherein, for selecting the list of reference images, the one or more processors: Select the list of reference images associated with the TMV; or Select the list of reference images associated with the DMV. 24. The apparatus of claim 21, wherein the inter-view reference video block comprises a plurality of prediction units, and in order to identify the TMV of the inter-view reference video block, the one or more processors identify a TMV associated with one of the plurality of prediction units containing the center position of the inter-view reference video block. 25. The device of claim 21, wherein the inter-view reference video block contains a first set of motion information corresponding to a first list of reference images and a second set of motion information corresponding to a second list of reference images, and wherein, for identifying the TMV of the inter-view reference video block, the one or more processors: If the first motion information set contains a TMV, then the TMV is selected from the first motion information set; and If the first set of motion information does not contain a TMV, the TMV is selected from the second set of motion information. 26. The device of claim 25, wherein the first reference image list includes RefPicList0. 27. The device of claim 21, wherein the one or more processors are further configured to: Apply view order difference scaling to scale the identified DMV to a target reference view for residual prediction of the current video block; and The reference video blocks between views are identified based on the scaled DMV. 28. The apparatus of claim 21, wherein the video decoder includes a video decoder, and the one or more processors are configured to: Decode the encoded video bitstream to identify the DMV of the current video block and the residual block; and The residual predictor block and the residual block identified from the encoded video bitstream are applied to the inter-view reference video block to reconstruct the current video block. 29. The apparatus of claim 21, wherein the video decoder includes a video encoder, and the one or more processors are configured to encode the encoded video bitstream to indicate the DMV of the current video block and the residual block to the video decoder. 30. A computer-readable storage medium having instructions stored thereon, said instructions, when executed, cause one or more processors of a video decoder to perform inter-view residual prediction of HEVC video data by means of the following steps: Identify the disparity motion vector (DMV) of the current video block, wherein the current video block is in the current view, and wherein the DMV is used for inter-view prediction of the current video block and the inter-view prediction of the current video block is based on inter-view reference images in a reference view and in the same access unit as the current video block. Identify the temporal motion vector (TMV) and associated reference images of the inter-view reference video blocks; The TMV identifies time-referenced video blocks in the associated reference images of the reference view based on the inter-view reference video blocks; The TMV identifies the time reference video block in the current view based on the inter-view reference video block in the reference view, wherein the time reference video block in the current view and the time reference video block in the reference view are located in the same access unit; The residual predictor block of the current video block is determined based on the difference between the time reference video block in the current view and the time reference video block in the reference view; and The encoded video bitstream is decoded to identify the DMV and residual blocks of the current video block, wherein the residual blocks identified by decoding the encoded video bitstream include the difference between the inter-view reference video block and the residual predictor block of the current video block.