HK1223757B - Block-based advanced residual prediction for 3d video coding - Google Patents

Description

Block-based advanced residual prediction for 3D video coding

This application claims the benefit of U.S. Provisional Application No. 61/926,290, filed January 11, 2014, the entire contents of which are incorporated herein by reference in their entirety.

Technical Field

The present invention relates to video decoding.

Background Art

Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio telephones (so-called "smartphones"), video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10 Advanced Video Coding (AVC), the High Efficiency Video Coding (HEVC) standard currently under development, and extensions of such standards. By implementing such video coding techniques, video devices can more efficiently transmit, receive, encode, decode, and/or store digital video information.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (i.e., a video frame or portion of a video frame) may be partitioned into video blocks, which may also be referred to as treeblocks, coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames.

Spatial or temporal prediction results in a prediction block for the block to be coded. Residual data represents the pixel differences between the original block to be coded and the prediction block. Inter-coded blocks are encoded based on a motion vector pointing to a block of reference samples forming the prediction block and residual data indicating the difference between the coded block and the prediction block. Intra-coded blocks are encoded based on an intra-coding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, thereby producing residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients, and entropy coding can be applied to achieve even more compression.

Summary of the Invention

In general, this disclosure relates to multi-view video coding, in which the coded video data includes two or more views. Specifically, this disclosure describes various techniques related to advanced residual prediction (ARP). The techniques of this disclosure can reduce the number of times a video coder (e.g., a video encoder and/or video decoder) accesses motion information in order to perform ARP or any basic motion compensation process (i.e., using assigned motion vectors with potential interpolation operations to generate prediction blocks). In this way, the speed of video coding (i.e., encoding or decoding) can be increased because fewer memory accesses to motion information are performed.

In one example of the present invention, a method for decoding video data includes: receiving a first encoded video data block in a first access unit of a first view, wherein the first encoded video data block is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction; determining temporal motion information for the first prediction direction of the first encoded video data block; determining disparity motion information for the second prediction direction of the first encoded video data block; using the determined temporal motion information for the first prediction direction to identify a reference block for the second prediction direction different from the first prediction direction, wherein the reference block is in an access unit different from the first access unit; and performing advanced residual prediction on the first encoded video data block using the identified reference block for the second prediction direction.

In another example of the present invention, an apparatus configured to decode video data includes: a video data memory configured to store a first encoded video data block in a first access unit of a first view, wherein the first encoded video data block is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction; and one or more processors in communication with the video data memory and configured to: determine temporal motion information for the first prediction direction of the first encoded video data block; determine disparity motion information for the second prediction direction of the first encoded video data block; use the determined temporal motion information for the first prediction direction to identify a reference block for the second prediction direction different from the first prediction direction, wherein the reference block is in an access unit different from the first access unit; and perform advanced residual prediction on the first encoded video data block using the identified reference block for the second prediction direction.

In another example of the present invention, an apparatus configured to decode video data includes: a device for receiving a first encoded video data block in a first access unit of a first view, wherein the first encoded video data block is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction; a device for determining temporal motion information for the first prediction direction of the first encoded video data block; a device for determining disparity motion information for the second prediction direction of the first encoded video data block; a device for using the determined temporal motion information for the first prediction direction to identify a reference block for the second prediction direction different from the first prediction direction, wherein the reference block is in an access unit different from the first access unit; and a device for performing advanced residual prediction on the first encoded video data block using the identified reference block for the second prediction direction.

In another example, the present disclosure describes a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device configured to decode video data to: receive a first block of encoded video data in a first access unit of a first view, wherein the first block of encoded video data is encoded using advanced residual prediction and bi-directional prediction, the bi-directional prediction including temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction; determine temporal motion information for the first prediction direction of the first block of encoded video data; determine disparity motion information for the second prediction direction of the first block of encoded video data; use the determined temporal motion information for the first prediction direction to identify a reference block for a second prediction direction different from the first prediction direction, wherein the reference block is in an access unit different from the first access unit; and perform advanced residual prediction on the first block of encoded video data using the identified reference block for the second prediction direction.

The details of one or more embodiments of the present invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a block diagram illustrating an example video encoding and decoding system that may utilize the techniques described in this disclosure.

2 is a graphical diagram illustrating an example multi-view encoding or decoding order.

3 is a conceptual diagram illustrating example temporal and inter-view prediction patterns for multi-view video coding.

FIG. 4 is a conceptual diagram illustrating texture and depth values for 3D video.

5 is a conceptual diagram illustrating an example relationship between neighboring blocks and a current block used to predict motion information of the current block.

6 is a conceptual diagram illustrating an example of derivation of inter-view predicted motion vector candidates and inter-view disparity motion vector candidates for predicting motion information of a current block.

7 is a conceptual diagram illustrating example spatial neighboring blocks relative to a current video block from which a disparity vector for the current video block may be derived using Neighbor-Based Disparity Vector (NBDV) derivation.

FIG. 8 is a conceptual diagram illustrating inter-view motion prediction of a sub-prediction unit (PU).

9 is a conceptual diagram illustrating an example prediction structure for temporal advanced residual prediction (ARP) for temporally predicted video blocks.

10 is a conceptual diagram illustrating an example bi-directional prediction structure for temporal ARP.

11 is a conceptual diagram of an example prediction structure for inter-view ARP of an inter-view predicted video block, according to the techniques described in this disclosure.

12 is a conceptual diagram illustrating an example prediction structure for bidirectional ARP using inter-view prediction for one reference picture list and temporal prediction for another reference picture list.

13 is a conceptual diagram illustrating an example prediction structure for bidirectional ARP using inter-view prediction for one reference picture list and temporal prediction for another reference picture list, according to the techniques of this disclosure.

FIG. 14 is a conceptual diagram illustrating block-based time ARP.

FIG15 is a conceptual diagram illustrating block-based inter-view ARP.

FIG. 16 is a conceptual diagram illustrating block-based ARP using sub-PU merge candidates.

17 is a block diagram illustrating an example video encoder that may implement the techniques described in this disclosure.

18 is a block diagram illustrating an example video decoder that may utilize the techniques described in this disclosure.

19 is a flowchart illustrating an example ARP method for encoding video blocks according to the techniques described in this disclosure.

20 is a flowchart illustrating an example ARP method for decoding video blocks according to the techniques described in this disclosure.

DETAILED DESCRIPTION

In general, the present invention relates to multi-view video coding, wherein the coded video data includes two or more views. In some examples, multi-view video coding includes a multi-view plus depth video coding process. In some examples, multi-view coding may include coding of three-dimensional (3D) video and may be referred to as 3D video coding. In various examples of the present invention, techniques for advanced residual prediction (APR) in non-base views of multi-view and/or 3D video coding sequences are described. The techniques of the present invention can reduce the number of times a video coder (e.g., a video encoder and/or a video decoder) accesses motion information from memory in order to perform APR or any basic inter-frame prediction (e.g., temporal and/or inter-view inter-frame prediction and bidirectional prediction). In this way, the speed of video coding (i.e., encoding or decoding) can be increased because fewer memory accesses to motion information are performed.

For example, the present disclosure describes a method for decoding video data, comprising: receiving a first block of encoded video data in a first access unit, wherein the first block of encoded video data is encoded using advanced residual prediction and bi-directional inter-view prediction; determining temporal motion information for a first prediction direction of the first block of encoded video data; and using the temporal motion information determined for the first prediction direction to identify a reference block for a second prediction direction different from the first prediction direction, wherein the reference block is in a second access unit.

FIG1 is a block diagram illustrating an example video encoding and decoding system 10 that may utilize the techniques of this disclosure. As shown in FIG1 , system 10 includes a source device 12 that provides encoded video data to be decoded at a later time by a destination device 14. Specifically, source device 12 may provide the video data to destination device 14 via a computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

Destination device 14 may receive the encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 may comprise any type of medium or device capable of moving the encoded video data from source device 12 to destination device 14. In one example, computer-readable medium 16 may comprise a communication medium that enables source device 12 to transmit the encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 12 to destination device 14.

In some examples, the encoded data can be output from output interface 22 to a storage device. Similarly, the encoded data can be accessed from the storage device via the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by source device 12. Destination device 14 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing encoded video data and transmitting the encoded video data to destination device 14. Example file servers include a network server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. Destination device 14 can access the encoded video data via any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of the two suitable for accessing encoded video data stored on a file server. The transmission of the encoded video data from the storage device may be streaming, downloading, or a combination of both.

The techniques of this disclosure are not necessarily limited to wireless applications or environments. The techniques may be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air protocol television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (e.g., Dynamic Adaptive Streaming over HTTP (DASH)), digital video encoded onto data storage media, decoding of digital video stored on data storage media, or other applications. In some examples, system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

1 , source device 12 includes a video source 18, a depth estimation unit 19, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, a depth image-based rendering (DIBR) unit 31, and a display device 32. In other examples, the source and destination devices may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18, such as an external camera. Similarly, destination device 14 may interface with an external display device rather than including an integrated display device.

The illustrated system 10 of FIG. 1 is merely an example. The techniques of this disclosure may be performed by any digital video encoding and/or decoding device. Although the techniques of this disclosure are generally performed by a video encoding device, the techniques may also be performed by a video encoder/decoder (often referred to as a "codec"). Furthermore, the techniques of this disclosure may also be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of such coding devices in which source device 12 generates coded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate in a substantially symmetrical manner, such that each of devices 12, 14 includes video encoding and decoding components. Thus, system 10 may support one-way or two-way video transmission between video devices 12, 14, for example, for video streaming, video playback, video broadcasting, or video telephony.

Video source 18 of source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, and/or a video feed interface for receiving video from a video content provider. As another alternative, video source 18 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, as mentioned above, the techniques described in this disclosure may be applicable to video coding in general and may be applied to wireless and/or wired applications. In each case, the captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output to computer-readable medium 16 via output interface 22.

Video source 18 may provide one or more views of video data to video encoder 20. For example, video source 18 may correspond to an array of cameras, each of which has a unique horizontal position relative to the particular scene being captured. Alternatively, video source 18 may generate video data from different horizontal camera perspectives, for example using computer graphics. Depth estimation unit 19 may be configured to determine the values of depth pixels corresponding to pixels in the texture image. For example, depth estimation unit 19 may represent a sound navigation and ranging (SONAR) unit, a light detection and ranging (LIDAR) unit, or other unit capable of directly determining depth values substantially simultaneously as video data of a scene is recorded.

Additionally or alternatively, depth estimation unit 19 may be configured to indirectly calculate depth values by comparing two or more images captured at substantially the same time from different horizontal camera perspectives. By calculating horizontal disparity between substantially similar pixel values in the images, depth estimation unit 19 can approximate the depths of various objects in the scene. In some examples, depth estimation unit 19 may be functionally integrated with video source 18. For example, when video source 18 generates computer graphics images, depth estimation unit 19 may provide an actual depth map for the graphical objects, such as using the pixels and z-coordinates of the objects used to reproduce the texture image.

Computer-readable medium 16 may include transient media, such as wireless broadcasts or wired network transmissions, or storage media (that is, non-transitory storage media) such as hard disks, flash drives, compact discs, digital video discs, Blu-ray discs, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from source device 12, for example, via a network transmission, and provide the encoded video data to destination device 14. Similarly, a computing device at a media production facility (such as a disc stamping facility) may receive encoded video data from source device 12 and produce an optical disc containing the encoded video data. Thus, in various examples, computer-readable medium 16 may be understood to include one or more computer-readable media in various forms.

Input interface 28 of destination device 14 receives information from computer-readable medium 16. The information of computer-readable medium 16 may include syntax information defined by video encoder 20, which is also used by video decoder 30, including syntax elements that describe characteristics and/or processing of blocks and other coded units (e.g., GOPs). Display device 32 displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. In some examples, display device 32 may include a device capable of displaying two or more views simultaneously or substantially simultaneously, for example, to produce a 3D visual effect for a viewer.

DIBR unit 31 of destination device 14 can reproduce a synthesized view using texture and depth information for the decoded view received from video decoder 30. For example, DIBR unit 31 can determine the horizontal disparity of pixel data for the texture image based on the values of pixels in the corresponding depth map. DIBR unit 31 can then generate a synthesized image by shifting pixels in the texture image left or right by the determined horizontal disparity. In this manner, display device 32 can display one or more views, which may correspond to decoded views and/or synthesized views, in any combination. According to the techniques of this disclosure, video decoder 30 can provide original and updated precision values for depth ranges and camera parameters to DIBR unit 31, which can use these depth ranges and camera parameters to properly synthesize the view.

Although not shown in FIG1 , in some aspects, video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams. If applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP).

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in the respective device. A device including video encoder 20 and/or video decoder 30 may include an integrated circuit, a microprocessor, and/or a wireless communication device, such as a cellular telephone.

In one example of the present invention, video decoder 30 may be configured to receive a first block of encoded video data in a first access unit of a first view, wherein the first block of encoded video data is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction. Video decoder 30 may be further configured to determine temporal motion information for a first prediction direction for the first block of encoded video data and to determine parallax motion information for a second prediction direction for the first block of encoded video data. Video decoder 30 may be further configured to use the determined temporal motion information for the first prediction direction to identify a reference block for a second prediction direction different from the first prediction direction, wherein the reference block is in an access unit different from the first access unit, and to perform advanced residual prediction on the first block of encoded video data using the identified reference block for the second prediction direction. In this manner, the temporal motion information for the first prediction direction is reused for the second prediction direction. Consequently, fewer memory accesses are required for temporal motion information because the temporal motion information for the block identified by the motion vector of the first encoded block corresponding to the second prediction direction does not need to be accessed, thereby allowing for faster video decoding. Additionally, the total number of reference blocks used when performing ARP can be reduced from 6 to 5, which results in less computational complexity in interpolation using multiplication and addition operations. Likewise, when performing bidirectional inter prediction, video encoder 20 can be configured to reuse temporal motion information for a first prediction direction when encoding a second prediction direction.

Video encoder 20 and video decoder 30 may operate according to a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC Test Model (HM). Alternatively, video encoder 20 and video decoder 30 may operate according to other proprietary or industry standards, such as the ITU-T H.264 standard, alternatively referred to as MPEG-4 Part 10 Advanced Video Coding (AVC), or extensions of such standards, such as the MVC extension of ITU-T H.264/AVC. The latest joint draft of MVC is described in "Advanced Video Coding for Generic Audiovisual Services" (ITU-T Recommendation H.264), dated March 2010. Specifically, video encoder 20 and video decoder 30 may operate according to 3D and/or multi-view coding standards, including a 3D extension of the HEVC standard (e.g., 3D-HEVC).

A draft of the HEVC standard, referred to as “HEVC Working Draft 10” or “WD10,” is described in document JCTVC-L1003v34, “High Efficiency Video Coding (HEVC) Textual Specification Draft 10 (for FDIS and Last Call)” by Bloss et al., Joint Collaboration Team on Video Coding (JCT-VC) of ITU-TSG16 WP3 and ISO/IEC JTC1/SC29/WG11, 12th Meeting, Geneva, Switzerland, January 14-23, 2013, available for download from http://phenix.int-evry.fr/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1003-v34.zip as of January 5, 2015.

Another draft of the HEVC standard, known as the "WD10 revision," is described in Bloss et al., "Editors' proposed corrections to HEVC version 1," Joint Collaboration Team on Video Coding (JCT-VC) of ITU-TSG16 WP3 and ISO/IEC JTC1/SC29/WG11, 13th Meeting, Incheon, South Korea, April 2013, available as of January 5, 2015 at http://phenix.int-evry.fr/jct/doc_end_user/documents/13_Incheon/wg11/JCTVC-M0432-v3.zip. A multi-view extension to HEVC, namely MV-HEVC, is also being developed by JCT-3V.

Currently, the Joint Collaboration Team on 3D Video Coding (JCT-3C) of VCEG and MPEG is developing the HEVC-based 3DV standard. Part of this standardization work includes the standardization of the HEVC-based Multi-View Video Codec (MV-HEVC) and another part for HEVC-based 3D Video Coding (3D-HEVC). For MV-HEVC, only the high-level syntax (HLS) changes are required, ensuring that modules at the coding unit/prediction unit level in HEVC do not need to be redesigned and are fully reusable in MV-HEVC. For 3D-HEVC, new coding tools for both texture and depth views, including tools at the coding unit/prediction unit level, can be included and supported.

A version of the 3D-HTM reference software for 3D-HEVC is available for download at the following link: [3D-HTM Version 9.01r1]: https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware/tags/HTM-9.0r1/. A version of the reference software is described in Li Zhang, Gerhard Tech, Krzysztof Wegner, and Sehoon Yea, "Test Model 6 for 3D-HEVC and MV-HEVC," JCT3V-F1005, Joint Collaboration Group on 3D Video Coding Extension Development of ITU-TSG 16 WP3 and ISO/IEC JTC 1/SC 29/WG 11, 6th Meeting, Geneva, Switzerland, November 2013 (JCT3V-F1005). JCT3V-F1005 can be downloaded from http://phenix.it-sudparis.eu/jct2/doc_end_user/current_document.php?id=1636.

A working draft of 3D-HEVC is described in Gerhard Tech, Krzysztof Wegner, Ying Chen, and Sehoon Yea, “3D-HEVC Draft Text 2,” JCT3V-F1001, Joint Collaboration Group on 3D Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 6th Meeting, Geneva, Switzerland, November 2013 (JCT3V-F1-001). JCT3V-F1001 is available at http://phenix.it-sudparis.eu/jct2/doc_end_user/current_document.php?id=1361. The latest software description (document number: E1005) is available at http://phenix.it-sudparis.eu/jct2/doc_end_user/current_document.php?id=1360.

A more recent version of the 3D-HTM software for 3D-HEVC can be downloaded from the following link: [3D-HTM Version 12.0]: https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware/tags/HTM-12.0/. The corresponding working draft of 3D-HEVC (Document No.: I1001) is available at http://phenix.int-evry.fr/jct3v/doc_end_user/current_document.php?id=2299. The latest software description (Document No.: I1005) is available at http://phenix.int-evry.fr/jct3v/doc_end_user/current_document.php?id=2301.

Initially, we will discuss example coding techniques for HEVC. The HEVC standardization effort is based on an evolved model of a video coding device known as the HEVC Test Model (HM). The HM assumes several additional capabilities compared to existing devices, such as those based on ITU-T H.264/AVC. For example, while H.264 provides nine intra-frame prediction coding modes, the HM can provide up to 33 angular intra-frame prediction coding modes, plus DC and planar modes.

In HEVC and other video coding specifications, a video sequence typically contains a series of pictures. A picture may also be referred to as a "frame." A picture may contain three sample arrays, denoted _SL , _SCb , and _SCr . _SL is a two-dimensional array (i.e., a block) of luma samples. _SCb is a two-dimensional array of Cb chroma samples. _SCr is a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to herein as "chroma" samples. In other cases, a picture may be monochrome and may contain only a luma sample array.

To generate an encoded representation of a picture, video encoder 20 may generate a set of coding tree units (CTUs). Each of the CTUs may include a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples, and a syntax structure for coding the samples of the coding tree block. In a monochrome picture or a picture with three separate color planes, a CTU may include a single coding tree block and a syntax structure for coding the samples of the coding tree block. A coding tree block may be an N×N block of samples. A CTU may also be referred to as a “tree block” or a “largest coding unit (LCU)”. A CTU of HEVC may be broadly similar to macroblocks of other standards, such as H.264/AVC. However, a CTU is not necessarily limited to a specific size and may include one or more coding units (CUs). A slice may include an integer number of CTUs ordered consecutively in raster scan order.

To generate a coded CTU, video encoder 20 may recursively perform quadtree partitioning on the coding tree block of the CTU to divide the coding tree block into coding blocks, hence the name "coding tree unit". A coding block is an N×N block of samples. A coding unit (CU) may include a coding block of luma samples and two corresponding coding blocks of chroma samples for a picture having a luma sample array, a Cb sample array, and a Cr sample array, and syntax structures used to code the samples of the coding blocks. In a monochrome picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures used to code the samples of the coding block.

Video encoder 20 may partition the coding blocks of a CU into one or more prediction blocks. A prediction block is a rectangular (i.e., square or non-square) block of samples to which the same prediction is applied. A prediction unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax structures used to predict the prediction blocks. In monochrome pictures or pictures with three separate color planes, a PU may include a single prediction block and syntax structures used to predict the prediction blocks. Video encoder 20 may generate predictive luma, Cb, and Cr blocks for the luma, Cb, and Cr prediction blocks of each PU of a CU.

Video encoder 20 may use intra prediction or inter prediction to generate the prediction block for the PU. If video encoder 20 uses intra prediction to generate the prediction block for the PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of the picture associated with the PU. In some versions of HEVC, for the luma component of each PU, intra prediction methods are utilized with 33 angular prediction modes (indexed from 2 to 34), DC mode (indexed with 1), and planar mode (indexed with 0).

If video encoder 20 uses inter-prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of one or more pictures other than the picture associated with the PU. Inter-prediction may be unidirectional inter-prediction (i.e., unidirectional prediction or unidirectional predictive prediction) or bidirectional inter-prediction (i.e., bidirectional prediction or bidirectional predictive prediction). To perform unidirectional prediction or bidirectional prediction, video encoder 20 may generate a first reference picture list (RefPicList0) and a second reference picture list (RefPicList1) for the current slice. Each of the reference picture lists may include one or more reference pictures. When using unidirectional prediction, video encoder 20 may search the reference pictures in either or both of RefPicList0 and RefPicList1 to determine a reference position within the reference pictures. Furthermore, when using unidirectional prediction, video encoder 20 may generate a block of prediction samples for the PU based at least in part on samples corresponding to the reference position. Furthermore, when using unidirectional prediction, video encoder 20 may generate a single motion vector indicating the spatial displacement between the prediction block of the PU and the reference position. To indicate the spatial displacement between the PU's prediction block and the reference position, the motion vector may contain a horizontal component specifying the horizontal displacement between the PU's prediction block and the reference position and may contain a vertical component specifying the vertical displacement between the PU's prediction block and the reference position.

When encoding a PU using bi-prediction, video encoder 20 may determine a first reference location in a reference picture in RefPicList0 and a second reference location in a reference picture in RefPicList1. Video encoder 20 may then generate a prediction block for the PU based at least in part on samples corresponding to the first and second reference locations. Furthermore, when encoding a PU using bi-prediction, video encoder 20 may generate a first motion vector indicating a spatial displacement between the sample block of the PU and the first reference location, and a second motion vector indicating a spatial displacement between the prediction block of the PU and the second reference location.

Typically, reference picture list construction for the first or second reference picture list (e.g., RefPicList0 or RefPicList1) of a B picture involves two steps: reference picture list initialization and reference picture list reordering (modification). Reference picture list initialization is an explicit mechanism that places reference pictures in the reference picture memory (also known as the decoded picture buffer) into a list based on their POC (Picture Order Count) order values, which align with the display order of the pictures. The reference picture list reordering mechanism can modify the position of pictures placed in the list during reference picture list initialization to any new position, or place any reference picture in the reference picture memory at any position, even if the picture does not belong to the initialized list. Some pictures may be placed further into the list after the reference picture list reordering (modification). However, if a picture's position exceeds the number of valid reference pictures for a list, it is not considered an entry in the final reference picture list. The number of valid reference pictures may be signaled in the slice header of each list.

After the reference picture lists (ie, RefPicList0 and RefPicList1, if available) are constructed, reference indexes to the reference picture lists may be used to identify any reference pictures included in the reference picture lists.

After video encoder 20 generates predictive luma, Cb, and Cr blocks for one or more PUs of a CU, video encoder 20 may generate a luma residual block for the CU. Each sample in the luma residual block of the CU indicates the difference between a luma sample in one of the CU's predictive luma blocks and a corresponding sample in the CU's original luma coding block. Additionally, video encoder 20 may generate a Cb residual block for the CU. Each sample in the Cb residual block of the CU may indicate the difference between a Cb sample in one of the CU's predictive Cb blocks and a corresponding sample in the CU's original Cb coding block. Video encoder 20 may also generate a Cr residual block for the CU. Each sample in the Cr residual block of the CU may indicate the difference between a Cr sample in one of the CU's predictive Cr blocks and a corresponding sample in the CU's original Cr coding block.

In addition, video encoder 20 may use quadtree partitioning to decompose the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (e.g., square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax structures used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. The luma transform block associated with a TU may be a subblock of the CU's luma residual block. The Cb transform block may be a subblock of the CU's Cb residual block. The Cr transform block may be a subblock of the CU's Cr residual block. In monochrome pictures or pictures with three separate color planes, a TU may include a single transform block and syntax structures used to transform the samples of the transform block.

Video encoder 20 may apply one or more transforms to the luma transform block of a TU to generate a luma coefficient block for the TU. A coefficient block may be a two-dimensional array of transform coefficients. A transform coefficient may be a scalar. Video encoder 20 may apply one or more transforms to the Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to the Cr transform block of a TU to generate a Cr coefficient block for the TU.

After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform context-adaptive binary arithmetic coding (CABAC) on the syntax elements indicating the quantized transform coefficients.

Video encoder 20 may output a bitstream that includes a sequence of bits that form a representation of a coded picture and associated data. The bitstream may include a series of network abstraction layer (NAL) units. A NAL unit is a syntax structure that contains an indication of the type of data in the NAL unit and bytes containing the data in the form of a raw byte sequence payload (RBSP) interspersed with emulation prevention bits as needed. Each NAL unit includes a NAL unit header and encapsulates the RBSP. The NAL unit header may include a syntax element that indicates a NAL unit type code. The NAL unit type code, specified by the NAL unit header of the NAL unit, indicates the type of the NAL unit. The RBSP may be a syntax structure that contains an integer number of bytes encapsulated within the NAL unit. In some cases, the RBSP includes zero bits.

Different types of NAL units may encapsulate different types of RBSPs. For example, a first type of NAL unit may encapsulate RBSPs for picture parameter sets (PPSs), a second type of NAL unit may encapsulate RBSPs for coded slices, a third type of NAL unit may encapsulate RBSPs for SEIs, and so on. NAL units that encapsulate RBSPs for video coding data (as opposed to RBSPs for parameter sets and SEI messages) may be referred to as video coding layer (VCL) NAL units.

Video decoder 30 may receive a bitstream generated by video encoder 20. Additionally, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct a picture of the video data based, at least in part, on the syntax elements obtained from the bitstream. The process used to reconstruct the video data may be generally inverse to the process performed by video encoder 20. For example, video decoder 30 may use the motion vectors of the PUs to determine prediction blocks for the PUs of the current CU. Additionally, video decoder 30 may inverse quantize coefficient blocks associated with the TUs of the current CU. Video decoder 30 may perform an inverse transform on the coefficient blocks to reconstruct transform blocks associated with the TUs of the current CU. Video decoder 30 may reconstruct the coding blocks of the current CU by adding samples of the prediction blocks for the PUs of the current CU to corresponding samples of the transform blocks of the TUs of the current CU. By reconstructing the coding blocks for each CU of the picture, video decoder 30 may reconstruct the picture.

In some examples, video encoder 20 may signal the motion information of a PU using merge mode or advanced motion vector prediction (AMVP) mode. In other words, in HEVC, there are two modes for predicting motion parameters, one is merge mode and the other is AMVP. Motion prediction may include determining the motion information of a video unit (e.g., a PU) based on the motion information of one or more other video units. The motion information of a PU may include a motion vector of the PU and a reference index of the PU.

When video encoder 20 uses merge mode to signal the motion information of the current PU, video encoder 20 generates a merge candidate list. In other words, video encoder 20 may perform a motion vector predictor list construction process. The merge candidate list includes a set of merge candidates that indicate the motion information of PUs that are spatially or temporally neighboring to the current PU. That is, in merge mode, a candidate list of motion parameters (e.g., reference index, motion vector, etc.) is constructed, where the candidates can come from spatially and temporally neighboring blocks. In some examples, the candidates may also include manually generated candidates.

Furthermore, in merge mode, video encoder 20 may select a merge candidate from a merge candidate list and may use the motion information indicated by the selected merge candidate as the motion information for the current PU. Video encoder 20 may signal the position of the selected merge candidate in the merge candidate list. For example, video encoder 20 may signal the selected motion vector parameters by transmitting an index into the candidate list. Video decoder 30 may obtain the index into the candidate list (i.e., the candidate list index) from the bitstream. Alternatively, video decoder 30 may generate the same merge candidate list and may determine the selected merge candidate based on the indication of the position of the selected merge candidate. Video decoder 30 may then use the motion information of the selected merge candidate to generate a prediction block for the current PU. That is, video decoder 30 may determine the selected candidate in the candidate list based at least in part on the candidate list index, where the selected candidate specifies the motion vector for the current PU. In this way, at the decoder side, once the index is decoded, all motion parameters for the corresponding block pointed to by the index may be inherited by the current PU.

Skip mode is similar to merge mode. In skip mode, video encoder 20 and video decoder 30 generate and use a merge candidate list in the same manner as they use a merge candidate list in merge mode. However, when video encoder 20 signals the motion information of the current PU using skip mode, video encoder 20 does not signal any residual data for the current PU. Therefore, video decoder 30 can determine the prediction block of the PU based on the reference block indicated by the motion information of the selected candidate in the merge candidate list without using residual data.

AMVP mode is similar to merge mode in that video encoder 20 may generate a candidate list and may select a candidate from the candidate list. However, when video encoder 20 signals the RefPicListX motion information of the current PU using AMVP mode, video encoder 20 may signal the RefPicListX motion vector difference (MVD) of the current PU and the RefPicListX reference index of the current PU in addition to signaling the RefPicListX flag of the current PU. The RefPicListX MVP flag of the current PU may indicate the position of the selected AMVP candidate in the AMVP candidate list. The RefPicListX MVD of the current PU may indicate the difference between the RefPicListX motion vector of the current PU and the motion vector of the selected AMVP candidate. In this way, video encoder 20 can signal the RefPicListX motion information of the current PU by signaling the RefPicListX motion vector predictor (MVP) flag, the RefPicListX reference index value, and the RefPicListX MVD. In other words, the data representing the motion vector of the current PU in the bitstream may include data representing the reference index, an index to the candidate list, and the MVD.

In addition, when the motion information of the current PU is signaled using the AMVP mode, the video decoder 30 may obtain the MVD and MVP flag of the current PU from the bitstream. The video decoder 30 may generate the same AMVP candidate list and may determine the selected AMVP candidate based on the MVP flag. The video decoder 30 may recover the motion vector of the current PU by adding the MVD to the motion vector indicated by the selected AMVP candidate. That is, the video decoder 30 may determine the motion vector of the current PU based on the motion vector indicated by the selected AMVP candidate and the MVD. The video decoder 30 may then use the recovered one or more motion vectors of the current PU to generate a prediction block for the current PU.

When video decoder 30 generates an AMVP candidate list for a current PU, video decoder 30 may derive one or more AMVP candidates based on motion information of PUs that cover positions spatially adjacent to the current PU (i.e., spatially adjacent PUs). When the prediction block of a PU includes a position, the PU may cover the position.

Candidates in the merge candidate list or the AMVP candidate list that are based on the motion information of a PU that is temporally adjacent to the current PU (i.e., a PU at a different time instance than the current PU) may be referred to as TMVPs. That is, TMVPs may be used to improve HEVC coding efficiency, and unlike other coding tools, TMVPs may require access to the motion vectors of frames in the decoded picture buffer, more specifically, in the reference picture list.

The use of TMVP can be enabled or disabled on a per-CVS (coded video sequence) basis, per-slice basis, or on another basis. A syntax element in the SPS (e.g., sps_temporal_mvp_enable_flag) can indicate whether the use of TMVP is enabled for a CVS. Furthermore, when the use of TMVP is enabled for a CVS, the use of TMVP can be enabled or disabled for a specific slice within that CVS. For example, a syntax element in the slice header (e.g., slice_temporal_mvp_enable_flag) can indicate whether the use of TMVP is enabled for a slice. Thus, in an inter-predicted slice, when TMVP is enabled for the entire CVS (e.g., sps_temporal_mvp_enable_flag in the SPS is set to 1), the slice_temporal_mvp_enable_flag is signaled in the slice header to indicate whether TMVP is enabled for the current slice.

To determine the TMVP, the video codec may first identify a reference picture that includes a PU that is co-located with the current PU. In other words, the video decoder may identify a co-located picture. If the current slice of the current picture is a B slice (i.e., a slice that allows inclusion of bi-directionally inter-predicted PUs), video encoder 20 may signal in the slice header a syntax element (e.g., collocated_from_l0_flag) that indicates whether the co-located picture is from RefPicList0 or RefPicList1. In other words, when use of TMVP is enabled for the current slice and the current slice is a B slice (e.g., a slice that allows inclusion of bi-directionally inter-predicted PUs), video encoder 20 may signal in the slice header a syntax element (e.g., collocated_from_l0_flag) that indicates whether the co-located picture is in RefPicList0 or RefPicList1. In other words, to derive the TMVP, the co-located picture will first be identified. If the current picture is a B slice, collocated_from_10_flag is signaled in the slice header to indicate whether the co-located picture is from RefPicList0 or from RefPicList1.

After video decoder 30 identifies a reference picture list that includes a co-located picture, video decoder 30 may use another syntax element (e.g., collocated_ref_idx) that may be signaled in a slice header to identify the picture in the identified reference picture list (i.e., the co-located picture). That is, after identifying a reference picture list, the collocated_ref_idx signaled in the slice header is used to identify the picture in the reference picture list.

The video coder can identify the co-located PU by examining the co-located picture. The TMVP may indicate the motion information of the bottom-right PU of the CU containing the co-located PU or the motion information of the bottom-right PU within the center PU of the CU containing the co-located PU. Therefore, the motion of the bottom-right PU of the CU containing the co-located PU or the motion of the bottom-right PU within the center PU of the CU containing the co-located PU is used. The bottom-right PU of the CU containing the co-located PU may be the PU that covers a position directly below and to the right of the bottom-right sample of the prediction block of the co-located PU. In other words, the TMVP may indicate the motion information of the PU in the reference picture that covers a position co-located with the bottom-right corner of the current PU, or the TMVP may indicate the motion information of the PU in the reference picture that covers a position co-located with the center of the current PU.

When the motion vector identified by the above process (i.e., the motion vector of the TMVP) is used to generate a motion candidate for merge mode or AMVP mode, the video coder may scale the motion vector based on the temporal position (reflected by the POC value). For example, the video coder may increase the magnitude of the motion vector by a larger amount when the difference between the POC values of the current picture and the reference picture is larger, and increase the magnitude of the motion vector by a smaller amount when the difference between the POC values of the current picture and the reference picture is smaller.

The target reference indexes of all possible reference picture lists for temporal merging candidates derived from TMVP may always be set to 0. However, for AMVP, the target reference indexes of all possible reference pictures are set equal to the decoded reference indexes. In other words, the target reference indexes of all possible reference picture lists for temporal merging candidates derived from TMVP are set to 0, while for AMVP, they are set equal to the decoded reference indexes. In HEVC, the SPS may include a flag (e.g., sps_temporal_mvp_enable_flag) and when sps_temporal_mvp_enable_flag is equal to 1, the slice header may include a flag (e.g., pic_temporal_mvp_enable_flag). When both pic_temporal_mvp_enable_flag and temporal_id are equal to 0 for a particular picture, motion vectors from pictures that precede the particular picture in decoding order are not used as TMVP in the decoding of the particular picture or pictures that follow the particular picture in decoding order.

In some examples, video encoder 20 and video decoder 30 ( FIG. 1 ) may use techniques for multi-view and/or 3D video coding, e.g., coding of video data including two or more views. In such examples, video encoder 20 may encode a bitstream of encoded video data including two or more views, and video decoder 30 may decode the encoded video data to provide the two or more views, e.g., to display device 32. In some examples, video decoder 30 may provide multiple views of the video data to enable display device 32 to display 3D video. In some examples, video encoder 20 and video decoder 30 may conform to the 3D-HEVC extension of the HEVC standard, e.g., where multi-view coding or multi-view plus depth coding processes are used. Multi-view and/or 3D video coding may involve coding of two or more texture views and/or views including texture and depth components. In some examples, video data encoded by video encoder 20 and decoded by video decoder 30 includes, or data from which, two or more pictures at any given time instance (i.e., within an "access unit").

In some examples, a device (such as video source 18) may generate the two or more pictures by, for example, capturing a common scene using two or more spatially offset cameras or other video capture devices. Two pictures of the same scene captured simultaneously or nearly simultaneously from slightly different horizontal positions may be used to produce a three-dimensional effect. In some examples, video source 18 (or another component of source device 12) may use depth information or disparity information to generate a second (or other additional) picture of a second (or other additional) view at a given time instance from a first picture of a first view at the given time instance. In this case, a view within an access unit may include a texture component corresponding to the first view and a depth component that may be used with the texture component to generate the second view. The depth or disparity information may be determined by the video capture device that captured the first view, for example, based on camera parameters or other known information about the configuration of the video capture device and the capture of video data for the first view. The depth or disparity information may additionally or alternatively be calculated, for example, by video source 18 or another component of source device 12 from the camera parameters and/or video data in the first view.

To present 3D video, display device 32 may simultaneously or nearly simultaneously display two pictures associated with different views of a common scene, which were captured simultaneously or nearly simultaneously. In some examples, the user of destination device 14 may wear active glasses to quickly and alternately cover the left and right lenses, and display device 32 may quickly switch between the left and right views in sync with the active glasses. In other examples, display device 32 may display both views simultaneously, and the user may wear passive glasses (e.g., with polarized lenses) that filter the views so that the appropriate view enters the user's eyes. In other examples, display device 32 may include a naked-eye stereoscopic display that does not require glasses for the user to perceive a 3D effect.

Multi-view video coding refers to a method of coding multiple views. In the case of 3D video coding, the multiple views may correspond to a left-eye view and a right-eye view, for example. Each of the multiple views includes multiple pictures. The viewer's perception of a 3D scene is due to the horizontal disparity between objects in pictures of different views.

The disparity vector (DV) of a current block in a current picture is a vector that points to a corresponding block in a corresponding picture in a different view than the current picture. Thus, using the DV, a video coder can locate a block in the corresponding picture that corresponds to the current block in the current picture. In this case, the corresponding picture is a picture that is at the same time instance as the current picture but in a different view. The corresponding block in the corresponding picture and the current block in the current picture may contain similar video content; however, there is at least horizontal disparity between the position of the current block in the current picture and the position of the corresponding block in the corresponding picture. The DV of the current block provides a measure of this horizontal disparity between the block in the corresponding picture and the current block in the current picture.

In some cases, there may also be vertical disparity between the position of a block in the corresponding picture and the position of the current block in the current picture. The DV of the current block may also provide a measure of this vertical disparity between the block in the corresponding picture and the current block in the current picture. The DV contains two components (x and y), but in many cases the vertical component will be zero. The current picture of the current view and the corresponding picture of the different view may be displayed at the same time, that is, the current picture and the corresponding picture are pictures of the same time instance.

In video coding, there are generally two types of prediction, commonly referred to as intra-prediction and inter-prediction. In intra-prediction, the video coder predicts a video block in a picture based on already coded blocks in the same picture. In inter-prediction, the video coder predicts a video block in a picture based on already coded blocks in a different picture (i.e., a reference picture). As used in this disclosure, a reference picture generally refers to any picture containing samples that can be used for inter-prediction in the decoding process of subsequent pictures in decoding order. When coding multi-view content relative to the current picture, such as in 3D-HEVC, the reference pictures may belong to the same temporal instance but in different views, or may be in the same view but in different temporal instances. In the case of multi-view coding, such as in 3D-HEVC, inter-picture prediction can include predicting the current video block (e.g., the current coding node of a CU) from another video block in a temporally different picture (i.e., from a different access unit than the current picture), as well as predicting from a different picture in the same access unit as the current picture but associated with a different view than the current picture.

In the latter case of inter-frame prediction, it may be referred to as inter-view coding or inter-view prediction. Reference pictures in the same access unit as the current picture but associated with a different view than the current picture may be referred to as inter-view reference pictures. In multi-view coding, inter-view prediction is performed among pictures captured in different views of the same access unit (i.e., with the same temporal instance) to remove correlation between views. When coding pictures of non-base views, such as dependent views, pictures from the same access unit but different views (e.g., from a reference view, such as a base view) may be added to a reference picture list. Inter-view reference pictures may be placed in any position in a reference picture list, just as with any inter-predicted (e.g., temporal or inter-view) reference picture.

The blocks of the reference picture used to predict the blocks of the current picture are identified by motion vectors. In multi-view coding, there are at least two types of motion vectors. A temporal motion vector (TMV) is a motion vector that points to a block in a temporal reference picture that is in the same view as the block being coded (e.g., as in the first example of inter-frame prediction described above), but in a different temporal instance or access unit than the block being coded, and the corresponding inter-frame prediction is called motion-compensated prediction (MCP). Another type of motion vector is a disparity motion vector (DMV), which points to a block in a picture that is in the same access unit as the current picture but belongs to a different view. With DMV, the corresponding inter-frame prediction is called disparity-compensated prediction (DCP) or inter-view prediction.

In the next section, multi-view (e.g., as in H.264/MVC) and multi-view plus depth (e.g., as in 3D-HEVC) coding techniques will be discussed. Initially, MVC techniques will be discussed. As described above, MVC is a multi-view coding extension of ITU-T H.264/AVC. In MVC, data for multiple views is coded in a time-first order, and accordingly, the decoding order arrangement is referred to as time-first coding. Specifically, the view components (i.e., pictures) of each of the multiple views at a common time instance may be coded, followed by another set of view components at a different time instance, and so on. An access unit may include coded pictures for all views of one output time instance. It should be understood that the decoding order of an access unit is not necessarily equal to the output (or display) order.

A typical MVC decoding order (i.e., bitstream order) is shown in FIG2 . This decoding order arrangement is referred to as time-first coding. Note that the decoding order of access units may not be equal to the output or display order. In FIG2 , S0 to S7 each refer to a different view of the multi-view video. T0 to T8 each represent an output time instance. An access unit may include coded pictures for all views of an output time instance. For example, the first access unit may include all views S0 to S7 for time instance T0, the second access unit may include all views S0 to S7 for time instance T1, and so on.

For the purpose of simplicity, the following definitions may be used in this disclosure:

View component: A coded representation of a view in a single access unit. When a view includes both coded texture and depth representations, a view component consists of a texture view component and a depth view component.

Texture view component: A coded representation of the texture of a view in a single access unit.

Depth view component: A coded representation of the depth of a view in a single access unit.

In FIG2 , each of the views includes several picture sets. For example, view S0 includes the set of pictures 0, 8, 16, 24, 32, 40, 48, 56, and 64, view S1 includes the set of pictures 1, 9, 17, 25, 33, 41, 49, 57, and 65, and so on. For 3D video coding, such as 3D-HEVC, each picture may include two component pictures: one component picture is called a texture view component, and the other component picture is called a depth view component. The texture view components and depth view components within a view's picture set may be considered to correspond to each other. For example, a texture view component within a view's picture set is considered to correspond to a depth view component within the view's picture set, and vice versa (i.e., a depth view component corresponds to its texture view component in the set, and vice versa). As used in this disclosure, a texture view component corresponding to a depth view component may be considered to be a texture view component and a depth view component that are part of the same view of a single access unit.

The texture view component contains the actual image content being displayed. For example, the texture view component may include luma (Y) and chroma (Cb and Cr) components. The depth view component may indicate the relative depth of pixels in its corresponding texture view component. As an example, the depth view component is a grayscale image that only includes luma values. In other words, the depth view component may not convey any image content, but rather provides a measure of the relative depth of pixels in the texture view component.

For example, a pure white pixel in a depth view component indicates that its corresponding pixel in the corresponding texture view component is closer to the viewer's perspective, and a pure black pixel in a depth view component indicates that its corresponding pixel in the corresponding texture view component is farther from the viewer's perspective. Various shades of gray between black and white indicate different depth levels. For example, a dark gray pixel in a depth view component indicates that its corresponding pixel in the texture view component is farther away than a light gray pixel in the depth view component. Because only grayscale is needed to identify the depth of a pixel, the depth view component does not need to include chroma components, as the color values of the depth view component may not serve any purpose.

The use of only luma values (eg, intensity values) to identify depth view components is provided for illustration purposes and should not be considered limiting. In other examples, any technique may be utilized to indicate the relative depth of pixels in a texture view component.

A typical MVC prediction structure for multi-view video coding (including both inter-picture prediction within each view and inter-view prediction) is shown in FIG3 . The prediction direction is indicated by an arrow; the pointed object uses the pointed object as a prediction reference. In MVC, inter-view prediction is supported by disparity motion compensation, which uses the syntax of H.264/AVC motion compensation but allows pictures in different views to be used as reference pictures.

3, eight views are illustrated (with view IDs "S0" through "S7"), and twelve temporal positions ("T0" through "T11") are illustrated for each view. That is, each row in FIG3 corresponds to a view, and each column indicates a temporal position.

Although MVC has so-called base views that can be decoded by H.264/AVC decoders, and MVC can also support stereoscopic view pairs, an advantage of MVC is that it can support instances where more than two views are used as 3D video input and such 3D video is decoded using multiple views. A renderer of a client with an MVC decoder can expect 3D video content with multiple views.

3 is indicated at the intersection of each row and each column. The H.264/AVC standard may use the term frame to refer to a portion of a video. This disclosure may use the terms picture and frame interchangeably.

The pictures in FIG3 are illustrated using blocks containing letters indicating whether the corresponding picture is intra-coded (that is, an I-picture), inter-coded in one direction (that is, as a P-picture), or inter-coded in multiple directions (that is, as a B-picture). In general, prediction is indicated by an arrow, where the pointed picture uses the indicated picture for prediction reference. For example, the P-picture of view S2 at temporal location T0 is predicted from the I-picture of view S0 at temporal location T0.

As with single-view video coding, pictures of a multi-view video coding video sequence can be predictively encoded relative to pictures at different temporal locations. For example, a b-picture of view S0 at temporal location T1 has an arrow pointing to it from an I-picture of view S0 at temporal location T0, indicating that the b-picture is predicted from the I-picture. However, in the case of multi-view video coding, pictures can also be inter-view predicted. That is, a view component can use view components in other views for reference. For example, in MVC, inter-view prediction is achieved as if a view component in another view is an inter-prediction reference. Potential inter-view references are signaled in the Sequence Parameter Set (SPS) MVC extension and can be modified through the reference picture list construction process, which enables flexible ordering of inter-prediction or inter-view prediction references. Inter-view prediction is also a feature of proposed multi-view extensions of HEVC, including 3D-HEVC (Multi-view Plus Depth).

FIG3 provides various examples of inter-view prediction. In the example of FIG3, pictures of view S1 are illustrated as being predicted from pictures at different temporal locations of view S1, and as being inter-view predicted from pictures of views S0 and S2 at the same temporal location. For example, the b-picture of view S1 at temporal location T1 is predicted from each of the B-pictures of view S1 at temporal locations T0 and T2, as well as the b-pictures of views S0 and S2 at temporal location T1.

In some examples, FIG3 can be considered to illustrate texture view components. For example, the I, P, B, and b pictures illustrated in FIG2 can be considered to be texture view components for each of the views. According to the techniques described in this disclosure, for each of the texture view components illustrated in FIG3, there is a corresponding depth view component. In some examples, the depth view component can be predicted in a manner similar to that illustrated in FIG3 for the corresponding texture view components.

MVC can also support the decoding of two views. One of the advantages of MVC is that an MVC encoder can treat more than two views as 3D video input and an MVC decoder can decode such multi-view representations. Therefore, any renderer with an MVC decoder can expect 3D video content with more than two views.

In MVC, inter-view prediction is allowed among pictures in the same access unit (i.e., with the same time instance). When coding a picture in one of the non-base views, if the picture is in a different view but in the same time instance, the picture can be added to the reference picture list. An inter-view prediction reference picture can be placed in any position in the reference picture list, just like any inter-prediction reference picture. As shown in Figure 3, a view component can use view components in other views for reference purposes. In MVC, inter-view prediction is implemented as if the view component in another view is an inter-prediction reference.

In the context of multi-view video coding, there are generally two types of motion vectors. One is called a normal motion vector. It points to a temporal reference picture and its corresponding temporal inter-frame prediction is motion compensated prediction (MCP). The other motion vector is a disparity motion vector (DMV). It points to a picture in a different view (i.e., an inter-view reference picture) and its corresponding inter-frame prediction is disparity compensated prediction (DCP).

Another type of multi-view video coding format introduces the use of depth values (e.g., in 3D-HEVC). In the Multi-view Video Plus Depth (MVD) data format, commonly used for 3D television and free-viewpoint video, texture images and depth maps can be coded independently in multi-view texture pictures. FIG4 illustrates an MVD data format with a texture image and its associated per-sample depth map. The depth range can be limited to a minimum znear and maximum zfar distance from the camera for the corresponding 3D point.

In HEVC, techniques for motion vector prediction may include merge mode, skip mode, and advanced motion vector prediction (AMVP) mode. Generally, according to merge mode and/or skip mode, a current video block (e.g., a PU) inherits motion information, such as a motion vector, prediction direction, and reference picture index, from another previously coded neighboring block (e.g., a spatially neighboring block in the same picture, or a block in a temporal or inter-view reference picture). When implementing merge/skip mode, video encoder 20 constructs a list of merge candidates that are motion information of a reference block in a defined target, selects one of the merge candidates, and signals a candidate list index identifying the selected merge candidate to video decoder 30 in the bitstream.

In implementing merge/skip mode, video decoder 30 reconstructs a merge candidate list according to a defined manner and selects one of the merge candidates in the candidate list indicated by an index. Video decoder 30 may then use the selected one of the merge candidates as the motion vector for the current PU at the same resolution and pointing to the same reference picture as the motion vector of the selected one of the merge candidates. Merge mode and skip mode improve bitstream efficiency by allowing video encoder 20 to signal an index into the merge candidate list rather than all motion information for inter-frame prediction of the current video block.

When implementing AMVP, video encoder 20 constructs a list of candidate motion vector predictors (MVPs) in a defined manner, selects one of the candidate MVPs, and signals a candidate list index identifying the selected MVP in the bitstream to video decoder 30. Similar to merge mode, when implementing AMVP, video decoder 30 reconstructs a list of candidate MVPs in a defined manner and selects one of the MVPs based on the candidate list index.

However, in contrast to merge/skip mode, when implementing AMVP, video encoder 20 also signals a reference picture index and prediction direction, thereby specifying the reference picture to which the MVP, specified by the candidate list index, points. Furthermore, video encoder 20 determines a motion vector difference (MVD) for the current block, where the MVD is the difference between the MVP and the actual motion vector that would have been used for the current block. With AMVP, in addition to the reference picture index, reference picture direction, and candidate list index, video encoder 20 also signals the MVD for the current block in the bitstream. Due to the signaling of the reference picture index and prediction vector difference for a given block, AMVP may not be as efficient as merge/skip mode, but may provide improved fidelity of the coded video data.

FIG5 shows an example of a current video block 47, five spatially neighboring blocks (41, 42, 43, 44, and 45), and a temporal reference block 46 from another picture but in the same view as the current picture. Temporal reference block 46 may, for example, be a co-located block in a picture at a different temporal instance but in the same view as current video block 47. In some examples, current video block 47 and reference video blocks 41-46 may be defined as generally in the HEVC standard currently under development. Reference video blocks 41-46 are labeled A0, A1, B0, B1, B2, and T according to the HEVC standard. Video encoder 20 and video decoder 30 may predict motion information, including TMVs, for current video block 47 based on the motion information of reference video blocks 41-46 according to a motion information prediction mode (e.g., merge/skip mode or AMVP mode). As described in more detail below, the TMVs of a video block may be used together with the DMVs to implement advanced residual prediction according to the techniques of this disclosure.

5 , video blocks 42, 44, 43, 41, and 45 may be respectively to the left, above, above right, below left, and above left relative to current video block 47. However, the number and positions of neighboring blocks 41-45 relative to current video block 47 illustrated in FIG5 are merely examples. A different number of neighboring blocks and/or motion information of blocks at different positions may be considered for inclusion in the motion information prediction candidate list for current video block 47.

The spatial relationship of each of spatially neighboring blocks 42, 44, 43, 41, and 45 to current video block 47 can be described as follows. Luma position (xP, yP) specifies the top-left luma sample of the current block relative to the top-left sample of the current picture; the variables nPSW and nPSH refer to the width and height of the current block for luma. The top-left luma sample of spatially neighboring block 42 is xP-1, yP+nPSH-1. The top-left luma sample of spatially neighboring block 44 is xP+nPSW-1, yP-1. The top-left luma sample of spatially neighboring block 43 is xP+nPSW, yP-1. The top-left luma sample of spatially neighboring block 41 is xP-1, yP+nPSH. The top-left luma sample of spatially neighboring block 45 is xP-1, yP-1. Although described relative to luma position, the current and reference blocks may include chroma components.

Each of spatially neighboring blocks 41-45 may provide a spatial motion information candidate for predicting motion information (e.g., TMV) for current video block 47. A video coder, such as video encoder 20 ( FIG. 1 ) and/or video decoder 30 ( FIG. 1 ), may consider the motion information of spatially neighboring reference blocks in a predetermined order (e.g., scan order). For example, in the case of 3D-HEVC, a video decoder may consider the motion information of reference blocks included in a merge candidate list for merge mode in the following order: 42, 44, 43, 41, and 45. In the illustrated example, spatially neighboring blocks 41-45 are to the left and/or above current video block 47. This arrangement is typical because most video coders code video blocks in raster scan order, starting from the top left of a picture. Therefore, in such examples, spatially neighboring blocks 41-45 will typically be coded before current video block 47. However, in other examples, such as when a video coder codes video blocks in a different order, spatially neighboring blocks 41 - 45 may be located to the right and/or below current video block 47 .

Temporal reference block 46 is located in a temporal reference picture that is coded before (but not necessarily immediately before in coding order) the current picture of current video block 47. Additionally, the reference picture of block 46 does not necessarily precede the picture of current video block 47 in display order. Reference video block 46 may generally be co-located in the reference picture relative to the position of current video block 47 in the current picture. In some examples, reference video block 46 is located to the right and below the position of current video block 47 in the current picture, or overlaps the center position of current video block 47 in the current picture.

6 is a conceptual diagram illustrating an example of deriving inter-view predicted motion vector candidates (IPMVC) and inter-view disparity motion vector candidates (IDMVC) for predicting motion information of a current video block 50, e.g., according to merge/skip mode or AMVP mode. When inter-view prediction is enabled, video encoder 20 and/or video decoder 30 may add a new motion vector candidate, IPMVC or IDMVC, to the motion information candidate list for the current video block 50. IPMVC may predict the TMV for the current video block 50, which video encoder 20 and/or video decoder 30 may use for the ARP of the current video block 50 or another video block, as described in more detail below, according to the techniques of this disclosure. IDMVC may predict the DMV for the current video block 50, which video encoder 20 and/or video decoder 30 may use for the ARP of the current video block 50.

6 , current block 50 is in current view Vm. Video encoder 20 and/or video decoder 30 may use disparity vector (DV) 51 to locate corresponding or reference block 52 in reference view V0. The video coder may determine DV 51 based on camera parameters or according to any of the techniques described herein. For example, the video coder may determine DV 51 for current video block 50 based on DMVs or DVs of neighboring blocks, e.g., using neighbor-based disparity vector derivation (NBDV).

If reference block 52 is not intra-coded and not inter-view predicted, and its reference picture (e.g., reference picture 58 or reference picture 60) has a picture order count (POC) value equal to the POC value of an entry in the same reference picture list of the current video block 50, then video encoder 20 and/or video decoder 30 may derive its motion information (prediction direction, reference picture, and motion vector) after converting the POC-based reference index to the IPMVC for the current video block 50.

In the example of FIG6 , reference video block 52 is associated with a TMV 54 specified in a first reference picture list (RefPicList0) pointing to a first reference picture 58 in reference view V0 and a TMV 56 specified in a second reference picture list (RefPicList1) pointing to a second picture 60 in reference view V0. That current video block 50 inherits TMVs 54 and 56 is illustrated by the dashed arrows in FIG6 . Based on the motion information of reference video block 52, the video coder derives the IPMVC of current video block 50 as at least one of TMV 62 specified in the first reference picture list (RefPicList0) pointing to a first reference picture 66 in current view Vm (e.g., having the same POC as reference picture 58 in the first reference picture list) and TMV 64 specified in the second reference picture list (RefPicList1) pointing to a second picture 68 in current view Vm (e.g., having the same POC as reference picture 60).

Video encoder 20 and/or video decoder 30 may use TMV 62 and/or TMV 64 for the ARP of current video block 50. Video encoder 20 and/or video decoder 30 may also convert DV 51 into an IDMVC for current video block 50, and add the IDMVC to the motion information candidate list of current video block 50 in a different location from the IPMVC. Each of IPMVC or IDMVC may be referred to as an 'inter-view candidate' in this context.

In merge/skip mode, the video coder inserts the IPMVC (if available) preceding all spatial and temporal merge candidates into the merge candidate list. In merge/skip mode, the video coder inserts the IDMVC (block 41 of FIG. 5 ) preceding the spatial merge candidate derived from A0. The conversion of DV 51 to IDMVC can be viewed as a conversion of DV 51 to the DMV of the current video block 50. Video encoder 20 and/or video decoder 30 can use the DMV for the ARP of the current video block 50.

In some cases, the video coder may derive the DV for the current video block. For example, as described above with reference to FIG6 , video encoder 20 and/or video decoder 30 may derive DV 51 for current video block 50. In some examples, the video coder may use NBDV derivation to derive the DV for the current video block. NBDV derivation is used as a disparity vector derivation method in 3D-HEVC.

The proposal for 3D-HEVC uses a texture-first coding order for all views. In other words, for each of the multiple views in the bitstream, the texture component is coded, e.g., encoded or decoded, before any depth component of the view. In some cases, such as for inter-view prediction, a DV is required to code a video block in the texture component of a view in a particular access unit. However, in texture-first coding, the corresponding depth component of the current video block is not available for determining the DV for the current video block. NBDV derivation can be employed by the video coder and is proposed for 3D-HEVC to derive the DV for the current video block in such cases. In the current 3D-HEVC design, the DV derived from the NBDV derivation can be further improved by retrieving depth data from the depth map of the reference view pointed to by the DV from the NBDV process.

DV is used to estimate the displacement between two views. Because neighboring blocks share almost the same motion/disparity information in video decoding, the current video block can use the motion vector information in the neighboring blocks as a good predictor of its motion/disparity information. Following this idea, NBDV derivation uses neighboring disparity information to estimate DV in different views.

Based on the NBDV derivation, the video coder identifies several spatial and temporal neighboring blocks. Two sets of neighboring blocks are utilized. One set comes from spatial neighboring blocks and the other comes from temporal neighboring blocks. The video coder then checks each of the spatial and temporal neighboring blocks in a predefined order determined by the relative priority between the current block and the candidate (neighboring) block. When the video coder identifies a DMV in the candidate's motion information (i.e., a motion vector pointing from a neighboring candidate block to an inter-view reference picture (in the same access unit but in a different view)), the video coder converts the DMV to a DV and returns the associated view order index. For example, the video coder may set the horizontal component of the DV of the current block equal to the horizontal component of the DMV and may set the vertical component of the DV to 0.

3D-HEVC initially adopted the NBDV derivation technique proposed in Zhang et al., "3D-CE5.h: Disparity vector generation results" (Joint Collaborative Team on Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 1st Meeting: Stockholm, Sweden, July 16-20, 2012, document JCT3V-A0097 (MPEG No. m26052, hereinafter "JCT3V-A0097"). JCT3V-A0097 can be downloaded from the following link: http://phenix.int-evry.fr/jct2/doc_end_user/current_document.php?id=89. The entire contents of JCT3V-A0097 are incorporated herein by reference.

In some proposals for 3D-HEVC, when a video coder performs the NBDV derivation process, the video coder sequentially checks the disparity motion vectors in temporally neighboring blocks, the disparity motion vectors in spatially neighboring blocks, and then checks the implicit disparity vector (IDV). The IDV may be the disparity vector of a spatially or temporally neighboring PU coded using inter-view prediction. The IDV may also be referred to as a derived disparity vector. The IDV may be generated when the PU employs inter-view prediction, i.e., a candidate for AMVP or merge mode is derived from a reference block in another view using the disparity vector. This disparity vector is referred to as the IDV. The IDV may be stored in the PU for the purpose of DV derivation. For example, although a block is coded using motion prediction, the derived DV of the block is not discarded for the purpose of coding the following video block. Therefore, when the video coder identifies a DMV or IDV, the video coder may return the identified DMV or IDV.

The simplified NBDV derivation process described in Sung et al., “3D-CE5.h: Simplification of disparity vector derivation for HEVC-based 3D video coding” (ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 Joint Collaboration Group on Video Coding Extension Development, 1st Meeting: Stockholm, Sweden, July 16–20, 2012, document JCT3V-A0126 (MPEG No. m26079, hereinafter “JCT3V-A0126”)) includes an implicit disparity vector (IDV). JCT3V-A0126 can be downloaded from the following link: http://phenix.int-evry.fr/jct2/doc_end_user/current_document.php?id=142.

Further development of the NBDV derivation process for 3D-HEVC is described in Kang et al., “3D-CE5.h: Improvement for disparity vector derivation,” Joint Collaboration Group on Video Coding Extensions Development of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 2nd Meeting: Shanghai, China, October 13–19, 2012, document JCT3V-B0047 (MPEG No. m26736, hereinafter “JCT3V-B0047”). JCT3V-B0047 can be downloaded from the following link: http://phenix.it-sudparis.eu/jct2/doc_end_user/current_document.php?id=236.

In JCT3V-B0047, the NBDV derivation process for 3D-HEVC is further simplified by removing the IDV stored in the decoded picture buffer. Coding gain is also improved through random access point (RAP) picture selection. The video decoder can convert the returned disparity motion vector or IDV into a disparity vector and use the disparity vector for inter-view motion prediction and inter-view residual prediction. Random access refers to the decoding of a bitstream starting from a coded picture that is not the first coded picture in the bitstream. Random access can be achieved by inserting random access pictures or random access points into the bitstream at regular intervals. Example types of random access pictures include instant decoder refresh (IDR) pictures, clean random access (CRA) pictures, and broken link access (BLA) pictures. Therefore, IDR pictures, CRA pictures, and BLA pictures are collectively referred to as RAP pictures. In some examples, a RAP picture may have the NAL unit type equal to BLA_W_LP, BLA_W_RADL, BLA_N_LP, IDR_W_RADL, IDR_N_LP, RSV_IRAP_VCL22, RSV_IRAP_VCL23, or CRA_NUT.

Kang et al., “CE2.h: CU-based disparity vector derivation in 3D-HEVC,” Joint Collaboration Group on Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29/WG11, 4th Meeting, Incheon, South Korea, April 20–26, 2013, document JCT3V-D0181 (MPEG No. m29012, hereinafter “JCT3V-D0181”), proposes techniques for CU-based DV derivation for 3D-HEVC. JCT3V-D0181 can be downloaded from the following link: http://phenix.it-sudparis.eu/jct3v/doc_end_user/current_document.php?id=866.

When the video coder identifies a DMV or IDV, the video coder can terminate the checking process. Therefore, once the video coder finds the DV for the current block, the video coder can terminate the NBDV derivation process. When the video coder is unable to determine the DV for the current block by performing the NBDV derivation process (i.e., when there are no DMVs or IDVs found during the NBDV derivation process), the NBDV is marked as unavailable. In other words, the NBDV derivation process can be considered to return an unavailable disparity vector.

If the video coder is unable to derive the DV of the current block by performing the NBDV derivation process (i.e., if no disparity vector is found), the video coder may use 0DV as the DV of the current PU. 0DV is a DV having both horizontal and vertical components equal to 0. Therefore, even when the NBDV derivation process returns a result that is not available for use, other decoding processes of the video coder that require DV may use a 0 disparity vector for the current block. In some examples, if the video coder is unable to derive the DV of the current block by performing the NBDV derivation process, the video coder may disable inter-view residual prediction for the current block. However, regardless of whether the video coder is able to derive the DV of the current block by performing the NBDV derivation process, the video coder may use inter-view prediction for the current block. That is, if no DV is found after checking all predefined neighboring blocks, a 0 disparity vector may be used for inter-view prediction, while inter-view residual prediction may be disabled for the corresponding CU.

FIG7 is a conceptual diagram illustrating example spatial neighboring blocks relative to a current video block 90 from which a DV can be derived using NBDV. The five spatial neighboring blocks illustrated in FIG7 are a lower left block 96, a left block 95, an upper right block 92, an upper block 93, and an upper left block 94 relative to the current video block. The spatial neighboring blocks may be the lower left, left, upper right, upper, and upper left blocks of the CU covering the current video block. It should be noted that these spatial neighboring blocks of NBDV may be the same spatial neighboring blocks used by the video coder for motion information prediction of the current video block, for example, according to the merge/AMVP mode in HEVC. In such cases, additional memory access by the video coder for the NBDV may not be required because the motion information of the spatial neighboring blocks has already been taken into account for motion information prediction of the current video block.

To check temporally neighboring blocks, the video coder constructs a candidate picture list. In some examples, the video coder may process up to two reference pictures from the current view, i.e., the same view as the current video block, as candidate pictures. The video coder may first insert the co-located reference pictures into the candidate picture list, and then insert the rest of the candidate pictures in ascending order of reference picture index. When a reference picture with the same reference index in two reference picture lists is available, the video coder may insert one of the reference picture lists that is the same as the co-located picture before another reference picture from the other reference picture list. In some examples, the video coder may identify three candidate regions for deriving temporally neighboring blocks from each of the candidate pictures in the candidate picture list. The three candidate regions may be defined as follows:

●CPU: The area co-located with the current PU or current CU.

● CLCU: Largest coding unit (LCU) covering the co-located region of the current block.

BR: The lower right 4×4 block of the CPU.

If the PU covering the candidate area specifies a DMV, the video coder may determine the DV for the current video unit based on the disparity motion vector of the PU.

As discussed above, in addition to DMVs derived from spatial and temporal neighboring blocks, the video coder can also check IDVs. In the proposed NBDV derivation process of 3D-HTM 7.0 and later versions, the video coder sequentially checks DMVs in temporally neighboring blocks, then DMVs in spatially neighboring blocks, and then IDVs. Once a DMV or IDV is found, the process terminates. In addition, the number of spatially neighboring blocks checked during the NBDV derivation process is further reduced to two.

When the video coder checks a neighboring PU (i.e., a spatial or temporal neighboring PU), the video coder may first check whether the neighboring PU has a disparity motion vector. If none of the neighboring PUs has a disparity motion vector, the video coder may determine whether any of the spatial neighboring PUs has an IDV. If one of the spatial neighboring PUs has an IDV and the IDV is coded as a merge/skip mode, the video coder may terminate the checking process and use the IDV as the final disparity vector for the current PU.

As noted above, a video coder may apply an NBDV derivation process to derive the DV for a current block (e.g., a CU, PU, etc.). The disparity vector for the current block may indicate a location in a reference picture (i.e., a reference component) in a reference view. In some 3D-HEVC designs, the video coder is allowed to access depth information for the reference view. In some such 3D-HEVC designs, when the video coder derives the DV for the current block using the NBDV derivation process, the video coder may apply a boosting process to further refine the disparity vector for the current block. The video coder may refine the DV for the current block based on a depth map of a reference picture. The video coder may use a similar refinement process to refine the DMV for backward view synthesis prediction. In this way, depth can be used to refine either the DV or the DMV for backward view synthesis prediction. This refinement process may be referred to herein as NBDV refinement ("NBDV-R"), an NBDV refinement process, or depth-oriented NBDV (Do-NBDV).

When the NBDV derivation process returns a usable disparity vector (e.g., when the NBDV derivation process returns a variable indicating that the NBDV derivation process is capable of deriving the disparity vector of the current block based on the disparity motion vector or IDV of the neighboring block), the video coder may further refine the disparity vector by retrieving depth data from a depth map of the reference picture. In some examples, the refinement process includes the following two steps:

1) Locate the corresponding depth block by the derived DV in a previously coded reference depth view, such as the base view; the size of the corresponding depth block is the same as that of the current PU.

2) Select one depth value from the four corner pixels of the corresponding depth block and convert it to the horizontal component of the refined DV. The vertical component of DV remains unchanged.

The refined DV can be used for inter-view prediction of the current video block, while the unrefined DV can be used for inter-view residual prediction of the current video block. In addition, the refined DV is stored as the motion vector of one PU (if it is coded using backward view synthesis prediction (BVSP) mode), which is described in more detail below. In the proposed NBDV process of 3D-HTM 7.0 and later versions, the depth view component of the base view is accessed regardless of the value of the view order index derived from the NBDV process.

A sub-PU level inter-view motion prediction technique for generating new merge candidates was proposed in An et al., “3D-CE3: Sub-PU level inter-view motion prediction” (Joint Collaborative Team on 3D Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 6th Meeting, Geneva, Switzerland, October 25–November 1, 2013 (document JCT3V-F0110, hereinafter referred to as “JCT3V-F0110”). JCT3V-F0110 can be downloaded from the following link: http://phenix.it-sudparis.eu/jct2/doc_end_user/current_document.php?id=1447 ). The new candidate is added to the merge candidate list.

FIG8 is a conceptual diagram illustrating inter-view motion prediction for sub-prediction units (PUs). As shown in FIG8, current PU 98 in current view V1 may be split into multiple sub-PUs (e.g., four sub-PUs). The disparity vector of each sub-PU may be used to locate a corresponding reference block in reference view V0. Video encoder 20 and/or video decoder 30 may be configured to copy (i.e., reuse) the motion vector associated with each of the reference blocks for use with the corresponding sub-PU of current PU 8.

In one example, a new candidate, called a sub-PU merge candidate, is derived using the following technique. First, the size of the current PU is denoted by nPSW×nPSH, the signaled sub-PU size is denoted by N×N, and the final sub-PU size is denoted by subW×subH. Depending on the PU size and the signaled sub-PU size, the current PU can be divided into one or more sub-PUs as follows:

subW=max(N,nPSW)! =N? N: nPSW;

subH=max(N,nPSH)! =N? N: nPSH;

Video encoder 20 and/or video decoder 30 may set the default motion vector tmvLX to (0,0) and the reference index refLX to -1 for each reference picture list (where X represents reference picture list 0 or reference picture list 1). For each sub-PU in raster scan order, the following applies:

- Add the DV obtained from the DoNBDV derivation process or the NBDV process to the middle position of the current sub-PU to obtain the reference sample position (xRefSub, yRefSub) by the following formula:

xRefSub＝Clip3(0,PicWidthInSamplesL-1,xPSub+nPSWsub/2+((mvDisp[0]+2)>>2))

yRefSub＝Clip3(0,PicHeightInSamplesL-1,yPSub+nPSHSub/2+((mvDisp[1]+2)>>2))

The block covering (xRefSub, yRefSub) in the reference view is used as the reference block of the current sub-PU.

-For the identified reference block:

1) If the identified reference block is coded using a temporal motion vector, then the following applies:

- The associated motion parameters may be used as candidate motion parameters for the current sub-PU.

-Update tmvLX and refLX to the motion information of the current sub-PU.

If the current sub-PU is not the first in raster scan order, all previous sub-PUs inherit the motion information (tmvLX and refLX).

2) Otherwise (the reference block is intra-coded), the motion information of the current sub-PU is set to tmvLX and refLX.

Different sub-PU block sizes can be used in the above-mentioned techniques for sub-PU level inter-view motion prediction, including 4×4, 8×8, and 16×16. The size of the sub-PU block can be signaled in a parameter set such as a view parameter set (VPS).

Advanced Residual Prediction (ARP) is a coding tool that seeks to exploit residual correlation between views to provide additional coding efficiency. In ARP, a residual predictor is generated by aligning the motion information of the current view for motion compensation in the reference view. In addition, a weighting factor is introduced to compensate for quality differences between views. When ARP is enabled for a block, the difference between the current residual and the residual predictor is signaled. That is, the residual of the current block is subtracted from the residual predictor's residual, and the resulting difference is signaled. In some proposals for 3D-HEVC, ARP applies only to inter-coded CUs with a partitioning mode equal to Part_2Nx2N.

FIG9 is a conceptual diagram illustrating an example proposed prediction structure for ARP for temporally predicted video blocks. As proposed in Zhang et al., "CE4: Advanced residual prediction for multiview coding," ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Joint Collaboration Group on Video Coding Extension Development, 4th Meeting: Incheon, South Korea, April 20-26, 2013, document JCT3V-D0177 (MPEG No. m29008, hereinafter "JCT3V-D0177"), ARP applied to CUs with a partition mode equal to Part_2Nx2N was adopted at the 4th JCT3V meeting. JCT3V-D0177 can be downloaded from the following link: http://phenix.it-sudparis.eu/jct3v/doc_end_user/current_document.php? id= 862 .

As shown in FIG. 9 , the video coder calls upon or identifies the following blocks in the prediction of the residual of the current video block 100 in the current picture 102 of the current (eg, dependent) view Vm.

1) Current video block 100 (in view V _m ): Curr

2) An inter-view reference video block 106 in the inter-view reference picture 108 of the reference/base view (V ₀ in FIG9 ): Base. The video coder derives the inter-view reference video block 106 (Curr) based on the DV 104 of the current video block 100. The video coder may determine the DV 104 using NBDV derivation, as described above.

3) A temporal reference video block 112 in a temporal reference picture 114 in the same view (V _m ) as the current video block 100 (Curr): CurrTRef. The video coder derives the temporal reference video block 112 based on the TMV 110 of the current video block 100. The video coder may determine the TMV 110 using any of the techniques described herein.

4) Temporal reference video block 116 in a temporal reference picture 118 in a reference view (i.e., the same view as inter-view reference video block 106 (Base)): BaseTRef. The video coder derives temporal reference video block 116 in the reference view using TMV 110 of current video block 100 (Curr). A vector 121 of TMV+DV may identify temporal reference video block 116 (BaseTRef) relative to current video block 100 (Curr).

When video encoder 20 performs temporal inter-frame prediction on current video block 100 based on temporal reference video block 112 identified by video encoder 20 using TMV 110, video encoder 20 determines the pixel-by-pixel difference between current video block 100 and temporal reference video block 112 as a residual block. Without ARP, video encoder 20 transforms, quantizes, and entropy encodes the residual block. Video decoder 30 entropy decodes the encoded video bitstream, performs inverse quantization and transforms to derive the residual block, and applies the residual block to the reconstruction of reference video block 112 to reconstruct current video block 100.

By using ARP, the video coder determines a residual predictor block that predicts the value of the residual block (i.e., predicts the difference between the current video block 100 (Curr) and the temporal reference video block 112 (CurrTRef)). Video encoder 20 may then only need to encode the difference between the residual block and the residual predictor block, thereby reducing the amount of information included in the encoded video bitstream used to encode the current video block 100. In the temporal ARP example of FIG9, a predictor for the residual of the current video block 100 is determined based on blocks in the reference/base view ( _V0 ) that correspond to the current video block 100 (Curr) and the temporal reference video block 112 (CurrTRef) and are identified by DV 104. The difference between these corresponding blocks in the reference view may be a good predictor of the residual, i.e., the difference between the current video block 100 (Curr) and the temporal reference video block 112 (CurrTRef). Specifically, the video coder identifies an inter-view reference video block 106 (Base) and a temporal reference video block 116 (BaseTRef) in a reference view and determines a residual predictor block based on the difference (BaseTRef-Base) between the inter-view reference video block 106 and the temporal reference video block 116, wherein a subtraction operation is applied to each pixel of the represented pixel array. In some examples, the video coder may apply a weighting factor w to the residual predictor. In such examples, the final predictor for the current block (i.e., the sum of the reference block and the residual predictor block) may be expressed as: CurrTRef+w*(BaseTRef-Base).

FIG10 is a conceptual diagram illustrating an example bidirectional prediction structure for temporal ARP for current video block 120 in the current view (Vm). The above description and FIG9 illustrate unidirectional prediction. When extending ARP to the case of bidirectional prediction, the video coder may apply the above techniques to one or both of the reference picture lists to identify the residual predictor block for current video block 120. Specifically, the video coder may examine one or both of the reference lists for current video block 120 to determine whether one of them contains a TMV that can be used for temporal ARP. In the example illustrated by FIG10, current video block 120 is associated with TMV 130 pointing to a first temporal reference picture 134 in a first reference picture list (RefPicList0), and TMV 132 pointing to a second temporal reference picture 136 in a second reference picture list (RefPicList1).

In some examples, the video coder checks the reference picture lists according to a checking order to determine whether one of them includes a TMV that can be used for temporal ARP, and if the first list includes such a TMV, then the second list need not be checked according to the checking order. In some examples, the video coder checks two reference picture lists and, if both lists include a TMV, determines which TMV to use based on, for example, a comparison of a residual predictor generated using the TMV with respect to a residual of the current video block. Notably, according to current proposals for ARP (e.g., JCT3VC-D0177), the residual prediction process is disabled when the current block uses an inter-view reference picture (in a different view) for one reference picture list.

10 , the video coder may use DV 124 identified for current video block 120, e.g., according to an NBDV derivation process, to identify a corresponding inter-view reference video block 126 (Base) in an inter-view reference picture 128 that is in a different reference view (V ₀ ) than current picture 122 but in the same access unit. The video coder may also use TMVs 130 and 132 for current video block 120 to identify a temporal reference block (BaseTRef) for inter-view reference video block 126 (Base) in respective temporal reference pictures of reference views in two reference picture lists (e.g., RefPicList0 and RefPicList1). In the example of Figure 10, the video decoder identifies a temporal reference video block (BaseTRef) 140 in a temporal reference picture 142 in a first reference picture list (e.g., RefPicList0) and a temporal reference video block (BaseTRef) 144 in a temporal reference picture 146 in a second reference picture list (e.g., RefPicList1) based on the TMVs 130 and 132 of the current video block 120.

The use of TMVs 130 and 132 for current video block 120 in the reference view is illustrated by the dashed arrows in FIG 10. In FIG 10, temporal reference video blocks 140 and 144 in the reference view are referred to as motion compensated reference blocks due to their identification based on TMVs 130 and 132. The video coder may determine a residual predictor block for current video block 120 based on a difference between temporal reference video block 140 and inter-view reference video block 126 or based on a difference between temporal reference video block 144 and inter-view reference video block 126.

Again, the proposed temporal ARP process on the decoder side can be described (see FIG. 10 ) as follows:

1. The video decoder 30 obtains the DV 124 as specified in the current 3D-HEVC, for example, using NBDV derivation pointing to the target reference view (V ₀ ). Then, in a picture 128 of a reference view within the same access unit, the video decoder 30 identifies the corresponding inter-view reference video block 126 (Base) through the DV 124 .

2. Video decoder 30 then uses the motion information (e.g., TMV 130, 132) of current video block 120 to derive motion information for a corresponding inter-view reference video block 126. Video decoder 30 may apply motion compensation for the corresponding inter-view reference video block 126 based on the TMV 130, 132 of the current video block 120 and the derived reference pictures 142, 146 in the reference view of the reference video block 126 to identify a motion-compensated temporal reference video block 140, 144 (BaseTRef) and determine a residual predictor block by determining BaseTRef-Base. The relationship between the current block, the corresponding block (Base), and the motion compensated block (BaseTRef) is shown in Figures 9 and 10. In some examples, a reference picture in the reference view ( _V0 ) having the same POC (Picture Order Count) value as the reference picture of the current view ( _Vm ) is selected as the reference picture for the corresponding block.

3. Video decoder 30 may apply the weighting factor w to the residual predictor block to obtain a weighted residual predictor block, and add the values of the weighted residual block to the predicted samples to reconstruct the current video block 120 .

FIG11 is a conceptual diagram of an example prediction structure for inter-view ARP for inter-view predicted video blocks according to the techniques described in this disclosure. Techniques related to FIG11 are presented in Zhang et al., "CE4: Further Improvements on Advanced Residual Prediction" (Joint Collaboration Team on 3D Video Coding Extensions Development of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11, 6th Meeting, Geneva, Switzerland, October 25–November 1, 2013, hereinafter referred to as "JCT3V-F0123"). JCT3V -F0123 can be downloaded from the following link: http://phenix.it-sudparis.eu/jct2/doc_end_user/current_document.php?id=1460 .

According to the example technique illustrated in FIG 11 , a video coder (e.g., video encoder 20 and/or video decoder 30) may use inter-view residuals from different access units to predict a residual for an inter-view predicted current block. Compared to proposals for ARP in which ARP is not performed when the motion vector of the current block is a DMV and is performed only when the motion vector of the current video block is a TMV, the example technique of FIG 11 performs ARP using DMVs.

11 may be performed by a video coder, such as video encoder 20 or video decoder 30, when the motion vector of a current video block 150 (Curr) in a current picture 152 is a DMV 154, and an inter-view reference video block 156 (Base) in an inter-view reference picture 158 in a reference view (V0) contains at least one TMV 160. In some examples, DMV 154 may be a DV that is converted to a DMV to serve as an IDMVC for motion information prediction of the current video block 150.

The video coder uses DMV 154 of current video block 150 to identify inter-view reference video block 156 (Base) in inter-view reference picture 158. The video coder uses TMV 160 of inter-view reference video block 156 and an associated reference picture (e.g., temporal reference picture 164 in reference view (V ₀ )) along with the DMV to identify temporal reference video block 162 (BaseTRef) in temporal reference picture 164 in reference view (V ₀ ). Identification of temporal reference video block 162 (BaseTRef) based on TMV 160 and DMV 154 is represented by dashed vector 170 (TMV+DMV). The video coder also uses TMV 160 to identify temporal reference video block 166 (CurrTRef) in temporal reference picture 168 in current view (V _m ). Temporal reference video block 162 (BaseTRef) in reference view (V ₀ ) and temporal reference video block 166 (CurrTRef) in current view (V _m ) may be within the same access unit, i.e., temporal reference picture 164 in reference view (V ₀ ) and temporal reference picture 168 in current view (V _m ) may be in the same access unit.

A video coder (e.g., video encoder 20 and/or video decoder 30) may then calculate an inter-view residual predictor block in a different access unit from current video block 150 based on the pixel-by-pixel difference between these latter two blocks (i.e., the difference between temporal reference video block 166 in the current view and temporal reference video block 162 in the reference view, or CurrTRef - BaseTRef). The difference signal (denoted as the inter-view residual predictor) may be used to predict the residual for current video block 150. The prediction signal for current video block 150 may be the sum of the inter-view predictor (i.e., inter-view reference video block 156 (Base)) and the predicted inter-view residual in the different access unit determined based on the difference between temporal reference video block 166 in the current view and temporal reference video block 162 in the reference view. In some examples, a weighting factor w is applied to the predicted inter-view residual in the different access units. In such examples, the prediction signal for current video block 150 may be: Base + w * (CurrTRef - BaseTRef).

In some examples, the video coder may determine a target reference picture in a target access unit for inter-view ARP, e.g., similar to the determination of a target reference picture for temporal ARP, as discussed above. In some examples, as discussed above with reference to JCT3V-D0177, the target reference picture for each reference picture list is the first reference picture in the reference picture list. In other examples, the target reference picture (e.g., target POC) for one or both reference picture lists may be signaled from video encoder 20 to video decoder 30, e.g., on a PU, CU, slice, picture, or other basis. In other examples, the target reference picture for each reference picture list is the temporal reference picture in the reference picture list that has the smallest POC difference and a smaller reference picture index than the current block. In other examples, the target reference picture for both reference picture lists is the same.

If the picture containing the temporal reference video block in the reference view indicated by TMV 160 is in a different access unit (time instance) than the target ARP reference picture, the video coder may scale TMV 160 to the target reference picture (e.g., target reference picture 164) to identify temporal reference video block 162 (BaseTRef) in the reference view for inter-view ARP. In these examples, the video coder locates temporal reference video block 162 in the access unit containing the target ARP reference picture. The video coder may scale TMV 160 by POC scaling. Furthermore, the scaled TMV is used to identify a temporal reference video block (CurrTRef) 166 in the current view that is located in the target ARP reference picture.

In some examples, the video coder scales TMV 160 to an LX (X is 0 or 1) target reference picture, where LX corresponds to the RefPicListX of the PU that includes the TMV. In some examples, the video coder may scale TMVs from either or both of RefPicList0 or RefPicList1 to an L0 or L1 target reference picture, respectively. In some examples, the video coder scales TMV 160 to an LX target reference picture, where X satisfies the condition that DMV 154 of current video block 150 (e.g., current PU) corresponds to RefPicListX.

Similarly, in some examples, the video coder scales DMV 154 to the target reference view of the ARP before identifying inter-view reference video blocks 156 in reference pictures 158 in the target reference view. The video coder may scale DMV 154 by view order difference scaling. The target reference view may be predetermined and known by video encoder 20 and video decoder 30, or may be signaled from video encoder 20 to video decoder 30, for example, on a PU, CU, slice, picture, or other basis.

In some examples of inter-view ARP, a video coder (e.g., video encoder 20 and/or video decoder 30) may derive a prediction signal for current block 150 using the same prediction structure illustrated in FIG. 11 and the identified reference video blocks 156, 162, and 166, but may determine a residual predictor block based on the difference between reference blocks 156 and 162 in a reference view rather than between reference blocks 162 and 166 in different access units. In such examples, the video coder may apply a weighting factor to the other sample array (e.g., the difference between reference blocks 156 and 162 in a reference view) and accordingly derive a prediction signal for current video block 150 as follows: CurrTRef+w*(Base-BaseTRef). In some examples of inter-view ARP, the video coder may use various interpolation filters, including bilinear filters, to derive reference video blocks 156, 162, and 166 when they are aligned with fractional pixel positions.

Although FIG11 illustrates an inter-view ARP example in which the TMV associated with an inter-view reference block and the inter-view reference video block's associated reference picture are used to identify temporal reference video blocks in the current and reference views, in other examples, other TMVs and associated reference pictures may be used to identify temporal reference video blocks in the current and reference views. For example, if the DMV for the current video block is from the first reference picture list of the current video block (e.g., RefPicList0 or RefPicList1), the video coder may use the TMV corresponding to the second reference picture list of the current block and the associated reference picture from the second reference picture list of the current video block (e.g., the other of RefPicList0 or RefPicList1). In such examples, the video coder may identify the temporal reference video block in the current view in the reference picture associated with the TMV, or scale the TMV to the target access unit and target reference picture of the ARP to identify the temporal reference video block in the current view. In such examples, the video coder may identify the temporal reference video block in the reference view in a reference picture in the same access unit as the reference picture in which the temporal reference video block in the current view is located. In other examples, instead of the TMV of an inter-view reference video block or the TMV of another reference picture list of the current video block, the video coder may similarly use the TMV and associated reference pictures derived from motion information of spatially or temporally neighboring video blocks of the current video block to identify temporal reference video blocks in the current and reference views of the ARP.

In the following description, if the corresponding reference of a reference picture list is a temporal reference picture and ARP is applied, the ARP process is labeled as temporal ARP. Otherwise, if the corresponding reference of a reference picture list is an inter-view reference picture and ARP is applied, the ARP process is labeled as inter-view ARP.

In some proposals for ARP, three weighting factors, namely 0, 0.5, and 1, can be used. The weighting factor that produces the minimum rate-distortion cost for the current CU is selected as the final weighting factor, and the corresponding weighting factor index (0, 1, and 2, which correspond to weighting factors of 0, 1, and 0.5, respectively) is transmitted in the bitstream at the CU level. All PU predictions in a CU share the same weighting factor. When the weighting factor is equal to 0, ARP is not used for the current CU.

Aspects related to ARP for 3D-HEVC are described in Zhang et al., “3D-CE4: Advanced residual prediction for multiview coding” (3rd meeting of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29/WG 11 Joint Collaboration Group on Development of Extensions for Video Coding, Geneva, Switzerland, January 17–23, 2013, document JCT3V-C0049 (MPEG No. m27784, hereinafter “JCT3V-C0049”). JCT3V-C0049 can be downloaded from the following link:

http://phenix.int-evry.fr/jct3v/doc_end_user/current_document.php?id=487 .

In JCT3V-C0049, the reference pictures of different PUs coded with non-zero weighting factors may be different between different PUs (or current video blocks). Therefore, it may be necessary to access different pictures from the reference view to generate motion-compensated blocks (BaseTRef) (e.g., inter-view reference video blocks 116, 140, and 144 in Figures 9 and 10), or corresponding inter-view reference video blocks in the reference view (Base), such as inter-view reference video blocks 106 and 126 in Figures 9 and 10.

When the weighting factor is not equal to 0, for temporal residual, the motion vector of the current PU is scaled towards the fixed picture before motion compensation is performed for the residual and residual predictor generation process. When ARP is applied to inter-view residual, the temporal motion vector of the reference block (e.g., block 156 in Figure 11) is scaled towards the fixed picture before motion compensation is performed for the residual and residual predictor generation process. For both cases (i.e., temporal residual or inter-view residual), a fixed picture is defined as the first available temporal reference picture in each reference picture list. When the decoded motion vector does not point to a fixed picture, it is first scaled and then used to identify CurrTRef and BaseTRef.

This reference picture used for ARP is called the target ARP reference picture. It should be noted that when the current slice is a B slice, the target ARP reference picture is associated with a specific reference picture list. Therefore, two target ARP reference pictures can be used.

The availability check of the target ARP reference picture can be performed as follows: First, the target ARP reference picture associated with one reference picture list X (where X is 0 or 1) is indicated by RpRefPicLX, and the picture in the view with a view order index equal to the view order index derived from the NBDV derivation process and the same POC value of RpRefPicLX is indicated by RefPicInRefViewLX.

ARP is disabled for reference picture list X when one of the following conditions is false:

-RpRefPicLX is not available

-RefPicInRefViewLX is not stored in the decoded picture buffer

-RefPicInRefViewLX is not included in any of the reference picture lists of the corresponding block located by the DV from the NBDV derivation process or the DMV associated with the current block (e.g., block 106 in Figure 9 or block 156 in Figure 11), ARP is disabled for this reference picture list.

When ARP is applied, a bilinear filter may be used when generating the residual and the residual predictor. That is, for example, blocks 106, 112, and 116 in FIG9 may be generated using a bilinear filter.

FIG12 is a conceptual diagram illustrating an example prediction structure for bidirectional ARP that uses inter-view prediction for one reference picture list and temporal prediction for another reference picture list. The example technique of FIG12 may be performed by video encoder 20 and/or video decoder 30 when one prediction direction of the bidirectional prediction of the current video block 250 uses temporal prediction (e.g., for reference picture list X) and the other prediction direction of the current video block 250 uses inter-view prediction (e.g., for reference picture list Y (Y=1-X)).

12 , current video block 250 may be associated with TMV 210 and DMV 254. Video encoder 20 and/or video decoder 30 may be configured to identify and access reference blocks of reference picture list X (i.e., first prediction direction) in a similar manner as described above with reference to FIG.

The video encoder 20 and/or video decoder 30 identify subsequent blocks used in the prediction of the residual of the current video block 250 in the current picture 253 of the current (e.g., dependent) view Vm. The video encoder 20 and/or video decoder 30 identify an inter-view reference video block 206 (BaseX) in the inter-view reference picture 258 of the reference/base view ( _V0 in FIG. 12). The video encoder 20 and/or video decoder 30 identify the inter-view reference video block 206 based on the DV 204 of the current video block 250 (Curr). The video encoder 20 and/or video decoder 30 may determine the DV 204 using NBDV derivation, as described above.

The video encoder 20 and/or video decoder 30 may further identify a temporal reference video block 212 (CurrTRefX) in a temporal reference picture 270 in the same view (V _m ) as the current video block 250 (Curr). The video encoder 20 and/or video decoder 30 may identify the temporal reference video block 212 using the TMV 210 of the current video block 250. The video encoder 20 and/or video decoder 30 may determine the TMV 210 using any of the techniques described herein. The video encoder 20 and/or video decoder 30 may further identify a temporal reference video block 216 (BaseTRefX) in a temporal reference picture 272 in a reference view (i.e., the same view) as the inter-view reference video block 206 (BaseX). The video encoder 20 and/or video decoder 30 may identify the temporal reference video block 216 in the reference view using the TMV 210 of the current video block 250 (Curr). The vector 220 of TMV 210+DV 204 may identify a temporal reference video block 216 (BaseTRefX) relative to the current video block 250 (Curr). As can be seen in FIG12, for reference picture list X (i.e., the first prediction direction), video encoder 20 and/or video decoder 30 are configured to identify and access three reference blocks (i.e., reference blocks 206, 212, and 216).

The video encoder 20 and/or video decoder 30 may be configured to identify and access reference blocks of reference picture list Y (i.e., first prediction direction) in a similar manner as described above with reference to FIG. 11 . The video encoder 20 and/or video decoder 30 identifies an inter-view reference video block 256 (BaseY) in an inter-view reference picture 258 of a reference/base view ( V ₀ in FIG. 12 ). The video encoder 20 and/or video decoder 30 identifies the inter-view reference video block 256 based on the DMV 254 of the current video block 250 (Curr).

Video encoder 20 and/or video decoder 30 may further identify a temporal reference video block 273 (CurrTRefY) in a temporal reference picture 268 in the same view (V _m ) as the current video block 250 (Curr). Video encoder 20 and/or video decoder 30 may identify temporal reference video block 273 using TMV' 285 of current video block 250. The video coder uses TMV' 285 and the associated reference picture of inter-view reference video block 256 (e.g., temporal reference picture 265 in reference view (V _{0 )} ) and DMV 254 to identify temporal reference video block 271 (BaseTRefY) in temporal reference picture 265 in reference view (V 0 ). The identification of temporal reference video block 271 (BaseTRefY) based on TMV' 285 and DMV 254 is represented by the dashed vector (TMV'+DMV). The temporal reference video block 271 (BaseTRefY) in the reference view (V ₀ ) and the temporal reference video block 273 (CurrTRefY) in the current view (V _m ) may be in the same access unit, that is, the temporal reference picture 265 in the reference view (V ₀ ) and the temporal reference picture 268 in the current view (V _m ) may be in the same access unit.

As can be seen in FIG. 12 , for reference picture list Y (ie, the second prediction direction), video encoder 20 and/or video decoder 30 are configured to identify and access three additional reference blocks (ie, reference blocks 256 , 271 , and 273 ).

The aforementioned techniques for ARP in 3D-HEVC exhibit several drawbacks. For example, performing block-level or PU-level ARP in conjunction with bidirectional prediction increases the number of memory accesses for motion information, as bidirectional prediction inherently involves using motion information for two different reference picture lists. Furthermore, the number of reference blocks identified and accessed is high. Consequently, combining bidirectional prediction with ARP increases decoder complexity.

This disclosure proposes various example techniques to address the above-mentioned issues of ARP in order to reduce video decoder complexity. Each of the techniques listed below reduces the number of memory accesses required to perform ARP and other associated video coding techniques relative to current proposals for ARP.

FIG13 is a conceptual diagram illustrating an example prediction structure for bidirectional ARP using inter-view prediction for one reference picture list and temporal prediction for another reference picture list in accordance with the techniques of this disclosure. In the example of FIG13 , video encoder 20 and/or video decoder 30 is configured to code current video block 250 using bidirectional prediction and ARP. The bidirectional prediction includes temporal prediction (e.g., a first prediction direction) for reference picture list X and inter-view prediction (e.g., a second prediction direction) for reference picture list Y.

According to the technique of FIG13 , video encoder 20 and/or video decoder 30 are configured to identify reference block 206 (BaseX), reference block 216 (BaseTrefX), and reference block 212 (CurrTrefX) of reference picture list X (e.g., a first prediction direction) in the same manner as described above with reference to FIG12 . TMV 210 is used to identify reference block 216 (BaseTrefX) and reference block 212 (CurrTrefX) relative to reference block 206 (BaseX) and current video block 250, respectively. Additionally, video encoder 20 and/or video decoder 30 are configured to identify reference block 256 (BaseY) of reference picture list Y (e.g., a second prediction direction) using DMV 254 (i.e., in the same manner as described above with reference to FIG12 ).

However, video encoder 20 and/or video decoder 30 do not use temporal motion information associated with reference block 256 (BaseY) to identify CurrTrefY and BaseTrefY. Instead, according to the techniques of this disclosure, video encoder 20 and/or video decoder 30 may be configured to use temporal motion information of reference list X (i.e., TMV 210) to identify CurrTrefY and BaseTrefY. As shown in FIG. 13 , video encoder 20 and/or video decoder 30 is configured to use TMV 210 to identify reference block 290 (BaseTrefY) in view V ₀ relative to reference block 256 (BaseY). That is, video encoder 20 and/or video decoder 30 is configured to use both DMV 254 and TMV 210 to identify reference block 290 (BaseTrefY). Video encoder 20 and/or video decoder 30 is further configured to use TMV 210 to identify CurrTrefY in the same view (Vm) as current video block 250. Thus, reference block 212 serves as both CurrTrefX and CurrTrefY.Thus, using the techniques of this disclosure, video encoder 20 and/or video decoder 30 only identifies and accesses five reference blocks instead of six when performing ARP using bi-prediction.

In summary, by identifying a reference block for inter-view ARP corresponding to reference picture list Y (e.g., the second prediction direction), video encoder 20 and video decoder 30 can use temporal motion information associated with temporal prediction of reference picture list X (e.g., TMV 210 in FIG. 13 ) to identify reference blocks in different access units (i.e., reference blocks 290 and 212). In this case, when performing inter-view ARP, there is no need to generate a reference block in an access unit different from the current view (i.e., reference block 212) because it is the same reference block identified for temporal ARP of reference picture list X. That is, reference block 212 is used for both temporal ARP and inter-view ARP.

In this way, temporal motion information used for the first prediction direction is reused for the second prediction direction. Consequently, fewer memory accesses need to be made to the temporal motion information because the temporal motion information of the block identified by the motion vector corresponding to the first encoded block in the second prediction direction does not need to be accessed, thereby allowing for faster video decoding. Additionally, the total number of reference blocks used when performing ARP can be reduced from 6 to 5, which results in less computational complexity in interpolation using multiplication and addition operations. Similarly, when performing bidirectional inter-frame prediction, video encoder 20 can be configured to reuse the temporal motion information used for the first prediction direction when encoding the second prediction direction.

In another example of the present disclosure, the video encoder 20 and the video decoder 30 may be configured to perform a simplified ARP process when one prediction direction of bi-directional prediction (e.g., corresponding to reference picture list X) corresponds to a temporal reference picture and the other prediction direction (e.g., corresponding to reference picture list Y) corresponds to an inter-view reference picture. In this case, for temporal ARP corresponding to reference picture list X, the video encoder 20 and the video decoder 30 may be configured to use the disparity motion vector (MVY) associated with the inter-view reference picture to identify a reference block in the reference view (e.g., reference block 273 in FIG. 12 ) instead of using the disparity vector derived from the NBDV/DoNBDV derivation process. At the same time, the disparity vector from the NBDV or DoNBDV process remains unchanged and can still be used in inter-view motion prediction to generate IPMVC or IDMVC.

It should be noted that the above method can be applied to both PU-level ARP and block-level ARP. PU-level and block-level ARP will be described in more detail below.

Techniques for block-level ARP will now be discussed. Unlike the above description, in which all blocks within a PU share the same motion information for ARP (sometimes referred to as PU-level ARP), in block-level ARP, a PU is split into several sub-blocks (e.g., 8×8 sub-blocks) and each sub-block is associated with its own derived motion information to perform ARP. That is, each sub-block shares the same motion information as the current PU. However, a derived motion vector (i.e., a disparity vector in temporal ARP or a temporal motion vector in inter-view ARP) can be determined for each sub-block.

FIG14 is a conceptual diagram illustrating block-based temporal ARP. As shown in FIG14 , a current picture 302 includes a current block 300 (Curr) divided into four sub-blocks 300a, 300b, 300c, and 300d. A motion vector 310 (mvLX) is a motion vector used to perform inter-frame prediction on the current block 300. The motion vector 310 points to a reference block 312 (CurrTref) in a reference picture 314 (which includes sub-blocks 312a-d). The current picture 302 and the reference picture 314 are in the same view (Vm).

For block-based temporal ARP, the default derived motion vector is used for each of sub-blocks 300a-d. For temporal ARP, the default derived motion vector is the disparity vector denoted by DV[i] for the i-th sub-block in FIG14 and can be derived using the NBDV derivation process, as in the current ARP. That is, the NBDV derivation process can be performed on each of sub-blocks 300a-d to derive the DV for each of sub-blocks 300a-d. Each of the derived DVs points to a specific reference block 306a-d (Base) in reference view 308. For example, DV 304 (DV[0]) points to reference block 306a, and DV 305 (DV[1]) points to reference block 306b.

Reference view 308 is at the same time instance as current picture 302, but in another view. When the center position of one of sub-blocks 312a-d within reference block 312 contains a disparity motion vector, the disparity vector DV[i] of the corresponding one of current sub-blocks 300a-d is updated to use the disparity motion vector. That is, for example, if the center position of reference sub-block 312a corresponding to current sub-block 300a has an associated disparity motion vector, the disparity motion vector associated with reference sub-block 312a is used as the disparity vector for sub-block 300a.

Once each of the reference blocks 306a-d has been identified, the motion vector 310 can be used to find the reference block 316a-d (BaseTRef) in the reference picture 318. The reference picture 318 is in a different time instance and a different view than the current picture 302. The residual predictor can then be determined by subtracting the reference block 306a-d (Base) from the corresponding reference block 316a-d (BaseTref). ARP can then be performed on each of the sub-blocks 300a-d.

FIG15 is a conceptual diagram illustrating block-based inter-view ARP. As shown in FIG15 , a current picture 352 includes a current block 350 (Curr) divided into four sub-blocks 350a, 350b, 350c, and 350d. A disparity motion vector 360 (DMV) is a disparity motion vector used to perform inter-view prediction on the current block 350. The disparity motion vector 360 points to a reference block 356 (Base) in a reference picture 358 (which includes sub-blocks 356a-d). The current picture 352 and the reference picture 358 are in the same time instance but in different views.

For block-based inter-view ARP, the default derived motion vector is used for each of sub-blocks 350a-d. For inter-view ARP, the default derived motion vector is the motion vector denoted by mvLX[i] for the i-th sub-block in FIG15 and can be set to the temporal motion vector covering the center position of each of sub-blocks 356a-d, as in the current ARP. That is, the block covering the center position of the i-th 8×8 block within sub-block 356 contains the temporal motion vector, and mvLX[i] is updated to that temporal motion vector.

Each of the derived motion vectors points to a specific reference block 366a-d (BaseTref) in the reference view 368. For example, motion vector 354 (mvLX[0]) points to reference block 368a and motion vector 355 (mvLX[3]) points to reference block 366d.

Once each of the reference blocks 366a-d has been identified, the disparity motion vector 360 can be used to find a reference block 362a-d (CurrTRef) in the reference picture 364. The reference picture 364 is in a different time instance than the current picture 352. The residual predictor can then be determined by subtracting the reference block 362a-d (CurrTref) from the corresponding reference block 366a-d (BaseTref). ARP can then be performed on each of the sub-blocks 350a-d.

As described above, for block-based temporal ARP, the motion vector 310 is accessed and used to locate the reference block 312 (CurrTref). Likewise, for block-based inter-view ARP, the disparity motion vector 360 is accessed and used to locate the reference block 356 (Base).

FIG16 is a conceptual diagram illustrating block-based ARP using sub-PU merge candidates. When sub-PU inter-view motion prediction is enabled, the motion information of one reference block (406) identified by the derived disparity vector 410 from the NBDV/DoNBDV derivation process is accessed to derive the sub-PU merge candidates. After the sub-PU merge candidates are determined, that is, for each sub-PU within block 400 (Curr), it will have its temporal motion information, which is indicated by motion vector 404 (mvLX[0]) and motion vector 405 (mvLX[1]) as shown in FIG14. Motion vectors 404 and 405 can be used to identify reference block 412 (CurrTref) and reference block 416 (BaseTref).

When the ARP process is invoked, motion information is also accessed for each sub-block (e.g., an 8x8 block) within the reference block 412 (CurrTRef). When corresponding sub-blocks 412a-d (CurrRef) are associated with disparity motion vectors, the disparity motion vectors can be used to locate a reference block (e.g., block 406) in the reference view.

Therefore, it may be necessary to access the motion information of two blocks. That is, access the motion information of one block identified by the DV from the NBDV/DoNBDV process for the sub-PU inter-view merge candidate. In addition, access the motion information of the block identified by any derived temporal motion information.

The aforementioned techniques for ARP in 3D-HEVC exhibit several disadvantages. As an example, when both inter-sub-PU view merge prediction and block-level temporal ARP are used to decode a PU, motion information of two reference blocks is accessed. One is the reference block in the reference view identified by the disparity vector derived from the DoNBDV/NBDV derivation process. In addition, the corresponding motion information is accessed to derive the inter-sub-PU view merge candidate. After deriving the inter-sub-PU view merge candidate, another block in the temporal reference picture is accessed to check whether the block in the temporal reference picture contains the disparity motion vector. The dual access of motion information associated with different blocks significantly increases the complexity of the video decoder design and can reduce the decoder processing capacity.

As another disadvantage, when using sub-PU (i.e., block-level) ARP, the disparity motion vector associated with the reference block pointing to the temporal motion vector of the current block is used to update the default disparity vector. For a sub-block, even if the sub-block has the same disparity motion vector as its neighboring blocks (left, above, below, or right), the ARP process is still performed at each sub-block, thereby increasing the complexity of the video decoder.

This disclosure proposes various example techniques to address the above-mentioned issues of ARP in order to reduce video decoder complexity. Each of the techniques listed below reduces the number of memory accesses required to perform ARP and other associated video coding techniques relative to current proposals for ARP.

In one example of the present disclosure, when inter-sub-PU view motion prediction is enabled and the inter-sub-PU view merge candidate (which corresponds to temporal motion information) is applied to the current PU, video encoder 20 and video decoder 30 may be configured to disable block-level ARP. Instead, PU-level ARP may be enabled.

When sub-PU inter-view motion prediction is enabled and PU-level ARP is applied, video encoder 20 and video decoder 30 can determine temporal motion information for each sub-PU. That is, each sub-PU has its own temporal motion information. However, video encoder 20 and video decoder 30 determine the same disparity vector for all sub-PUs. The temporal motion information and disparity vector are used to derive the residual and residual predictor, as described above. It should be noted that when sub-PU inter-view motion prediction is applicable, the ARP process used is temporal ARP.

The following example techniques are used when inter-view motion prediction of sub-PUs is not used. In one example, for temporal ARP, video encoder 20 and video decoder 30 may determine a disparity vector for each sub-PU. In one example, the determined disparity vector may be disparity motion information derived from a reference block of the current sub-PU identified by the temporal motion information of the current sub-PU in a temporal reference picture. For inter-view ARP, video encoder 20 and video decoder 30 may determine temporal motion information for each sub-PU. In one example, the temporal motion information may be derived from a reference block of the current sub-PU identified by the disparity motion information of the current sub-PU in an inter-view reference picture.

In another example of the present disclosure, when sub-PU inter-view motion prediction is enabled, video encoder 20 and video decoder 30 may be configured to disable block-level ARP for one prediction direction corresponding to a specific reference picture list if the associated reference picture is a temporal reference picture. In this case, video encoder 20 and video decoder 30 may be configured to enable PU-level ARP only for this prediction direction.

In one example, the following process is applied. If the current PU uses an inter-view merge candidate, video encoder 20 and video decoder 30 determine temporal motion information for each sub-PU. However, video encoder 20 and video decoder 30 determine the same disparity vector for all sub-PUs. The temporal motion information and disparity vector are used to derive the residual and residual predictor, as described above.

Otherwise, if the current PU uses one of the other available merge candidates (i.e., not an inter-view merge candidate), the video encoder 20 and the video decoder 30 apply PU-level temporal ARP, where if the corresponding reference picture is a temporal reference picture, all blocks within the current PU share the same motion information for one prediction direction. For one prediction direction, if the corresponding reference picture is an inter-view reference picture, the video encoder 20 and the video decoder 30 use PU-level inter-view ARP, where all blocks within the current PU share the same motion information. In this case, block-level ARP can also be applied, where blocks within the current PU can share the same disparity motion information and different temporal motion information.

In another example of the present invention, when block-level ARP is enabled, video encoder 20 and video decoder 30 may determine the block size for performing ARP based on motion information. In one example, for one prediction direction corresponding to a particular reference picture list, if the corresponding reference picture is a temporal reference picture, video encoder 20 and video decoder 30 may use block-level ARP. In this case, the current block has the same disparity motion information as its neighboring blocks (e.g., the left, above, below, and/or right neighboring blocks). Furthermore, the current block and the neighboring blocks are merged together, and ARP is performed once for the merged block.

In another example of the present invention, for one prediction direction corresponding to a reference picture list, if the corresponding reference picture is an inter-view reference picture, the video encoder 20 and the video decoder 30 may use block-level ARP. In this case, the current block has the same temporal motion information as its neighboring blocks (e.g., the left, above, below, and/or right neighboring blocks). In addition, the current block and the neighboring blocks are merged together and ARP is performed once for the merged block.

FIG17 is a block diagram illustrating an example video encoder 20 that may be configured to perform the techniques described in this disclosure. The video encoder 20 may perform intra- and inter-coding on video blocks within a video slice. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture. Inter-coding relies on temporal or inter-view prediction to reduce or remove redundancy in video within adjacent frames or pictures of a video sequence. Intra-mode (I-mode) may refer to any of several spatially-based compression modes. Inter-modes, such as unidirectional prediction (P-mode) or bidirectional prediction (B-mode), may include any of several temporally-based compression modes.

In the example of FIG17 , video encoder 20 includes video data memory 235, prediction processing unit 241, decoded picture buffer (DPB) 264, summer 251, transform processing unit 252, quantization processing unit 255, and entropy encoding unit 257. Prediction processing unit 241 includes motion estimation unit 242, motion and disparity compensation unit 244, advanced residual prediction (ARP) unit 245, and intra-prediction processing unit 246. For video block reconstruction, video encoder 20 also includes inverse quantization processing unit 259, inverse transform processing unit 260, and summer 262. A deblocking filter (not shown in FIG17 ) may also be included to filter block boundaries to remove blockiness artifacts from the reconstructed video. If necessary, the deblocking filter will typically filter the output of summer 262. In addition to the deblocking filter, additional loop filters (in-loop or post-loop) may also be used.

In various examples, one or more hardware units of video encoder 20 may be configured to perform the techniques of this disclosure. For example, motion and disparity compensation unit 244 and ARP unit 245 may perform the techniques of this disclosure alone or in combination with other units of video encoder 20.

As shown in FIG. 17 , video encoder 20 receives video data (e.g., video data blocks (e.g., luma blocks, chroma blocks, or depth blocks)) within a video frame to be encoded (e.g., a texture image or a depth map). Video data memory 235 may store video data to be encoded by components of video encoder 20. Video data stored in video data memory 40 may be obtained, for example, from video source 18. DPB 264 is a memory buffer that stores reference video data for use by video encoder 20 in encoding video data (e.g., in intra- or inter-coding modes, also known as intra- or inter-prediction coding modes). Video data memory 235 and DPB 264 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 235 and DPB 264 may be provided by the same memory device or separate memory devices. In various examples, video data memory 235 may be on-chip with the other components of video encoder 20 , or off-chip relative to those components.

As shown in FIG17 , video encoder 20 receives video data and partitions the data into video blocks. This partitioning may also include partitioning into slices, tiles, or other larger units, as well as video block partitioning according to a quadtree structure of LCUs and CUs, for example. Video encoder 20 generally illustrates components that encode video blocks within a video slice to be encoded. A slice may be divided into multiple video blocks (and possibly into sets of video blocks called tiles).

Prediction processing unit 241 may select one of a plurality of possible coding modes for the current video block, e.g., one of a plurality of intra-coding modes or one of a plurality of inter-coding modes, based on the error results (e.g., coding rate and distortion level). Prediction processing unit 241 may provide the resulting intra-coded or inter-coded block to summer 251 to generate residual block data and to summer 262 to reconstruct the encoded block for use as a reference picture.

Intra-prediction processing unit 246 within prediction processing unit 241 performs intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression. Motion estimation unit 242 and motion and disparity compensation unit 244 within prediction processing unit 241 perform inter-predictive coding (including inter-view coding) of the current video block relative to one or more prediction blocks in one or more reference pictures (including inter-view reference pictures) to provide, for example, temporal and/or inter-view compression.

Motion estimation unit 242 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. Motion estimation unit 242 and motion and disparity compensation unit 244 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation performed by motion estimation unit 242 is the process of generating motion vectors, which estimate the motion of video blocks. For example, a motion vector may indicate the displacement of a PU of a video block within a current video frame or picture relative to a prediction block within a reference picture.

A prediction block is a block that is found to closely match the PU of the video block to be coded in terms of pixel difference, which may be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. In some examples, video encoder 20 may calculate values for sub-integer pixel positions of a reference picture stored in DPB 264. For example, video encoder 20 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Thus, motion estimation unit 242 may perform a motion search relative to full-pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

Motion estimation unit 242 calculates the motion vector and/or disparity motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a prediction block of a reference picture (including a temporal or inter-view reference picture). As described above, motion vectors can be used for motion compensated prediction, while disparity motion vectors can be used for disparity compensated prediction. The reference picture can be selected from a first reference picture list (List 0 or RefPicList0) or a second reference picture list (List 1 or RefPicList1), each of which identifies one or more reference pictures stored in DPB 264. Motion estimation unit 242 sends the calculated motion vector to entropy encoding unit 257 and motion and disparity compensation unit 244.

Motion and/or disparity compensation performed by motion and disparity compensation unit 244 may involve obtaining or generating a prediction block based on a motion vector determined by motion estimation (possibly performing interpolation with sub-pixel precision). Upon receiving the motion vector for the PU of the current video block, motion and disparity compensation unit 244 may locate the prediction block to which the motion vector points in one of the reference picture lists. Video encoder 20 forms a residual video block by subtracting the pixel values of the prediction block from the pixel values of the current video block being decoded, thereby forming pixel difference values. The pixel difference values form the residual data for the block and may include both luma and chroma difference components. Summer 251 represents the one or more components that perform this subtraction operation. Motion and disparity compensation unit 244 may also generate syntax elements associated with the video block and video slice for use by video decoder 30 when decoding the video block of the video slice.

Video encoder 20 (including ARP unit 245 and motion and disparity compensation unit 244) can perform any of bi-directional prediction and ARP techniques, such as the inter-view or temporal ARP techniques described herein. Specifically, in one example of the present disclosure, the video encoder can be configured to encode a current block of video data using bi-directional prediction and inter-view ARP. For the current block of video data, motion and disparity compensation unit 244 can be configured to determine temporal motion information for a first prediction direction (e.g., reference picture list X) for the current block of video data, and use the temporal motion information determined for the first prediction direction to identify a reference block for a second prediction direction (e.g., reference picture list Y), where the reference block for the second prediction direction is in a different access unit from the current block of video data. In this way, fewer memory accesses for motion information and reference blocks are required to encode the current block of video data.

As an alternative to the inter-prediction performed by motion estimation unit 242 and motion and disparity compensation unit 244 as described above, intra-prediction unit 246 can perform intra-prediction on the current block. Specifically, intra-prediction processing unit 246 can determine an intra-prediction mode to use to encode the current block. In some examples, intra-prediction processing unit 246 can encode the current video block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 246 (or prediction processing unit 241 in some examples) can select an appropriate intra-prediction mode to use from a test pattern.

For example, the intra-prediction processing unit 246 can use rate-distortion analysis to calculate rate-distortion values for various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics from the tested modes. Rate-distortion analysis generally determines the amount of distortion (or error) between an encoded block and the original unencoded block that was encoded to produce the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. The intra-prediction processing unit 246 can calculate a ratio based on the distortion and rate for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

In any case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 246 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 257. Entropy encoding unit 257 may encode the information indicating the selected intra-prediction mode according to the techniques of this disclosure. Video encoder 20 may include configuration data in the transmitted bitstream that may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also called codeword mapping tables), definitions of contexts for encoding various blocks, and indications of the most probable intra-prediction mode, intra-prediction mode index tables, and modified intra-prediction mode index tables to use for each of the contexts.

After prediction processing unit 241 generates a prediction block for the current video block via inter-frame prediction or intra-frame prediction, video encoder 20 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be contained in one or more TUs and applied to transform processing unit 252. Transform processing unit 252 transforms the residual video data into residual transform coefficients using, for example, a discrete cosine transform (DCT) or a conceptually similar transform. Transform processing unit 252 may convert the residual video data from the pixel domain to a transform domain, such as the frequency domain.

Transform processing unit 252 may send the resulting transform coefficients to quantization processing unit 255. Quantization processing unit 255 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the transform coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization processing unit 255 may then perform a scan of the matrix containing the quantized transform coefficients. Alternatively, entropy encoding unit 257 may perform the scan.

After quantization, entropy encoding unit 257 performs entropy encoding on the quantized transform coefficients. For example, entropy encoding unit 257 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method or technique. Following entropy encoding by entropy encoding unit 257, the encoded video bitstream may be transmitted to video decoder 30 or archived for later transmission or retrieval by video decoder 30. Entropy encoding unit 257 may also entropy encode the motion vectors and other syntax elements for the current video slice being coded.

Inverse quantization processing unit 259 and inverse transform processing unit 260 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, for later use as a reference block of a reference picture. Motion and disparity compensation unit 244 may calculate a reference block by adding the residual block to a prediction block of one of the reference pictures in one of the reference picture lists. Motion and disparity compensation unit 244 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for motion estimation. Summer 262 adds the reconstructed residual block to the motion compensated prediction block generated by motion and disparity compensation unit 244 to generate a reference block for storage in DPB 264. The reference block may be used by motion estimation unit 242 and motion and disparity compensation unit 244 as a reference block for inter-frame prediction of a block in a subsequent video frame or picture.

FIG18 is a block diagram illustrating an example video decoder 30 that may implement the techniques described in this disclosure. In the example of FIG18 , video decoder 30 includes video data memory 278, entropy decoding unit 280, prediction processing unit 281, inverse quantization processing unit 286, inverse transform processing unit 288, summer 291, and decoded picture buffer (DPB) 292. Prediction processing unit 281 includes motion and disparity compensation unit 282, ARP unit 283, and intra-prediction processing unit 284. In some examples, video decoder 30 may perform a decoding pass that is generally reciprocal to the encoding pass described with respect to video encoder 20 from FIG17 .

In various examples, one or more hardware units of video decoder 30 may be tasked to perform the techniques of this disclosure. For example, ARP unit 283 and motion and disparity compensation unit 282 may perform the techniques of this disclosure alone or in combination with other units of the video encoder.

Video data memory 278 may store video data, such as an encoded video bitstream, for decoding by components of video decoder 30. The video data stored in video data memory 278 may be obtained from a local video source, such as a camera, via wired or wireless network communication of video data, or by accessing physical data storage media. Video data memory 278 may form a coded picture buffer (CPB), which stores encoded video data from an encoded video bitstream. DPB 292 is an example of a DPB that stores reference video data used by video decoder 30 to decode video data (e.g., in intra- or inter-coding modes, also referred to as intra- or inter-prediction coding modes). Video data memory 278 and DPB 292 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 278 and DPB 292 may be provided by the same memory device or by separate memory devices. In various examples, video data memory 278 may be on-chip with other components of video decoder 30 or off-chip relative to those components.

During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video slice and associated syntax elements from video encoder 20. Entropy decoding unit 280 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. Entropy decoding unit 280 forwards the motion vectors and other syntax elements to prediction processing unit 281. Video decoder 30 may receive the syntax elements at the video slice level and/or the video block level.

When the video slice is coded as an intra-coded (I) slice, intra-prediction processing unit 284 of prediction processing unit 281 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When the video frame is coded as an inter-coded (i.e., B or P) slice or an inter-view coded slice, motion and disparity compensation unit 282 of prediction processing unit 281 generates prediction blocks for the video block of the current video slice based on motion vectors, disparity motion vectors, and other syntax elements received from entropy decoding unit 280. The prediction block may be generated from one of the reference pictures within one of the reference picture lists that includes inter-view reference pictures. Video decoder 30 may construct reference frame lists RefPicList0 and RefPicList1 based on the reference pictures stored in DPB 292 using a default construction technique or any other technique.

Motion and disparity compensation unit 282 determines prediction information for a video block of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate a prediction block for the current video block being decoded. For example, motion and disparity compensation unit 282 uses some of the received syntax elements to determine a prediction mode (e.g., intra-prediction or inter-prediction) for coding the video block of the video slice, an inter-prediction slice type (e.g., a B slice, a P slice, and/or an inter-view predicted slice), construction information for one or more of the slice's reference picture lists, a motion vector and/or a disparity motion vector for each inter-coded video block of the slice, an inter-prediction state for each inter-coded video block of the slice, and other information used to decode the video blocks in the current video slice.

Motion and disparity compensation unit 282 may also perform interpolation based on interpolation filters. Motion and disparity compensation unit 282 may calculate interpolated values for sub-integer pixels of a reference block using interpolation filters used by video encoder 20 during encoding of the video block. In this case, motion and disparity compensation unit 282 may determine the interpolation filters used by video encoder 20 from received syntax information elements and use the interpolation filters to generate a prediction block.

Inverse quantization processing unit 286 inverse quantizes, or dequantizes, the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 280. The inverse quantization process may include using quantization parameters calculated by video encoder 20 for each video block in a video slice to determine the degree of quantization and, likewise, the degree of inverse quantization that should be applied. Inverse transform processing unit 288 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients to produce a residual block in the pixel domain.

Video decoder 30 (including ARP unit 283 and motion and disparity compensation unit 282) may perform any of the bi-directional prediction and/or ARP techniques, such as the inter-view or temporal ARP techniques described herein. Specifically, in one example of the present disclosure, video decoder 30 may receive a current block of video data encoded using bi-directional prediction and inter-view ARP. The encoded block of video data may be stored in video data memory 278. For the encoded block of video data, motion and disparity compensation unit 282 may be configured to determine temporal motion information for a first prediction direction (e.g., reference picture list X) of the encoded block of video data and use the temporal motion information determined for the first prediction direction to identify a reference block for a second prediction direction (e.g., reference picture list Y), where the reference block for the second prediction direction is in a different access unit from the current block of video data. In this manner, fewer memory accesses for motion information and reference blocks are required to decode the encoded video block.

After motion and disparity compensation unit 282 generates a prediction block for the current video block based on the motion vector and other syntax elements, video decoder 30 forms a decoded video block by summing the residual block from inverse transform processing unit 288 with the corresponding prediction block generated by motion and disparity compensation unit 282. Summer 291 represents one or more components that can perform this summation operation. Optionally, a deblocking filter may also be applied to the decoded block to remove blocking artifacts. Other loop filters (in the decoding loop or after the decoding loop) may also be used to smooth pixel transitions or otherwise improve video quality. The decoded video blocks in a given frame or picture are then stored in DPB 292, which stores reference pictures for subsequent motion compensation. DPB 292 also stores the decoded video for later presentation on a display device (e.g., display device 32 of FIG. 1).

19 is a flowchart illustrating an example ARP method for encoding a video block according to the techniques described in this disclosure. The techniques of FIG. 19 may be performed by any combination of hardware structures of video encoder 20, including motion and disparity compensation unit 244 and ARP unit 245.

In one example of the present disclosure, video encoder 20 may be configured to encode a block of video data using ARP and bi-directional prediction. In this example, bi-directional prediction includes temporal prediction for a first prediction direction (e.g., for reference picture list X) and inter-view prediction for a second prediction direction (e.g., for reference picture list Y). Motion and disparity compensation unit 244 may be configured to determine temporal motion information for a first prediction direction of the block of video data (1900). ARP unit 245 may be configured to use the determined temporal motion information to identify a reference block for the first prediction direction (1910) and to use the determined temporal motion information for the first prediction direction to identify a reference block for a second prediction direction different from the first prediction direction (1920). The reference block may be in an access unit different from the access unit of the block of video data. ARP unit 245 may further be configured to perform advanced residual prediction (1930) using the identified reference blocks for the first and second prediction directions.

In other examples of the present disclosure, the motion and disparity compensation unit 244 may be configured to determine disparity motion information for a second prediction direction for the first block of encoded video data. Additionally, the ARP unit 245 may be configured to use the determined temporal motion information to identify a first reference block for the first prediction direction, wherein the first reference block is in a second access unit of the first view. The ARP unit 245 may further be configured to use the determined temporal motion information to identify a second reference block for the second prediction direction, and to use the determined temporal motion information and the determined disparity motion information to identify a third reference block for the second prediction direction, wherein the third reference block is in a third access unit of the second view.

20 is a flowchart illustrating an example ARP method for decoding a video block according to the techniques described in this disclosure. The techniques of FIG. 20 may be performed by any combination of hardware structures of the video decoder, ARP unit 283, and motion and disparity compensation unit 282.

In one example of the present disclosure, video decoder 30 may be configured to store a first block of encoded video data in a first access unit of a first view, wherein the first block of encoded video data is encoded using advanced residual prediction and bi-directional prediction (2000). The bi-directional prediction may include temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction.

Motion and disparity compensation unit 282 may be configured to determine temporal motion information for a first prediction direction for a first block of encoded video data (2010). ARP unit 283 may be configured to determine disparity motion information for a second prediction direction for the first block of encoded video data (2020), and use the determined temporal motion information for the first prediction direction to identify a reference block for a second prediction direction different from the first prediction direction (2030). The reference block may be in an access unit different from the first access unit. ARP unit 283 may further be configured to perform advanced residual prediction on the first block of encoded video data using the identified reference block for the second prediction direction (2040).

In another example of the present disclosure, the ARP unit 238 may be configured to use the determined temporal motion information to identify a reference block for a first prediction direction, and perform advanced residual prediction on the first block of encoded video data using the identified reference block for the first prediction direction. The ARP unit 283 may be further configured to use the determined temporal motion information to identify a second reference block for a second prediction direction, and use the determined temporal motion information and the determined disparity motion information to identify a third reference block for the second prediction direction, wherein the third reference block is in a third access unit of the second view. The first reference block for the first prediction direction is the same as the second reference block for the second prediction direction.

In another example of the disclosure, video decoder 30 may be configured to decode a first encoded block of video data using the identified reference block for the first prediction direction and the identified reference block for the second prediction direction.

In another example of the present invention, the video decoder 30 may be configured to: decode the first encoded video data block using one of block-level advanced residual prediction or prediction unit-level advanced residual prediction to generate residual video data; and decode the residual data using bi-directional prediction, the identified reference block for the first prediction direction, and the identified reference block for the second prediction direction to generate a decoded video data block.

In another example of the present disclosure, video decoder 30 may be further configured to store a second block of encoded video data in a fourth access unit of a third view, wherein the second block of encoded video data is encoded using advanced residual prediction and bi-directional prediction, which may include temporal prediction for a third prediction direction and inter-view prediction for a fourth prediction direction.

The motion and disparity compensation unit 282 may be configured to determine temporal motion information for a first prediction direction for a first block of encoded video data. The ARP unit 283 may be configured to use the determined temporal motion information to identify a reference block for the first prediction direction. The ARP unit 283 may further use the determined temporal motion information for the first prediction direction to identify a reference block for a second prediction direction different from the first prediction direction, wherein the reference block is in an access unit different from the first access unit. The ARP unit 283 may also perform advanced residual prediction on the first block of encoded video data using the identified reference blocks for the first and second prediction directions.

In another example of the present disclosure, the motion and disparity compensation unit 282 may be configured to determine disparity motion information for a second prediction direction for a first block of encoded video data. The ARP unit 283 may be configured to use the determined temporal motion information to identify a first reference block for the first prediction direction, wherein the first reference block is in a second access unit of the first view. The ARP unit 283 may further be configured to use the determined temporal motion information and the determined disparity motion information to identify a second reference block for a second prediction direction, wherein the second reference block is in a third access unit of the second view.

In another example of the present disclosure, video decoder 30 may be configured to decode a first block of encoded video data using the identified reference block for the first prediction direction and the identified reference block for the second prediction direction. Video decoder 30 may be further configured to decode the first block of encoded video data using one of block-level advanced residual prediction or prediction unit-level advanced residual prediction to generate residual video data, and to decode the residual data using bi-directional prediction, the identified reference block for the first prediction direction, and the identified reference block for the second prediction direction to generate a decoded block of video data.

In some examples, one or more aspects of the techniques described in this disclosure may be performed by an intermediate network device, such as a media-aware network element (MANE), a stream adaptation processor, a splicing processor, or an editing processor. For example, such an intermediate device may be configured to generate or receive any of the various signaling methods described in this disclosure.

In one or more instances, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media, or communication media that includes any media that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and/or data structures for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

By way of example, and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Likewise, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). The various components, modules, or units described herein are intended to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation via distinct hardware units. In practice, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and/or firmware, or provided via a collection of interoperating hardware units, including one or more processors as described above. Various examples have been described. These and other examples are within the scope of the appended claims.

Claims (14)

1. A method for decoding video data, the method comprising: The system receives a first encoded video data block in a first access unit of a first view, wherein the first encoded video data block is encoded using advanced residual prediction and bidirectional prediction, wherein the bidirectional prediction includes temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction, and wherein the advanced residual prediction includes a residual predictor block that determines the residual block of the first encoded video data block. First temporal motion information for determining the first predicted direction of the first encoded video data block; The first temporal motion information of the first predicted direction is used to identify a first temporal reference block in the first predicted direction, wherein the first temporal reference block is in the first view and in a second access unit, and the second access unit is different from the first access unit. Derive the disparity vector of the first predicted direction of the first encoded video data block; The disparity vector of the first prediction direction is used to identify a first view reference block, wherein the first view reference block is in a second view, and the second view is different from the first view; The first temporal motion information of the first inter-view reference block and the first predicted direction is used to identify the second inter-view reference block, which is in the second view; The first residual predictor block is determined from the first inter-view reference block and the second inter-view reference block; First disparity motion information for the second prediction direction of the first encoded video data block; The first temporal motion information of the first prediction direction is used to identify the second temporal reference block of the second prediction direction, wherein the second temporal reference block is the same as the first temporal reference block; The first disparity motion information in the second prediction direction is used to identify a third-view inter-reference block, wherein the third-view inter-reference block is in the second view; The fourth inter-view reference block is identified using the third inter-view reference block and the first temporal motion information of the first predicted direction, wherein the second inter-view reference block is in the second view; Determine the second residual predictor block from the reference block between the third and fourth views; and Advanced residual prediction is performed on the first encoded video data block using the first time reference block, the second time reference block, the first residual predictor block, and the second residual predictor block. 2. The method according to claim 1, further comprising: Decode the first encoded video data block using either block-level high-level residual prediction or prediction unit-level high-level residual prediction to generate residual video data; and The residual data is bidirectionally predicted and decoded using a first time reference block with the first prediction direction and a second time reference block with the second prediction direction to produce a decoded video data block. 3. The method according to claim 1, further comprising: The second encoded video data block is received in the third access unit of the third view, wherein the second encoded video data block is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including inter-view prediction for the third prediction direction and temporal prediction for the fourth prediction direction. Determine the second disparity motion information in the third prediction direction of the second encoded video data block; The second disparity motion information in the third prediction direction is used to identify the fifth inter-view reference block in the third prediction direction, wherein the fifth inter-view reference block is in the fourth view, and the fourth view is different from the third view; The motion vector of the third predicted direction of the second encoded video data block is derived from the fifth view inter-reference block; The motion vector in the third predicted direction is used to identify the third time reference block; The sixth view reference block is identified using the motion vector of the fifth view reference block and the third prediction direction; The third residual predictor block is determined from the fifth view reference block and the sixth view reference block; Determine the second temporal motion information of the fourth prediction direction of the second encoded video data block; The second temporal motion information of the fourth prediction direction is used to identify the fourth temporal reference block of the fourth prediction direction; The second disparity motion information in the third prediction direction is used to identify a seventh inter-view reference block, wherein the seventh inter-view reference block is the same as the fifth inter-view reference block; The eighth view reference block is identified using the second temporal motion information of the seventh view reference block and the fourth prediction direction; The fourth residual predictor block is determined from the seventh inter-view reference block and the eighth inter-view reference block; and Advanced residual prediction is performed on the second coded video data block using the third time reference block, the fourth time reference block, the third residual predictor block, and the fourth residual predictor block. 4. The method of claim 1, further comprising: The first time motion information is scaled based on the image sequence count value. 5. An apparatus configured to decode video data, the apparatus comprising: A video data storage device configured to store a first encoded video data block in a first access unit of a first view, wherein the first encoded video data block is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction, and wherein the advanced residual prediction includes a residual predictor block that determines the residual blocks of the first encoded video data block; and One or more processors, which communicate with the video data storage and are configured to: First temporal motion information for determining the first predicted direction of the first encoded video data block; The first temporal motion information of the first predicted direction is used to identify a first temporal reference block in the first predicted direction, wherein the first temporal reference block is in the first view and in a second access unit, and the second access unit is different from the first access unit. Derive the disparity vector of the first predicted direction of the first encoded video data block; The disparity vector of the first prediction direction is used to identify a first view reference block, wherein the first view reference block is in a second view, and the second view is different from the first view; The first temporal motion information of the first inter-view reference block and the first predicted direction is used to identify the second inter-view reference block, which is in the second view; The first residual predictor block is determined from the first inter-view reference block and the second inter-view reference block; First disparity motion information for the second prediction direction of the first encoded video data block; The first temporal motion information of the first prediction direction is used to identify the second temporal reference block of the second prediction direction, wherein the second temporal reference block is the same as the first temporal reference block; The first disparity motion information in the second prediction direction is used to identify a third-view inter-reference block, wherein the third-view inter-reference block is in the second view; The fourth inter-view reference block is identified using the third inter-view reference block and the first temporal motion information of the first predicted direction, wherein the second inter-view reference block is in the second view; The second residual predictor block is determined from the reference block between the third and fourth views; and Advanced residual prediction is performed on the first encoded video data block using the first time reference block, the second time reference block, the first residual predictor block, and the second residual predictor block. 6. The device of claim 5, wherein the one or more processors are further configured to: Decode the first encoded video data block using either block-level high-level residual prediction or prediction unit-level high-level residual prediction to generate residual video data; and The residual data is decoded using bidirectional prediction using a first time reference block with the first prediction direction and a second time reference block with the second prediction direction to produce a decoded video data block. 7. The device according to claim 6, further comprising: A display configured to show the decoded video data blocks. 8. The device of claim 5, wherein the video data storage and the one or more processors are housed in one of the following: a desktop computer, a laptop computer, a set-top box, a handset, a smartphone, a smart pad, a tablet computer, a television, a camera, a digital media player, a video game console, or a video streaming device. 9. The device according to claim 5, The one or more processors are further configured to: The second encoded video data block is received in the third access unit of the third view, wherein the second encoded video data block is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including inter-view prediction for the third prediction direction and temporal prediction for the fourth prediction direction. Determine the second disparity motion information in the third prediction direction of the second encoded video data block; The second disparity motion information in the third prediction direction is used to identify the fifth inter-view reference block in the third prediction direction, wherein the fifth inter-view reference block is in the fourth view, which is different from the third view; The motion vector of the third predicted direction of the second encoded video data block is derived from the fifth view inter-reference block; The motion vector in the third predicted direction is used to identify the third time reference block; The sixth view reference block is identified using the motion vector of the fifth view reference block and the third prediction direction; The third residual predictor block is determined from the fifth view reference block and the sixth view reference block; Determine the second temporal motion information of the fourth prediction direction of the second encoded video data block; The second temporal motion information of the fourth prediction direction is used to identify the fourth temporal reference block of the fourth prediction direction; The second disparity motion information in the third prediction direction is used to identify a seventh inter-view reference block, wherein the seventh inter-view reference block is the same as the fifth inter-view reference block; The eighth view reference block is identified using the second temporal motion information of the seventh view reference block and the fourth prediction direction; The fourth residual predictor block is determined from the seventh view reference block and the eighth view reference block; and Advanced residual prediction is performed on the second coded video data block using the third time reference block, the fourth time reference block, the third residual predictor block, and the fourth residual predictor block. 10. The device of claim 5, wherein the one or more processors are further configured to: The first time motion information is scaled based on the image sequence count value. 11. An apparatus configured to decode video data, the apparatus comprising: A means for receiving a first encoded video data block in a first access unit of a first view, wherein the first encoded video data block is encoded using advanced residual prediction and bidirectional prediction, the bidirectional prediction including time prediction for a first prediction direction and inter-view prediction for a second prediction direction, and wherein the advanced residual prediction includes a residual predictor block for determining the residual block of the first encoded video data block. A means for determining first temporal motion information of the first predicted direction of the first encoded video data block; A means for identifying a first time reference block in the first predicted direction using first time motion information in the first predicted direction, wherein the first time reference block is in the first view and in a second access unit, the second access unit being different from the first access unit; A means for deriving the disparity vector of the first predicted direction of the first encoded video data block; A means for identifying a first-view reference block using the disparity vector of the first predicted direction, wherein the first-view reference block is in a second view, the second view being different from the first view; A means for identifying a second inter-view reference block using the first temporal motion information of the first inter-view reference block and the first predicted direction, wherein the second inter-view reference block is in the second view; A means for determining a first residual predictor block from the first inter-view reference block and the second inter-view reference block; A means for determining first disparity motion information for the second prediction direction of the first coded video data block; A means for identifying a second time reference block in a second prediction direction using first time motion information in the first prediction direction, wherein the second time reference block is the same as the first time reference block; A means for identifying a third-view inter-reference block using first disparity motion information in the second predicted direction, wherein the third-view inter-reference block is in the second view; A means for identifying a fourth inter-view reference block using the third inter-view reference block and the first temporal motion information of the first predicted direction, wherein the second inter-view reference block is in the second view; A means for determining a second residual predictor block from the third view reference block and the fourth view reference block; and A means for performing advanced residual prediction on a first encoded video data block using the first time reference block, the second time reference block, the first residual predictor block, and the second residual predictor block. 12. The apparatus of claim 11, wherein the means for decoding the first encoded video data block comprises: A means for decoding the first encoded video data block using either block-level high-level residual prediction or prediction unit-level high-level residual prediction to generate residual video data; and An apparatus for bidirectional predictive decoding of the residual data using a first time reference block having a first prediction direction and a second time reference block having a second prediction direction to generate a decoded video data block. 13. A computer-readable storage medium storing instructions, which, when executed, cause one or more processors of an apparatus configured to decode video data: Receive a first encoded video data block in a first access unit of a first view, wherein the first encoded video data block is encoded using advanced residual prediction and bidirectional prediction, wherein the bidirectional prediction includes temporal prediction for a first prediction direction and inter-view prediction for a second prediction direction, and wherein the advanced residual prediction includes a residual predictor block that determines the residual block of the first encoded video data block. First temporal motion information for determining the first predicted direction of the first encoded video data block; The first temporal motion information of the first predicted direction is used to identify a first temporal reference block in the first predicted direction, wherein the first temporal reference block is in the first view and in a second access unit, and the second access unit is different from the first access unit. Derive the disparity vector of the first predicted direction of the first encoded video data block; The disparity vector of the first prediction direction is used to identify a first view reference block, wherein the first view reference block is in a second view, and the second view is different from the first view; The first temporal motion information of the first inter-view reference block and the first predicted direction is used to identify the second inter-view reference block, which is in the second view; The first residual predictor block is determined from the first inter-view reference block and the second inter-view reference block; First disparity motion information for the second prediction direction of the first encoded video data block; The first temporal motion information of the first prediction direction is used to identify the second temporal reference block of the second prediction direction, wherein the second temporal reference block is the same as the first temporal reference block; The first disparity motion information in the second prediction direction is used to identify a third-view inter-reference block, wherein the third-view inter-reference block is in the second view; The fourth inter-view reference block is identified using the third inter-view reference block and the first temporal motion information of the first predicted direction, wherein the second inter-view reference block is in the second view; The second residual predictor block is determined from the reference block between the third and fourth views; and Advanced residual prediction is performed on the first encoded video data block using the first time reference block, the second time reference block, the first residual predictor block, and the second residual predictor block. 14. The computer-readable storage medium of claim 13, wherein the instructions further cause the one or more processors to: Decode the first encoded video data block using either block-level high-level residual prediction or prediction unit-level high-level residual prediction to generate residual video data; and The residual data is decoded using bidirectional prediction using a first time reference block with the first prediction direction and a second time reference block with the second prediction direction to produce a decoded video data block.