HK40000797B - Intra video coding using a decoupled tree structure - Google Patents

Description

Video internal decoding using a decoupled tree structure

This application claims the benefits of U.S. Provisional Application No. 62/375,383, filed August 15, 2016, and U.S. Provisional Application No. 62/404,572, filed October 5, 2016, each of which is hereby incorporated herein by reference in its entirety.

Technical Field

This invention relates to video decoding.

Background Technology

Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones, so-called "smartphones," video conferencing devices, video streaming devices, and so on. Digital video devices implement video decoding technologies, such as those described in standards defined by various video decoding standards. Video decoding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including their Scalable Video Decoding (SVC) and Multi-View Video Decoding (MVC) extensions.

In addition, a new video decoding standard (i.e., High Efficiency Video Decoding (HEVC)) has recently been developed by the Joint Collaborative Video Decoding Group (JCT-VC) of the ITU-T Video Decoding Experts Group (VCEG) and the ISO/IEC Animation Experts Group (MPEG). The latest HEVC draft specification, referred to below as "HEVC WD," is available at http://phenix.int-evry.fr/jct/doc_end_user/documents/14_Vienna/wg11/JCTVC-N1003-v1.zip. The HEVC specification and its extensions (Format Range (RExt), Scalability (SHVC), Multi-View (MV-HEVC) extensions, and Screen Content Extensions) are available at http://phenix.int-evry.fr/jct/doc_end_user/current_document.php?id=10481. ITU-TVCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) are now studying the potential need to standardize future video decoding technologies with compression capabilities that significantly exceed those of the current HEVC standard (including its current extensions and recent extensions for screen content decoding and high dynamic range decoding).

The expert group is working together on this exploration activity within a joint collaborative effort (known as the Joint Video Exploration Team (JVET)) to evaluate compression technology designs recommended by the group's experts in this field. JVET held its first meeting between October 19 and 21, 2015. The latest version of the reference software (i.e., Joint Exploration Model 3 (JEM 3)) is available for download at https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/tags/HM-16.6-JEM-3.0/. The algorithm description for JEM3 is further described in "Algorithm description of Joint Exploration Test Model 3" by J. Chen, E. Alshina, G.J. Sullivan, J.-R. Ohm, and J. Boyce (JVET-C1001, Geneva, January 2016).

Video devices can more efficiently transmit, receive, encode, decode, and/or store digital video information by implementing such video decoding techniques. Video decoding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove inherent redundancy in video sequences. For block-based video decoding, video slices (e.g., video frames or portions of video frames) can be divided into video blocks, which, in some techniques, may also be referred to as tree blocks, decoding units (CUs), and/or decoding nodes. Video blocks in an intra-frame decoded (I) slice of a picture are encoded using spatial prediction relative to reference samples in adjacent blocks within the same picture. Video blocks in an inter-frame decoded (P or B) slice of a picture can use spatial prediction relative to reference samples in adjacent blocks within the same picture or temporal prediction relative to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame.

Spatial or temporal prediction generates predictive blocks for the block to be decoded. Residual data represents the pixel difference between the original block and the predictive block. Inter-frame decoded blocks are encoded based on the motion vector of the block pointing to the reference sample forming the predictive block and the residual data indicating the difference between the decoded block and the predictive block. Intra-frame decoded blocks are encoded based on the intra-frame decoding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain to generate residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged as a two-dimensional array, can be scanned to generate a one-dimensional vector of transform coefficients, and entropy decoding can be applied to achieve even more compression.

Summary of the Invention

Generally, this invention describes techniques related to decoding (e.g., decoding or encoding) video data using intra-frame prediction (in some cases, based on a tree structure providing different splitting information for the luma and chroma components). That is, the luma segmentation tree structure can be decoupled from the corresponding chroma segmentation tree structure, depending on various segmentation schemes compatible with the described techniques. The described techniques can be used in the context of advanced video codecs, such as extensions to HEVC or next-generation video decoding standards.

In one example, an apparatus for decoding video data includes a memory and processing circuitry in communication with the memory. The memory of the apparatus is configured to store video data. The processing circuitry is configured to determine a plurality of derived modes (DMs) that can be used to predict luminance blocks of the video data stored in the memory, and also to predict chroma blocks of the video data stored in the memory, the chroma blocks corresponding to the luminance blocks. The processing circuitry is further configured to form a candidate list of prediction modes for the chroma blocks, the candidate list containing one or more DMs that can be used to predict the chroma blocks. The processing circuitry is further configured to determine to decode the chroma blocks using any of the one or more DMs in the candidate list, and to decode an indication identifying a selected DM from the candidate list that will be used to decode the chroma blocks based on the determination to decode the chroma blocks using any of the one or more DMs in the candidate list. The processing circuitry is further configured to decode the chroma blocks according to the selected DM in the candidate list.

In another example, a method for decoding video data includes determining a plurality of derived modes (DMs) that can be used to predict luminance blocks of the video data and also to predict chroma blocks of the video data, the chroma blocks corresponding to the luminance blocks. The method further includes: forming a candidate list of prediction modes for the chroma blocks, the candidate list containing one or more DMs that can be used to predict the chroma blocks; and determining to use any of the one or more DMs in the candidate list to decode the chroma blocks. The method further includes: based on the determination to use any of the one or more DMs in the candidate list to decode the chroma blocks, decoding an indication of a selected DM in the candidate list that will be used to decode the chroma blocks; and decoding the chroma blocks according to the selected DM in the candidate list.

In another example, an apparatus includes means for determining a plurality of derived modes (DMs) that can be used to predict luminance blocks of video data, and means for predicting chroma blocks of the video data corresponding to the luminance blocks. The method further includes: forming a candidate list of prediction modes for the chroma blocks, the candidate list containing one or more DMs that can be used to predict the chroma blocks; and determining to decode the chroma blocks using any of the one or more DMs in the candidate list. The apparatus further includes: means for forming a candidate list of prediction modes for the chroma blocks, the candidate list containing one or more DMs that can be used to predict the chroma blocks; and means for determining to decode the chroma blocks using any of the one or more DMs in the candidate list. The device further includes: means for decoding an indication of a selected DM in the candidate list to be used for decoding the chroma block based on the determination of decoding the chroma block using any DM in the candidate list; and means for decoding the chroma block according to the selected DM in the candidate list.

In another example, a non-transitory computer-readable storage medium is encoded with instructions that, upon execution, cause a processor of a computing device to determine that a plurality of derived modes (DMs) for predicting luminance blocks of video data can also be used to predict chroma blocks of the video data, the chroma blocks corresponding to the luminance blocks. Upon execution, the instructions further cause the processor to: form a candidate list of prediction modes for the chroma blocks, the candidate list containing one or more of the plurality of DMs for predicting the chroma blocks; and determine to use any of the one or more DMs in the candidate list to decode the chroma blocks. Upon execution, the instructions further cause the processor to: form a candidate list of prediction modes for the chroma blocks, the candidate list containing one or more of the plurality of DMs for predicting the chroma blocks; and determine to use any of the one or more DMs in the candidate list to decode the chroma blocks. When executed, the instructions further cause the processor to perform the following operations: to decode an indication of a selected DM in the candidate list to be used for decoding the chroma block based on the determination to decode the chroma block using any DM in the candidate list; and to decode the chroma block according to the selected DM in the candidate list.

In another example, an apparatus for decoding video data includes a memory and processing circuitry in communication with the memory. The memory of the apparatus is configured to store video data. The processing circuitry is configured to: form a most probable mode (MPM) candidate list for chroma blocks of the video data stored in the memory, such that the MPM candidate list includes one or more derived modes (DMs) associated with luminance blocks of the video data associated with the chroma blocks, and a plurality of luminance prediction modes that can be used to decode the luminance components of the video data. The processing circuitry is further configured to: select a mode from the MPM candidate list; and decode the chroma blocks according to the mode selected from the MPM candidate list.

In another example, a method for decoding video data includes: forming a most probable mode (MPM) candidate list for chroma blocks of the video data, such that the MPM candidate list includes one or more derived modes (DMs) associated with luminance blocks of the video data associated with the chroma blocks, and a plurality of luminance prediction modes that can be used to decode the luminance components of the video data. The method further includes: selecting a mode from the MPM candidate list; and decoding the chroma blocks according to the mode selected from the MPM candidate list.

In another example, an apparatus includes means for performing the following operations: forming a most probable mode (MPM) candidate list for chroma blocks of the video data, such that the MPM candidate list includes one or more derived modes (DMs) associated with luminance blocks of the video data associated with the chroma blocks, and a plurality of luminance prediction modes for decoding the luminance components of the video data. The apparatus further includes: means for selecting a mode from the MPM candidate list; and means for decoding the chroma blocks according to the modes selected from the MPM candidate list.

In another example, a non-transitory computer-readable storage medium is encoded with instructions that, when executed, cause a processor of a computing device to: form a most probable mode (MPM) candidate list for chroma blocks of video data stored in the memory, such that the MPM candidate list includes one or more derived modes (DMs) associated with luminance blocks of the video data associated with the chroma blocks, and a plurality of luminance prediction modes that can be used to decode the luminance components of the video data. When executed, the instructions cause the processor of the computing device to: select a mode from the MPM candidate list; and decode the chroma blocks according to the mode selected from the MPM candidate list.

Details of one or more examples are set forth in the following figures and detailed descriptions. Other features, objectives, and advantages will be apparent from the detailed descriptions, the figures, and the claims.

Attached Figure Description

Figure 1 is a block diagram illustrating an example video encoding and decoding system that can be configured to perform the technology of the present invention.

Figure 2 is a block diagram illustrating an example of a video encoder that can be configured to perform the technology of the present invention.

Figure 3 is a block diagram illustrating an example of a video decoder that can be configured to perform the technology of the present invention.

Figure 4 is a conceptual diagram illustrating aspects of intra-frame prediction.

Figure 5 is a conceptual diagram illustrating the intra-prediction mode used for luma blocks.

Figure 6 is a conceptual diagram illustrating aspects of the planar pattern.

Figure 7 is a conceptual diagram illustrating aspects of the HEVC perspective model.

Figure 8 is a conceptual diagram illustrating examples of nominal vertical and horizontal brightness and chromaticity samples in an image.

Figure 9 is a conceptual diagram illustrating the location of samples used to derive the parameters used in predictions based on the linear model (LM) pattern.

Figure 10 is a conceptual diagram illustrating the structure of a quadtree binary tree (QTBT).

Figures 11A and 11B illustrate examples of independent segmentation structures for corresponding luma and chroma blocks according to the QTBT segmentation scheme.

Figures 12A and 12B illustrate adjacent block selection for adaptive sorting of chromaticity prediction modes according to one or more aspects of the present invention.

Figures 13A and 13B are conceptual diagrams illustrating an example of a video encoding and decoding device that can be used to select the block position of a chroma intra-frame prediction mode according to the technique based on multiple DM mode selection described above.

Figure 14 is a flowchart illustrating an example process that can be executed by the processing circuit of a video decoding apparatus according to an aspect of the present invention.

Figure 15 is a flowchart illustrating an example process that can be executed by the processing circuit of a video encoding apparatus according to an aspect of the present invention.

Figure 16 is a flowchart illustrating an example process executable by the processing circuitry of a video decoding apparatus according to an aspect of the present invention.

Figure 17 is a flowchart illustrating an example process that can be executed by the processing circuit of a video encoding apparatus according to an aspect of the present invention.

Detailed Implementation

Figure 1 is a block diagram illustrating an example video encoding and decoding system 10 that can be configured to perform motion vector prediction according to the technology of the present invention. As shown in Figure 1, system 10 includes a source device 12 that provides encoded video data that will later be decoded by a destination device 14. Specifically, the source device 12 provides the video data to the destination device 14 via a computer-readable medium 16. The source device 12 and the destination device 14 may include any of a wide range of devices, including desktop computers, laptop computers, tablet computers, set-top boxes, mobile phones (e.g., so-called "smartphones"), so-called "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and the like. In some cases, the source device 12 and the destination device 14 may be equipped for wireless communication.

Destination device 14 may receive encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 may include any type of media or device capable of moving encoded video data from source device 12 to destination device 14. In one example, computer-readable medium 16 may include communication media enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard (e.g., a wireless communication protocol) and transmitted to destination device 14. Communication media may include any wireless or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. Communication media may form part of a packet-based network (e.g., a local area network, wide area network, or global network, such as the Internet). Communication media may include routers, switches, base stations, or any other devices suitable for facilitating communication from source device 12 to destination device 14.

In some instances, encoded data can be output from output interface 22 to a storage device. Similarly, encoded data can be accessed from a storage device via an input interface. The storage device can comprise any of a variety of distributed or locally accessible data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In yet another instance, the storage device can correspond to a file server or another intermediate storage device capable of storing encoded video generated by source device 12. Destination device 14 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing and transmitting the encoded video data to destination device 14. Example file servers include web servers (e.g., for websites), FTP servers, network-attached storage (NAS) devices, or local disk drives. Destination device 14 can access the encoded video data via any standard data connection, including an Internet connection. This connection may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. Transmission of encoded video data from the storage device may be streaming transmission, download transmission, or a combination thereof.

The technology of this invention is not necessarily limited to wireless applications or settings. The technology can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (e.g., Dynamic Adaptive Streaming (DASH) via HTTP), digital video encoded to data storage media, decoding digital video stored on data storage media, or other applications. In some instances, system 10 can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

In the example of Figure 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to the invention, the video encoder 20 of source device 12 can be configured to apply the techniques of the invention regarding motion vector prediction. In other examples, the source and destination devices may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18 (e.g., an external camera). Similarly, destination device 14 may interface with an external display device, rather than including an integrated display device.

The system 10 illustrated in Figure 1 is only one example. The techniques of the present invention regarding motion vector prediction can be performed by any digital video encoding and/or decoding device. Although the techniques of the present invention are generally performed by video encoding devices, they can also be performed by video encoders/decoders (commonly referred to as "CODECs"). Furthermore, the techniques of the present invention can also be performed by video preprocessors. Source device 12 and destination device 14 are merely examples of decoding devices that generate decoded video data from source device 12 for transmission to destination device 14. In some instances, devices 12 and 14 can operate in a substantially symmetrical manner, such that each of devices 12 and 14 includes video encoding and decoding components. Thus, system 10 can support, for example, one-way or two-way video transmission between video devices 12 and 14 for video streaming, video playback, video broadcasting, or video telephony.

The video source 18 of source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, and/or a video feed interface for receiving video from a video content provider. Alternatively, video source 18 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, then source device 12 and destination device 14 may form a so-called camera phone or video phone. However, as mentioned above, the techniques described in this invention are generally applicable to video decoding and to wireless and/or wired applications. In each case, captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output from output interface 22 to computer-readable media 16.

Computer-readable media 16 may comprise transient media, such as wireless broadcasting or wired network transmissions, or storage media (i.e., non-transient storage media), such as hard disks, flash drives, compact optical discs, digital video optical discs, Blu-ray discs, or other computer-readable media. In some instances, a network server (not shown) may receive encoded video data from source device 12 and, for example, provide the encoded video data to destination device 14 via network transmission. Similarly, a computing device of a media production facility (e.g., an optical disc stamping facility) may receive encoded video data from source device 12 and produce an optical disc containing the encoded video data. Therefore, in various instances, computer-readable media 16 may be understood to comprise one or more computer-readable media of various forms.

The input interface 28 of the destination device 14 receives information from the computer-readable medium 16. The information on the computer-readable medium 16 may include grammatical information defined by the video encoder 20, which is also used by the video decoder 30, including description blocks and other characteristics and/or processed grammatical elements of the decoding unit (e.g., GOP). The display device 32 displays the decoded video data to the user and may include any of a variety of display devices, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device.

The video encoder 20 and video decoder 30 may operate according to a video decoding standard, such as the High Efficiency Video Decoding (HEVC) standard, an extension or successor to the HEVC standard, such as ITU-T H.266. Alternatively, the video encoder 20 and video decoder 30 may operate according to other proprietary or industry standards, such as the ITU-T H.264 standard, alternatively referred to as MPEG-4, Part 10, Advanced Video Decoding (AVC) or extensions of these standards. However, the technology of the present invention is not limited to any particular decoding standard. Other examples of video decoding standards include MPEG-2 and ITU-T H.263. Although not shown in Figure 1, in some aspects, the video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or separate data streams. Where applicable, the MUX-DEMUX unit may comply with the ITU H.223 multiplexer protocol or other protocols such as User Datagram Protocol (UDP).

The video encoder 20 and video decoder 30 can each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, solid-state, or any combination thereof. When the technology is implemented in part in software, the device may store instructions for software in a suitable non-transitory computer-readable medium, and execute the instructions in hardware using one or more processors to perform the technology of the invention. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, and any of the one or more encoders or decoders may be integrated as part of a combined encoder/decoder (CODEC) in the respective device.

Video decoding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), which include their Scalable Video Decoding (SVC) and Multi-View Video Decoding (MVC) extensions. A joint draft of MVC is described in "Advanced Video Decoding for General Audiovisual Services" (ITU-T Standard H.264) in March 2010.

In addition, there is a newly developed video decoding standard, namely High Efficiency Video Decoding (HEVC), developed by the ITU-T Video Decoding Experts Group (VCEG) and the Joint Collaborative Team on Video Decoding (JCT-VC) of the ISO/IEC Animation Experts Group (MPEG). The latest draft of HEVC is available at http://phenix.int-evry.fr/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1003-v34.zip. The HEVC standard is also jointly proposed in ITU-T H.265 and the international standard ISO/IEC 23008-2, both titled "High Efficiency Video Decoding," and both were published in October 2014.

JCT-VC developed the HEVC standard. The HEVC standardization effort is based on an evolutionary model of video decoding devices, known as the HEVC Test Model (HM). The HM assumes, for example, several additional capabilities of the video decoding device compared to existing devices, according to ITU-T H.264/AVC. For instance, while H.264 provides nine intra-frame prediction coding modes, HEVC HM can provide up to thirty-three intra-frame prediction coding modes.

Generally, the working model of HM describes video frames or images as sequences of tree blocks or maximum decoding units (LCUs) containing both luma and chroma samples. The syntax data within the bitstream defines the size of the LCU, which is the maximum decoding unit in terms of the number of pixels. A slice contains several consecutive tree blocks in decoding order. Video frames or images can be divided into one or more slices. Each tree block can be split into several decoding units (CUs) according to a quadtree. Generally, in a quadtree data structure, each CU contains one node, where the root node corresponds to the tree block. If a CU splits into four sub-CUs, then the node corresponding to that CU contains four leaf nodes, each of which corresponds to one of the sub-CUs.

Each node in a quadtree data structure provides syntax data for the corresponding CU. For example, a node in a quadtree may contain a split flag indicating whether the CU corresponding to that node has split into child CUs. The syntax elements for a CU can be defined recursively and can depend on whether the CU splits into child CUs. If a CU does not split further, then that CU is called a leaf CU. In this invention, even if there is no obvious split of the original leaf CU, the four child CUs of the leaf CU will also be called leaf CUs. For example, if a 16×16 CU does not split further, then the four 8×8 child CUs will also be called leaf CUs, even though the 16×16 CU has never split.

Except that CUs do not have size distinctions, they serve a similar purpose to macroblocks in the H.264 standard. For example, a tree block can be split into four child nodes (also called child CUs), and each child node can be a parent node and can be split into four more child nodes. The final unsplit child nodes of a leaf node, called a quadtree, include decoder nodes, also called leaf CUs. The syntax data associated with the decoded bitstream defines the maximum number of times the tree block can be split (called the maximum CU depth) and also defines the minimum size of the decoder nodes. Therefore, the bitstream can also define the minimum decoding unit (SCU). This invention uses the term "block" to refer to any of the CUs, PUs, or TUs in the context of HEVC, or similar data structures in the context of other standards (e.g., macroblocks and their child blocks in H.264/AVC).

A CU comprises a decoding node and associated prediction units (PUs) and transform units (TUs). The size of the CU corresponds to the size of the decoding node and must be square. The size of the CU can range from 8×8 pixels to a tree block size with a maximum of 64×64 pixels or more. Each CU may contain one or more PUs and one or more TUs. The syntax data associated with the CU may describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may differ between skip or direct mode coding, intra-frame prediction mode coding, or inter-frame prediction mode coding. PUs may be partitioned into non-square shapes. The syntax data associated with the CU may also describe, for example, the partitioning of the CU into one or more TUs according to a quadtree. TUs may be square or non-square (e.g., rectangular) shapes.

The HEVC standard allows for transformations based on the transform unit (TU), which can differ for different core cells (CUs). Typically, the TU size is set based on the size of the learning unit (PU) within a given CU defined by the segmented LCU, but this is not always the case. The TU size is usually the same as or smaller than the PU. In some instances, a quadtree structure called a "residual quadtree" (RQT) can be used to further divide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT can be called transform units (TUs). The pixel differences associated with the TU can be transformed to produce quantizable transform coefficients.

A leaf CU may contain one or more prediction units (PUs). Generally, a PU represents all or part of a spatial region corresponding to a given CU and may contain data for retrieving reference samples for the PU. Furthermore, a PU contains prediction-related data. For example, when a PU is coded in an intra-frame mode, the PU's data may be contained in a residual quadtree (RQT), which may contain data describing the intra-frame prediction mode used for the TU corresponding to the PU. As another example, when a PU is coded in an inter-frame mode, the PU may contain data defining one or more motion vectors of the PU. The data defining the motion vectors of the PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., quarter-pixel accuracy or eighth-pixel accuracy), the reference picture to which the motion vector points, and/or a list of reference pictures for the motion vector (e.g., list 0, list 1, or list C).

A leaf CU with one or more PUs may also contain one or more transform units (TUs). As discussed above, the transform unit can be specified using an RQT (also known as a TU quadtree structure). For example, a split flag can indicate whether a leaf CU is split into four transform units. Each transform unit can then be further split into several other sub-TUs. When a TU is not further split, it can be called a leaf TU. Generally, for intra-frame decoding, all leaf TUs belonging to a leaf CU share the same intra-frame prediction mode. That is, the same intra-frame prediction mode is generally used to calculate the prediction values of all TUs of the leaf CU. For intra-frame decoding, the video encoder can use the intra-frame prediction mode to calculate the residual value of each leaf TU as the difference between the portion of the CU corresponding to the TU and the original block. The TU is not necessarily limited to the size of the PU. Therefore, a TU can be larger or smaller than the PU. For intra-frame decoding, a PU can coexist with the corresponding leaf TU of the same CU. In some instances, the maximum size of a leaf TU can correspond to the size of the corresponding leaf CU.

Furthermore, the TU of a leaf CU can also be associated with a corresponding quadtree data structure (called a residual quadtree (RQT)). That is, a leaf CU can contain a quadtree indicating how the leaf CU is partitioned into TUs. The root node of the TU quadtree typically corresponds to a leaf CU, while the root node of the CU quadtree typically corresponds to a tree block (or LCU). The unsplit TU of the RQT is referred to as a leaf TU. Generally, unless otherwise indicated, the terms CU and TU are used in this invention to refer to leaf CU and leaf TU, respectively.

A video sequence typically comprises a series of video frames or pictures. A group of pictures (GOP) generally includes one or more of these video pictures. A GOP may contain a header, headers of one or more of the pictures, or syntax data elsewhere, which describes the number of pictures contained in the GOP. Each slice of a picture may contain slice syntax data describing the encoding mode of the corresponding slice. The video encoder 20 typically operates on video blocks within individual video slices to encode video data. Video blocks may correspond to decoding nodes within a CU. Video blocks may have a fixed or variable size and may vary in size depending on a specified decoding standard.

As an example, HM supports prediction for various PU sizes. Assuming a specific CU size of 2N×2N, HM supports intra-frame prediction for PU sizes of 2N×2N or N×N (in the case of an 8×8 CU), and inter-frame prediction for symmetrical PU sizes of 2N×2N, 2N×N, N×2N, or N×N. HM also supports asymmetric segmentation for inter-frame prediction of PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N. In asymmetric segmentation, one direction of the CU is not segmented, while the other direction is segmented into 25% and 75%. The portion of the CU corresponding to the 25% segmentation is indicated by “n” followed by “Up,” “Down,” “Left,” or “Right.” Thus, for example, “2N×nU” refers to a 2N×2N CU horizontally segmented with a top 2N×0.5N PU and a bottom 2N×1.5N PU.

In this invention, "N×N" and "N multiplied by N" are used interchangeably to refer to the pixel size of a video block in both the vertical and horizontal dimensions, for example, 16×16 pixels or 16 by 16 pixels. Generally, a 16×16 block will have 16 pixels in the vertical direction (y=16) and 16 pixels in the horizontal direction (x=16). Similarly, an N×N block typically has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. Pixels in a block can be arranged in rows and columns. Furthermore, a block does not necessarily need to have the same number of pixels in the horizontal and vertical directions. For example, a block may include N×M pixels, where M is not necessarily equal to N.

After intra-frame predictive or inter-frame predictive decoding of the PU of the CU, the video encoder 20 may compute residual data of the TU of the CU. The PU may include syntax data describing the method or pattern of generating predictive pixel data in the spatial domain (also known as the pixel domain), and the TU may include coefficients in the transform domain after applying a transform (e.g., discrete cosine transform (DCT), integer transform, wavelet transform, or a conceptually similar transform) to the residual video data. The residual data may correspond to the pixel difference between the pixels of the uncoded image and the predicted value corresponding to the PU. The video encoder 20 may form a TU containing the residual data of the CU, and then transform the TU to produce the transform coefficients of the CU.

After any transformation used to generate the transform coefficients, the video encoder 20 may perform quantization on the transform coefficients. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. The quantization process can reduce the bit depth associated with some or all of the coefficients. For example, an n-bit value can be down-rounded to an m-bit value during quantization, where n is greater than m.

After quantization, the video encoder can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) coefficients at the front of the array and lower-energy (and therefore higher-frequency) coefficients at the back. In some instances, the video encoder 20 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector that can be entropy-encoded. In other instances, the video encoder 20 can perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 20 can entropy-encode the one-dimensional vector, for example, according to context-adaptive variable-length decoding (CAVLC), context-adaptive binary arithmetic decoding (CABAC), syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioning entropy (PIPE) decoding, or another entropy coding method. The video encoder 20 can also entropy-encode syntax elements associated with the encoded video data for use by the video decoder 30 when decoding the video data.

To perform CABAC, video encoder 20 can assign context within a context model to the symbol to be emitted. The context might be, for example, whether the symbol's neighboring values are non-zero. To perform CAVLC, video encoder 20 can select variable-length codes for the symbol to be emitted.

Codewords in a VLC can be constructed such that relatively shorter codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, bit savings can be achieved using VLC compared to (for example) using codewords of equal length for each symbol to be transmitted. Probability determination can be based on the context assigned to the symbol.

According to one or more techniques of the present invention, the video encoder 20 and/or the video decoder 30 may implement one or more of the techniques of the present invention. For example, the video encoder 20 and/or the video decoder 30 may use affine models for motion estimation and compensation.

Figure 2 is a block diagram illustrating an example of a video encoder 20 that can be configured to perform motion vector prediction according to the technology of the present invention. The video encoder 20 can perform intra-frame decoding and inter-frame decoding of video blocks within a video slice. Intra-frame decoding relies on spatial prediction to reduce or remove spatial redundancy of video within a given video frame or image. Inter-frame decoding relies on temporal prediction to reduce or remove temporal redundancy of video within neighboring frames or images of a video sequence. Intra-frame mode (I-mode) can refer to any of several spatial-based decoding modes. Inter-frame mode (e.g., one-way prediction (P-mode) or two-way prediction (B-mode)) can refer to any of several time-based decoding modes.

As shown in Figure 2, the video encoder 20 receives the current video block within the video slice to be encoded. In the example of Figure 2, the video encoder 20 includes a mode selection unit 40, a reference image memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The mode selection unit 40 further includes a motion compensation unit 44, a motion estimation unit 42, an intra-frame prediction unit 46, and a segmentation unit 48. For video block reconstruction, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. A deblocking filter (not shown in Figure 2) may also be included to filter block boundaries to remove block artifacts from the reconstructed video. When needed, the deblocking filter will typically filter the output of the summer 62. In addition to the deblocking filter, additional filters (in-loop or post-loop) may be used. Such filters are not shown for simplicity, but when needed, they can filter the output of the summer 50 (as an in-loop filter).

During the encoding process, the video encoder 20 receives video frames or slices to be decoded. Frames or slices can be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-frame predictive decoding of the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-frame prediction unit 46 may alternatively perform intra-frame predictive decoding of the received video blocks relative to one or more adjacent blocks in the same frame or slice as the block to be decoded to provide spatial prediction. The video encoder 20 can perform multiple decoding passes, for example, to select an appropriate decoding mode for each video data block.

Furthermore, segmentation unit 48 can segment blocks of video data into sub-blocks based on an evaluation of previous segmentation schemes in previous decoding passes. For example, segmentation unit 48 can first segment frames or slices into LCUs, and then segment each of the LCUs into sub-CUs based on bitrate-distortion analysis (e.g., bitrate-distortion optimization). Mode selection unit 40 can further generate a quadtree data structure indicating the segmentation of LCUs into sub-CUs. The leaf nodes CU of the quadtree can contain one or more PUs and one or more TUs.

The mode selection unit 40 may, for example, select a decoding mode (intra-frame or inter-frame) based on the error result, and provide the resulting intra-frame decoded block or inter-frame decoded block to the summer 50 to generate residual block data, and to the summer 62 to reconstruct the coded block for use as a reference frame. The mode selection unit 40 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

Motion estimation unit 42 and motion compensation unit 44 can be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. For example, a motion vector may indicate the displacement of the PU of a video block within the current video frame or image relative to a predictive block within a reference image (or other decoded unit) relative to the current block being decoded within the current image (or other decoded unit). A predictive block is a block found to closely match the block to be decoded in terms of pixel differences, which can be determined by the sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some instances, video encoder 20 may calculate the value of the second-integer pixel position of a reference image stored in reference image memory 64. For example, video encoder 20 may interpolate the value of a quarter-pixel position, an eighth-pixel position, or other fractional pixel position of the reference image. Therefore, motion estimation unit 42 may perform motion search with respect to full-pixel positions and fractional pixel positions and output motion vectors with fractional-pixel precision.

The motion estimation unit 42 calculates the motion vector of the PU in the inter-frame decoded slice by comparing the position of the PU with the position of a predictive block in a reference image. The reference images can be selected from a first list of reference images (list 0) or a second list of reference images (list 1), each of which identifies one or more reference images stored in the reference image memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

Motion compensation performed by motion compensation unit 44 may involve extracting or generating predictive blocks based on motion vectors determined by motion estimation unit 42. Again, in some instances, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. After receiving the motion vector of the PU for the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference image lists. Summer 50 forms a residual video block by subtracting the pixel values of the predictive block from the pixel values of the properly decoded current video block, thus forming a pixel difference, as discussed below. Generally, motion estimation unit 42 performs motion estimation relative to the luminance component, and motion compensation unit 44 uses the motion vector calculated based on the luminance component for both the chroma and luminance components. Mode selection unit 40 may also generate syntax elements associated with video blocks and video slices for use by video decoder 30 when decoding video slices.

The video encoder 20 can be configured to perform any of the various techniques of the invention discussed above with respect to FIG1, as will be described in more detail below. For example, the motion compensation unit 44 can be configured to decode motion information for blocks of video data using AMVP or merging mode according to the techniques of the invention.

Assuming motion compensation unit 44 selects to execute the merging mode, it can form a candidate list containing a set of merging candidates. Motion compensation unit 44 can add candidates to the candidate list in a specific predetermined order. As discussed above, motion compensation unit 44 can also add additional candidates and perform pruning of the candidate list. Finally, mode selection unit 40 determines which candidates will be used to encode the motion information of the current block, and the encoded representation is the merging index of the selected candidates.

As described above, as an alternative to inter-frame prediction performed by motion estimation unit 42 and motion compensation unit 44, intra-frame prediction unit 46 may perform intra-frame prediction for the current block. Specifically, intra-frame prediction unit 46 may determine the intra-frame prediction mode to be used to encode the current block. In some instances, intra-frame prediction unit 46 may use various intra-frame prediction modes to encode the current block, for example, during individual encoding passes, and intra-frame prediction unit 46 (or, in some instances, mode selection unit 40) may select an appropriate intra-frame prediction mode from the tested modes for use.

For example, intra-prediction unit 46 can use bitrate-distortion analysis for various tested intra-prediction modes to calculate bitrate-distortion values and select the intra-prediction mode with the best bitrate-distortion characteristics among the tested modes. Bitrate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original, uncoded block (which is encoded to produce the coded block), and the bitrate (i.e., the number of bits) used to produce the coded block. Intra-prediction unit 46 can calculate a ratio based on the distortion and bitrate of various coded blocks to determine which intra-prediction mode exhibits the best bitrate-distortion value for the block.

After selecting an intra-prediction mode for a block, the intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 may encode the information indicating the selected intra-prediction mode. The video encoder 20 may include the following in the transmitted bitstream: configuration data, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also called codeword maps); definitions of the coding contexts for various blocks; and indications of the most probable intra-prediction mode to be used for each of the contexts, the intra-prediction mode index tables, and the modified intra-prediction mode index tables.

The video encoder 20 forms a residual video block by subtracting the prediction data from the mode selection unit 40 from the original video block being decoded. The summer 50 represents one or more components performing this subtraction. The transform processing unit 52 applies a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform) to the residual block, thereby producing a video block that includes the residual transform coefficient values. The transform processing unit 52 can perform other transforms conceptually similar to DCT. Wavelet transform, integer transform, subband transform, or other types of transforms can also be used.

In any case, transform processing unit 52 applies a transform to the residual block, thereby producing a residual transform coefficient block. The transform can convert residual information from the pixel value domain to the transform domain, such as the frequency domain. Transform processing unit 52 can send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some instances, quantization unit 54 may then perform a scan of a matrix containing the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

After quantization, entropy coding unit 56 performs entropy decoding on the quantized transform coefficients. For example, entropy coding unit 56 may perform context-adaptive variable-length decoding (CAVLC), context-adaptive binary arithmetic decoding (CABAC), syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioning entropy (PIPE) decoding, or another entropy decoding technique. In the case of context-based entropy decoding, the context may be based on adjacent blocks. After entropy decoding by entropy decoding unit 56, the encoded bitstream can be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, to be used later as a reference block. Motion compensation unit 44 can calculate the reference block by adding the residual block to a predictive block of one of the frames in reference image memory 64. Motion compensation unit 44 can also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion-compensated predictive block generated by motion compensation unit 44 to produce a reconstructed video block for storage in reference image memory 64. The reconstructed video block can be used as a reference block by motion estimation unit 42 and motion compensation unit 44 for inter-frame decoding of blocks in subsequent video frames.

Figure 3 is a block diagram illustrating an example of a video decoder 30 that can be configured to perform the motion vector prediction technique of the present invention. In the example of Figure 3, the video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intra-frame prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference image memory 82, and a summer 80. In some instances, the video decoder 30 may perform a decoding pass that is substantially the inverse of the encoding pass described with respect to the video encoder 20 (Figure 2). The motion compensation unit 72 may generate prediction data based on motion vectors received from the entropy decoding unit 70, while the intra-frame prediction unit 74 may generate prediction data based on intra-frame prediction mode indicators received from the entropy decoding unit 70.

During the decoding process, video decoder 30 receives from video encoder 20 a encoded video bitstream representing video blocks of encoded video slices and associated syntax elements. Entropy decoding unit 70 of video decoder 30 entropy decodes the bitstream to produce quantized coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive syntax elements at the video slice level and/or video block level.

When a video slice is decoded into an intra-decoded (I) slice, the intra-prediction processing unit 74 can generate prediction data for the video block of the current video slice based on the intra-prediction mode transmitted via signal transmission and data from the previously decoded block of the current frame or picture. When a video frame is decoded into an inter-decoded (i.e., B, P, or GPB) slice, the motion compensation unit 72 generates predictive blocks for the video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 70. The predictive blocks can be generated from one of the reference pictures in the reference picture list. The video decoder 30 can construct the reference frame list: list 0 and list 1, based on the reference pictures stored in the reference picture memory 82 using a default construction technique.

Motion compensation unit 72 determines prediction information for video blocks in the current video slice by analyzing motion vectors and other syntax elements, and uses the prediction information to generate predictive blocks for proper decoding of the current video slice. For example, motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction), inter-frame prediction slice type (e.g., B-slice or P-slice), construction information of one or more of the slice's reference picture list, motion vectors of each inter-frame encoded video block of the slice, inter-frame prediction state of each inter-frame decoded video block of the slice, and other information for decoding video blocks in the current video slice.

The motion compensation unit 72 can also perform interpolation based on an interpolation filter. The motion compensation unit 72 can use an interpolation filter, such as that used by the video encoder 20 during video block encoding, to calculate the interpolated values of the second-integer pixels of the reference block. In this case, the motion compensation unit 72 can determine the interpolation filter used by the video encoder 20 from the received syntax elements and use the interpolation filter to generate a predictive block.

The video decoder 30 can be configured to perform any of the various techniques of the invention discussed above with respect to FIG. 1, as will be discussed in more detail below. For example, the motion compensation unit 72 can be configured to determine whether to perform motion vector prediction using AMVP or merging mode according to the technique of the invention. The entropy decoding unit 70 can decode one or more syntax elements representing how motion information is used for decoding the current block.

Assuming the syntax element indicates that a merge mode is executed, motion compensation unit 72 can form a candidate list containing a set of merge candidates. Motion compensation unit 72 can add candidates to the candidate list in a specific predetermined order. As discussed above, motion compensation unit 72 can also add additional candidates and perform pruning of the candidate list. Finally, motion compensation unit 72 can decode the merge index indicating which candidate was used to decode the motion information of the current block.

The dequantization unit 76 dequantizes (i.e., dequantizes) the quantized transform coefficients provided in the bitstream and entropy-decoded by the entropy decoding unit 70. The dequantization process may include using the quantization parameter _QPY calculated by the video decoder 30 for each video block in the video slice to determine the degree of quantization to be applied and similarly determining the degree of dequantization to be applied.

The inverse transform unit 78 applies an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to produce a residual block in the pixel domain.

After the motion compensation unit 72 generates a predictive block for the current video block based on motion vectors and other syntax elements, the video decoder 30 forms a decoded video block by summing the residual block from the inverse transform unit 78 with the corresponding predictive block generated by the motion compensation unit 72. The summer 80 represents one or more components performing this summation operation. If necessary, a deblocking filter can be applied to the decoded block to remove block artifacts. Other loop filters (within or after the decoding loop) can also be used to smooth pixel transitions or otherwise improve video quality. The decoded video block in a given frame or image is then stored in a reference image memory 82, which stores reference images for subsequent motion compensation. The reference image memory 82 also stores the decoded video for later presentation on a display device (e.g., display device 32 of FIG. 1).

Figure 4 is a conceptual diagram illustrating aspects of intra-frame prediction. The video encoder 20 and/or video decoder 30 can implement intra-frame prediction to perform image block prediction using spatially adjacent reconstructed image samples of the block. A typical example of intra-frame prediction for a 16×16 image block is shown in Figure 4. As illustrated in Figure 4, through intra-frame prediction, a 16×16 image block (shown as a solid-lined square) is predicted based on the upper and left adjacent reconstructed samples (reference samples) located in the nearest upper row and left column along a selected prediction direction (as indicated by the arrows). In HEVC, intra-frame prediction for luma blocks includes 35 modes.

Figure 5 is a conceptual diagram illustrating the intra-prediction modes used for luma blocks. These modes include a planar mode, a DC mode, and 33 angular modes, as indicated in Figure 5. The 35 intra-prediction modes defined in HEVC are indexed and shown in Table 1 below:

Intra-prediction mode Related names 0 INTRA_PLANAR 1 INTRA_DC 2..34 INTRA_ANGULAR2..INTRA_ANGULAR34

Table 1 - Specifications of Intra-Frame Prediction Modes and Associated Names

Figure 6 is a conceptual diagram illustrating aspects of the planar mode. For the planar mode, which is typically the most commonly used intra-frame prediction mode, the prediction samples are generated as shown in Figure 6. To perform planar prediction on an N×N block, for each sample located at (x, y), the video encoder 20 and/or video decoder 30 can use a bilinear filter to compute a prediction using four specific adjacent reconstructed samples (i.e., reference samples). The four reference samples include the upper right reconstructed sample TR, the lower left reconstructed sample BL, and two reconstructed samples located in the same column (rx, -1) of the current sample (denoted as T) and in the same row (r-1, y) of the current sample (denoted as L). The planar mode is formulated as shown in the following equation: p _{_xy} = (Nx-1)·L + (Ny-1)·T + x·TR + y·BL.

For DC mode, the prediction block is simply filled with the average of neighboring reconstructed samples. Generally, both planar mode and DC mode are applied to modeled, smoothly varying, and constant image regions.

Figure 7 is a conceptual diagram illustrating aspects of the angular patterns according to HEVC. For the angular intra-prediction patterns in HEVC, which contain a total of 33 different prediction directions, the intra-prediction process is described below. For each given angular intra-prediction, the corresponding intra-prediction direction can be identified. For example, according to Figure 5, intra-prediction pattern 18 corresponds to a purely horizontal prediction direction, and intra-prediction pattern 26 corresponds to a purely vertical prediction direction. Given a specific intra-prediction direction, for each sample in the prediction block, the sample's coordinates (x, y) are first projected along the prediction direction to the row/column of adjacent reconstructed samples, as shown in the example in Figure 7. Assuming (x, y) is projected to a fractional position α between two adjacent reconstructed samples L and R, a two-tap bilinear interpolation filter is used to calculate the predicted value for (x, y), formulated as shown in the following equation: p_{_xy} = (1-α)·L + α·R. To avoid floating-point operations, HEVC actually uses integer arithmetic such as p _xy = ((32-a)·L+a·R+16)>>5 to approximate the above calculation, where a is an integer equal to 32*α.

The following describes the aspects of chroma encoding and decoding in general. The structure in a chroma signal often follows the structure of the corresponding luminance signal. As described, according to HEVC, each luminance block corresponds to one chroma block, and each chroma prediction block may correspond to one or four luminance prediction blocks based on a partition size equal to 2N×2N or N×N luminance prediction blocks. Taking advantage of these characteristics and general trends in the structure of chroma signals, HEVC provides a mechanism for the video encoder 20 to indicate to the video decoder 30 that the chroma PU is predicted using the same prediction mode as the corresponding selected luminance PU. Table 2 below specifies the mode arrangement of the chroma modes that the video encoder 20 can use to signal the chroma modes for the chroma PU. For example, an intra-coded chroma PU may be predicted using a mode selected from five (5) modes, including planar mode (INTRA_PLANAR), vertical mode (INTRA_ANGULAR26), horizontal mode (INTRA_ANGULAR10), DC mode (INTRA_DC), and derived mode (DM). The DM is set to the intra-prediction mode for predicting the corresponding selected luminance PU. For example, if the corresponding selected luminance PU is decoded using an intra-prediction mode with an index of 11, then the DM is set to the intra-prediction mode with an index of 11.

Table 2 - Specifications of Chroma Intra-Frame Prediction Modes and Associated Names

If the encoded video bitstream indicates that an export mode will be used for the PU, then the video decoder 30 can use the prediction mode for the corresponding luma PU to perform prediction for the chroma PU. To mitigate redundancy issues that may arise when the export mode indicates that one of the prediction modes is always present, the video encoder 20 and the video decoder 30 can use a specified alternative mode as a substitute for the repeating mode. As shown in Table 2 above, the video encoder 20 and the video decoder 30 can use the “INTRA_ANGULAR34” chroma alternative mode, also known as the “angle (34) mode,” as a substitute to remove redundancy. For example, if the relationship between the chroma PU and the luma PU is one-to-one or many-to-one, the video encoder 20 and the video decoder 30 can determine the prediction mode for the chroma PU by selecting a prediction mode applicable to a single corresponding luma PU.

However, in some cases, one chroma PU can correspond to multiple luma PUs. The scenario where a single chroma PU corresponds to multiple luma PUs is an exception or "special case" regarding chroma encoding and decoding. For example, in some of these special cases, one chroma PU can correspond to four luma PUs. In the special case where the chroma-luma relationship is one-to-many, the video encoder 20 and video decoder 30 can determine the prediction mode for the chroma PU by selecting the prediction mode corresponding to the upper left luma PU.

Video encoder 20 and video decoder 30 can entropy decode (entropy encoding and entropy decoding, respectively) the data indicating the chroma prediction mode for the video data block. Based on the chroma mode decoding, video encoder 20 can assign a 1-b syntax element (0) to a single most frequently occurring derived mode, while assigning 3-b syntax elements (100, 101, 110, and 111, respectively) to each of the remaining four modes. Video encoder 20 and video decoder 3 can decode only the first binary (bin) through a context model, and can bypass decoding the remaining two binary (if needed).

Video encoder 20 and video decoder 30 can entropy decode (entropy encoding and entropy decoding, respectively) video data using context-based adaptive binary arithmetic decoding (CABAC). CABAC is an entropy decoding method first introduced in H.264/AVC and described in "Context-based adaptive binaryarithmetic coding in the H.264/AVC video compression standard" by D. Marpe, H. Schwarz, and T. Wiegand (IEEE Trans. Circuits Syst. Video Technol., July 2003, Vol. 13, No. 7, pp. 620-636). CABAC is now used in the High Efficiency Video Decoding (HEVC) video decoding standard. Video encoder 20 and video decoder 30 can use CABAC for entropy decoding in a similar manner to CABAC performed for HEVC.

CABAC involves three main functions: binaryization, context modeling, and arithmetic decoding. The binaryization function maps syntax elements to binary symbols (bits) called binary strings. The context modeling function estimates the probabilities of the bits. The arithmetic decoding function (also known as binary arithmetic decoding) compresses the bits to bits based on the estimated probabilities.

The video encoder 20 and video decoder 30 can perform binaryization for CABAC using one or more of the various binaryization processes provided in HEVC. The binaryization processes provided in HEVC include unary (U), truncated unary (TU), k-order exponential Columbus (EGk), and fixed-length (FL) techniques. Details of these binaryization processes are described in V. Sze and M. Budagavi's "High throughput CABAC entropy coding in HEVC" (IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), December 2012, Vol. 22, No. 12, pp. 1778-1791).

According to unary-based encoding, video encoder 20 can transmit a binary string of length N+1, where "N" represents an integer value, the first N bits (values) are 1, and the last bit (value) is 0. According to unary-based decoding, video decoder 30 can search for the binary value of 0. After finding the binary value of 0, video decoder 30 can determine that the syntax element is complete.

According to truncated unary decoding, video encoder 20 can encode one less binary bit than in the case of unary decoding. For example, video encoder 20 can set the maximum possible value of the syntax element. The maximum value is indicated by "cMax" in this document. When (N+1) < cMax, video encoder 20 can perform the same signal transmission as in unary decoding. However, when (N+1) = cMax, video encoder 20 can set all binary bits to the corresponding value 1. Video decoder 30 can search for 0-value binary bits until cMax number of binary bits have been checked to determine when the syntax element is complete. Aspects of the binary strings used in unary and truncated unary decoding and their comparisons are illustrated in Table 3 below. The compared binary values are illustrated in Table 3 and highlighted in bold italics.

Table 3 - Examples of unary and truncated unary binary strings

The video encoder 20 and video decoder 30 also perform the context modeling aspect of CABAC. Context modeling provides relatively accurate probability estimates, which is an aspect of achieving efficient decoding. Therefore, context modeling is an adaptive process and is sometimes described as “highly adaptive.” Different context models can be used for different binary bits, where the probabilities of the context model can be updated based on the values of previously decoded binary bits. Binaries with similar distributions often share the same context model. The video encoder 20 and/or video decoder 30 can select the context modeling for each binary bit based on one or more factors including: the type of syntax element, the binary position (binIdx) in the syntax element, luma/chroma, adjacency information, etc.

The video encoder 20 and video decoder 30 can perform context switching after each instance of binary decoding (binary encoding or binary decoding, depending on the specific case). The video encoder 20 and video decoder 30 can store the probability model as a 7-bit entry (6 bits for probability state and 1 bit for most probable symbol (MPS)) in context memory, and the probability model can be addressed using a context index computed by the context selection logic. HEVC provides the same probability update method as H.264/AVC. However, the HEVC-based context selection logic is a modification of the H.264/AVC context selection logic to improve throughput. The video encoder 20 and video decoder 30 can also use the probability representation for CABAC entropy encoding and decoding, respectively. For CABAC, 64 representative probability values _pσ ∈ [0.01875, 0.5] are derived for the least probable symbol (LPS) through the following recursive equation:

_pσ = α* _pσ-1 , for all σ = 1,…,63,

in

In the above equation, the chosen scaling factor α ≈ 0.9492 and the cardinality N = 64 in the group probabilities represent a trade-off between the accuracy of the probability representation and the speed of adaptation. The parameters used in the above equation demonstrate a relatively good trade-off between the accuracy of the probability representation and the need for faster adaptation. The probability of MPS is equal to 1 minus the probability of LPS (i.e., (1-LPS)). Therefore, the probability range that can be represented by CABAC is [0.01875, 0.98125]. The upper limit of the range (MPS probability) is equal to one minus the lower limit (i.e., one minus the LPS probability). That is, 1 - 0.01875 = 0.98125.

Before encoding or decoding a specific slice, the video encoder 20 and video decoder 30 can initialize a probabilistic model based on some predefined values. For example, given the input quantization parameter indicated by "qp" and the predefined value indicated by "initVal", the video encoder 20 and/or video decoder 30 can derive the following 7-bit entries of the probabilistic model (indicated by "state" and "MPS"):

qp = Clip3(0,51,qp);

slope = (initVal >> 4) * 5 - 45;

offset=((initVal&15)<<3)-16;

initState=min(max(1,(((slope*qp)>>4)+offset)),126);

MPS = (initState >= 64);

state index=((mpState?(initState-64):(63-initState))<<1)+MPS;

The exported state index implicitly contains MPS information. That is, when the state index is an even number, the MPS value is equal to 0. Conversely, when the state index is an odd number, the MPS value is equal to 1. The value of “initVal” is in the range [0, 255] with 8-bit precision.

The predefined `initVal` is slice-dependent. That is, the video encoder 20 can use three sets of context initialization parameters respectively for probabilistic models specifically for decoding I-slices, P-slices, and B-slices. In this way, the video encoder 20 is enabled to select among three initialization tables for these three slice types, potentially enabling better adaptation to different decoding scenarios and/or different types of video content.

Recent advancements in JEM 3.0 include developments in intra-frame mode decoding. According to these advancements, video encoder 20 and video decoder 30 can perform intra-frame mode decoding with six most probable modes (MPMs). As described in “Neighbor-based intra most probable modes list derivation” (JVET-C0055, Geneva, May 2016) by V. Seregin, X. Zhao, A. Said, and M. Karczewicz ("Neighbor-based intra most probable modes list derivation"). The number of angular modes in HEVC has been expanded from 33 to 65, in addition to DC and planar modes with six most probable modes (MPMs). Video encoder 20 can encode a flag (e.g., “MPM flag”) to indicate whether an intra-frame luma mode is included in the MPM candidate list, which contains six modes (as described in JVET-C0055 cited above). If an intra-frame luma mode is included in the MPM candidate list (causing the video encoder 20 to set the MPM flag to a positive value), then the video encoder 20 can further encode and signal the index of the MPM candidate to indicate which MPM candidate in the list is the intra-frame luma mode. Otherwise (i.e., if the video encoder 20 sets the MPM flag to a negative value), the video encoder 20 can further signal the index of the remaining intra-frame luma modes.

Based on these advancements in JEM 3.0, the video decoder 30 can decode the MPM flag after receiving the encoded video bitstream transmitted as a signal to determine whether an intra-frame luma mode is included in the MPM candidate list. If the video decoder 30 determines that the MPM flag is set to a positive value, then the video decoder 30 can decode the received index to identify the intra-frame luma mode from the MPM candidate list. Conversely, if the video decoder 30 determines that the MPM flag is set to a negative value, then the video decoder 30 can receive and decode the indices of the remaining intra-frame luma modes.

Recent advancements in JEM3.0 have also been made regarding Adaptive Multi-Core Transform (AMT). In addition to DCT-II and 4×4 DST-VII used in HEVC, the AMT scheme is also applied to residual decoding of both inter-frame and intra-frame decoding blocks. AMT utilizes several selected transforms from the DCT/DST family, in addition to those currently defined in HEVC. The newly introduced transform matrices in JEM3.0 are DST-VII, DCT-VIII, DST-I, and DCT-V.

For intra-frame residual decoding, due to the different residual statistics of different intra-frame prediction modes, the video encoder 20 and video decoder 30 can use a mode-dependent transform candidate selection process. Three transform subsets are defined as shown in Table 4 below, and the video encoder 20 and/or video decoder 30 can select transform subsets based on intra-frame prediction modes, as specified in Table 5 below.

Table 4: Three predefined sets of transformation candidates

Table 5: Selected Horizontal (H) and Vertical (V) Transform Sets for Prediction Mode in Each Frame

Based on the subset concept, the video decoder 30 can first identify transform subsets based on Table 6 below. For example, to identify transform subsets, the video decoder 30 can use the intra-frame prediction mode of the CU, which is signaled with the CU-level AMT flag set to value 1. Subsequently, for each of the horizontal and vertical transforms, the video decoder 30 can select one of two transform candidates from the identified transform subset according to Table 7 below. The selected transform candidate for each of the horizontal and vertical transforms is selected based on the explicit signaling of flagged data. However, for the inter-frame prediction residual, the video decoder 30 can use only one transform set for all inter-frame modes and for both horizontal and vertical transforms, consisting of DST-VII and DCT-VIII.

Table 6 - Specifications of Chroma Intra-Frame Prediction Modes and Associated Names

Table 7 - Binary strings for each chroma mode

Recent advancements in JEM 3.0 have been made regarding LM (Linear Model) prediction modes for video decoding. The video decoding apparatus of this invention (e.g., video encoder 20 and video decoder 30) can handle aspects of color space and color format during video encoding and decoding. Color video plays a major role in multimedia systems, where various color spaces are used to efficiently represent colors. Color spaces use multiple components to specify colors using digital values. A commonly used color space is the "RGB" color space, where colors are represented as combinations of the three primary color component values (i.e., red, green, and blue). For color video compression, the YCbCr color space has been widely used, as described in A. Ford and A. Roberts' "Colour Space Conversions" (Tech. Rep, London, University of Westminster, August 1998). YCbCr can be relatively easily converted from the RGB color space via linear transformation. In the RGB to YCbCr conversion, redundancy between different components (i.e., cross-component redundancy) is significantly reduced in the resulting YCbCr color space.

One advantage of YCbCr is its backward compatibility with black and white TVs, because the Y signal conveys luminance information. Additionally, the chrominance bandwidth can be reduced by subsampling the Cb and Cr components in a 4:2:0 chrominance sampling format, resulting in significantly less subjective impact compared to subsampling in RGB. Due to these advantages, YCbCr has become the primary color space in video compression. Other color spaces, such as YCoCg, also exist for video compression. For illustrative purposes, regardless of the actual color space used, Y, Cb, and Cr signals are used throughout this invention to represent the three color components in the video compression scheme. In 4:2:0 sampling, the height and width of each of the two chrominance arrays (Cb and Cr) are half that of the luminance array (Y).

Figure 8 is a conceptual diagram illustrating examples of nominal vertical and horizontal luminance and chrominance samples in an image. The nominal vertical and horizontal relative positions of the luminance and chrominance samples in the image are roughly corresponding to the positions provided by a 4:2:0 sampling format, as shown in Figure 8.

Aspects of the LM prediction mode used for video decoding will be discussed in the following paragraphs. Although cross-component redundancy is significantly reduced in the YCbCr color space, correlation between the three color components still exists in the YCbCr color space. Various techniques have been investigated to improve video decoding performance by further reducing the correlation between color components. Regarding 4:2:0 chroma video decoding, the linear model (LM) prediction mode was studied during the development of the HEVC standard. The aspects of the LM prediction model are described in “CE6.a.4: Chroma intraprediction by reconstructed luma samples” by J. Chen, V. Seregin, W.-J. Han, J.-S. Kim and B.-M. Joen (Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16WP3 and ISO/IEC JTC1/SC29/WG11, JCTVC-E266, 5th meeting, Geneva, March 16-23, 2011).

When performing prediction according to the LM prediction mode, the video encoder 20 and the video decoder 30 can predict chroma samples based on the reduced-sampled reconstructed luminance samples of the same block by using the linear model shown in the following equation (1).

pred _C (i,j)＝α·rec _L (i,j)+β (1)

Where pred _C (i,j) represents the prediction of the chromaticity sample in the block and rec _L (i,j) represents the reconstructed luminance sample of the same block after reduction sampling. The parameters α and β are derived from the causal reconstructed samples around the current block.

Figure 9 is a conceptual diagram illustrating the location of samples used to derive the parameters used in predictions based on the linear model (LM) pattern. The instance of the selected reference sample depicted in Figure 9 is a derivation of α and β used in equation (1) above. If the chroma block size is represented by N×N (where N is an integer), then i and j are both in the range [0, N].

The video encoder 20 and the video decoder 30 can derive the parameters α and β in equation (1) by reducing or potentially minimizing the regression error between adjacent reconstructed luminance and chrominance samples around the current block according to the following equation (2).

The parameters α and β are solved as described below.

β=(∑y _i -α·∑x _i )/I (4)

Where x _{_i} represents the reduced-sampled reconstructed luminance reference sample, y_{_i} represents the reconstructed chrominance reference sample, and I represents the amount of reference samples (e.g., count). For a target N×N chrominance block, the total number of samples involved (I) is equal to 2N when both the left and top causal samples are available. The total number of samples involved (I) is equal to N when only the left or top causal samples are available.

In general, when applying LM prediction mode, the video encoder 20 and/or video decoder 30 may invoke the following steps in the following order:

a) Reduce the sampling of adjacent brightness samples;

b) Derive the linear parameters (i.e., α and β); and

c) Reduce the sampling of the current luminance block and derive the prediction from the reduced-sampled luminance block and linear parameters.

To further improve decoding efficiency, the video encoder 20 and/or the video decoder 30 may use downsampling filters (1,2,1) and (1,1) to derive adjacent samples x _i and downsampled luminance samples rec _L (i,j) within the corresponding luminance block.

Recent advancements in JEM 3.0 have also enabled predictions between chrominance components. In JEM, the LM prediction mode has been extended to predictions between two chrominance components. For example, the Cr component can be predicted from the Cb component. Instead of using reconstructed sample signals, the video encoder 20 and/or video decoder 30 can apply cross-component predictions in the residual domain. For example, the video encoder 20 and/or video decoder 30 can implement the residual domain application of cross-component predictions by adding the weighted reconstructed Cb residual to the original Cr intra-frame prediction to form the final Cr prediction. An example of this operation is shown in the following equation (3):

Video encoder 20 and/or video decoder 30 can derive a scaling factor α, as in LM mode. However, one difference is that a regression cost is added relative to the default α value in the error function, causing the derived scaling factor to be biased towards the default value (-0.5). LM prediction mode is added as an additional chroma intra-prediction mode. In this regard, video encoder 20 can add an extra RD cost check for the chroma component to select the chroma intra-prediction mode.

The Quad-tree Binary Tree (QTBT) structure is described in the following paragraphs. In the VCEG proposal COM16-C966 (“Block partitioning structure for next generation video coding” by J. An, Y.-W. Chen, K. Zhang, H. Huang, Y.-W. Huang, and S. Lei (International Telecommunication Union, COM16-C966, September 2015), a QTBT partitioning scheme is proposed for future video decoding standards exceeding HEVC. Simulations demonstrating that the QTBT structure proposed in COM16-C966 is more efficient than the quadtree structure used in HEVC. In the proposed QTBT structure of COM16-C966, the decoding tree block (CTB) is first partitioned according to the quadtree structure. The quadtree partitioning of a node can be repeated until the node reaches the minimum allowed quadtree leaf node size (MinQTSize).

According to the QTBT structure, if the size of a quadtree leaf node is no larger than the maximum allowed size of a binary tree root node (MaxBTSize), then the quadtree leaf nodes can be further divided according to the binary tree structure. The binary tree splitting of a given node can be repeated until the node reaches the minimum allowed size of a binary tree leaf node (MinBTSize), or until the repeated splitting reaches the maximum allowed depth of the binary tree (MaxBTDepth). The binary tree leaf node is called a CU (Cumulative Node), which can be used for prediction (e.g., intra-image or inter-image prediction) and transformation without any further splitting.

Based on the binary tree partitioning, the video encoder 20 and/or video decoder 30 can implement two partitioning types: symmetrical horizontal partitioning and symmetrical vertical partitioning. In an example of the QTBT partitioning structure, the CTU size is set to 128×128 (i.e., 128×128 luminance samples and two corresponding 64×64 chrominance samples), the MinQTSize is set to 16×16, the MaxBTSize is set to 64×64, the MinBTSize (for both width and height) is set to 4, and the MaxBTDepth is set to 4. The video encoder 20 and/or video decoder 30 can first apply the quadtree partitioning portion of the QTBT scheme to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes can have sizes ranging from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size).

If the leaf quadtree node is 128×128, then the video encoder 20 and/or video decoder 30 cannot further split the leaf quadtree node using the binary tree portion of the QTBT scheme because the node size exceeds MaxBTSize (64×64 in this case). Otherwise (i.e., if the node size does not exceed MaxBTSize of 64×64), the video encoder 20 and/or video decoder 30 can further split the leaf quadtree node using the binary tree segmentation portion of the QTBT structure. Therefore, the leaf node of the quadtree is also the root node of the binary tree portion of the QTBT scheme, and thus has a binary tree depth of 0. When repeated binary tree segmentation reaches a binary tree depth of MaxBTDepth (i.e., 4), the video encoder 20 and/or video decoder 30 does not perform any kind of further splitting with respect to the leaf node. When the binary tree portion of the QTBT scheme produces a binary tree node with a width equal to MinBTSize (i.e., 4), the video encoder 20 and/or video decoder 30 may not perform further horizontal splitting of the node. Similarly, when the binary tree portion of the QTBT scheme produces binary tree nodes with a height equal to MinBTSize (i.e., 4), the video encoder 20 and/or video decoder 30 may not perform further vertical splitting of the nodes. The leaf nodes of the binary tree portion of the QTBT scheme (in the case where the splitting is completely achieved to binary tree segmentation) are further processed by CUs through prediction and transformation without any other splitting.

Figure 10 is a conceptual diagram illustrating aspects of the QTBT partitioning scheme. The block diagram on the left side of Figure 10 illustrates an example of partitioning block 162 according to the QTBT partitioning structure. The quadtree partitioning aspect of the QTBT partitioning scheme is depicted using solid lines in block 162, while the binary tree partitioning aspect is depicted using dashed lines in block 162. Block 162 partitions into square leaf nodes when only the quadtree portion of the QTBT scheme is invoked, and into non-square rectangular leaf nodes in any case where the binary tree portion of the QTBT scheme is invoked (whether or not it is invoked in combination with the quadtree partitioning portion). Compared to HEVC partitioning techniques (where multiple transformations are possible), the QTBT partitioning scheme provides a system where the PU size is always equal to the CU size.

The schematic diagram on the right side of Figure 10 illustrates tree structure 164. Tree structure 164 is the corresponding tree structure for the segmentation shown with respect to block 162 in Figure 10. Also in the case of tree structure 164, with the support of the QTBT segmentation scheme in Figure 10, solid lines indicate quadtree splits, and dashed lines indicate binary tree splits. For each split (i.e., non-leaf) node of the binary tree type shown by the dashed lines in tree structure 164, the video encoder 20 can signal a corresponding one-bit flag to indicate which split type (i.e., horizontal or vertical) is used. According to some implementations of QTBT segmentation, the video encoder 20 can set the flag to a value of zero (0) to indicate a horizontal split and to a value of one (1) to indicate a vertical split. It should be understood that for the quadtree split portion of the QTBT segmentation structure, it is not necessary to indicate the split type, because quadtree splitting always splits the block horizontally and vertically into four equal-sized sub-blocks.

Figures 11A and 11B illustrate examples of independent segmentation structures for corresponding luma and chroma blocks according to a QTBT segmentation scheme. The QTBT block segmentation technique permits and supports the feature of corresponding luma and chroma blocks having independent QTBT-based segmentation structures. According to the QTBT segmentation scheme, for P and B slices, corresponding luma CTUs and chroma CTUs within a CTU share the same QTBT-based segmentation structure. However, for I slices, the luma CTU can be segmented into CUs according to a first QTBT-based segmentation structure, and the chroma CTU is segmented into chroma CUs according to a second QTBT-based segmentation structure, which may or may not be the same as the first QTBT-based segmentation structure. Therefore, a CU in an I slice can consist of a decoding block for the luma component or decoding blocks for the two chroma components, while for CUs in P and B slices, a CU can consist of decoding blocks for all three color components.

The independent tree structure supported by QTBT for I slices contains aspects related to chroma decoding. For example, JEM allows six (6) chroma modes per PU. The use of DM mode indicates that the video encoder 20 and/or video decoder 30 will use the same prediction mode for the corresponding luma PU for the chroma PU. As mentioned above, for I slices, the QTBT-based segmentation structures for luma blocks and their corresponding chroma blocks can be different. Therefore, when the DM mode is used in an I slice, the video encoder 20 and/or video decoder 30 can inherit the luma prediction mode covering the PU in the upper left position to perform prediction for the chroma PU. Compared to HEVC's segmentation technique (where luma blocks and their corresponding chroma blocks always share the same tree structure), JEM3.0's QTBT-based segmentation allows for possible differences between the luma tree structure and the chroma tree structure as shown in Figures 11A and 11B.

Figures 11A and 11B illustrate an example of the QTBT segmentation structure of a CTU in slice I. Figure 11A illustrates luma block 172, where the left segment 174 is indicated using upper and lower title marks. Figure 11B illustrates the corresponding chroma block 176, where the left segment 178 is indicated using upper and lower title marks. The corresponding left segments 174 and 178 contain finer segmentations, as shown in Figures 11A and 11B. L(i) (where “i” represents the corresponding integer value drawn within the corresponding segment) indicates that the luma intra-frame prediction mode used for the corresponding segment has an index equal to i. In the example illustrated in Figures 11A and 11B, video encoder 20 and/or video decoder 30 can encode/decode the left segment of chroma block 176 according to DM mode. Therefore, video encoder 20 and/or video decoder 30 can select LM mode to predict the left segment 178 of chroma block 176 according to the corresponding luma block segmentation in the upper left corner. In the usage scenarios illustrated in Figures 11A and 11B, the video encoder 20 and/or the video decoder 30 may select an intra-prediction mode with an index equal to 1 to encode/decode the left segment 178 of the chroma block 176, because "i" has a value of 1 in the upper left segment of the luma block 172.

Table 7 above specifies the pattern arrangement that the video encoder 20 can use to transmit chroma modes via signaling. To remove potential redundancy in chroma mode signaling when the derived mode (DM) refers to one of the always-present modes, the video encoder 20 can use an angled (66 when there are a total of 67 intra-frame modes) mode to replace the repeating modes shown in Table 7.1 below. In the use cases illustrated in Table 7.1 below, the angled mode (denoted as INTRA_ANGULAR66) is referred to as the "alternative mode".

Table 7.1 - Specifications of Chroma Intra-Frame Prediction Modes and Associated Names

As discussed above, the video encoder 20 and video decoder 30 can perform entropy decoding of the chroma prediction mode. In chroma mode decoding, the 1-b syntax element (0) is assigned to the most frequently occurring derived mode, the two bits (10) are assigned to the LM mode, and the 4-b syntax elements (1100, 1101, 1110, 1111) are assigned to the remaining four modes. The first two bits are decoded using a context model, and the remaining two bits (if needed) can be bypassed and decoded.

Table 7.2 - Binary strings for each chroma mode

The techniques of this invention relate to improving the performance of the various techniques discussed above. As mentioned above, JEM3.0 supports independent tree structures for chroma block segmentation and luma block segmentation for the same CTU. However, one chroma PU can correspond to multiple luma PUs. According to the QTBT segmentation aspect of JEM3.0, inheriting only one of the luma intra-prediction modes from multiple luma PUs used for chroma decoding can provide suboptimal results, which can be improved or possibly optimized by the various techniques of this invention. In addition, for a given PU in JEM, the total number of possible chroma modes is six (6). However, for luma decoding, the total number of possible modes is sixty-seven (67). The various techniques of this invention can improve decoding efficiency by increasing the total number of chroma modes.

Various techniques of the present invention are listed below by way of detailed enumeration. It should be understood that the video encoder 20 and/or video decoder 30 may employ the various techniques discussed below, individually or in various combinations of two or more of the described techniques. Although described as being performed by the video encoder 20 and/or video decoder 30, it should be understood that one or more components of the video encoder 20 illustrated in FIG. 2 and/or one or more components of the video decoder 30 illustrated in FIG. 3 may perform the various techniques of the present invention.

The following description represents the size of a chroma block as W*H (where "W" is the width of the chroma block and "H" is the height of the chroma block). The position of the top-left pixel in a chroma block relative to the entire slice is represented by a tuple (x, y), where "x" and "y" are the horizontal and vertical offsets, respectively. A luma block corresponding to a given chroma block has a size equal to 2W*2H (for a 4:2:0 color format) or W*H (for a 4:4:4 color format). The position of the top-left pixel in a corresponding luma block relative to the entire slice is represented by a tuple (2x, 2y) (for 4:2:0) or (x, y) (for 4:4:4). The examples given below are for the 4:2:0 color format. It should be understood that the techniques described herein can be extended to other color formats.

According to certain aspects of the invention, multiple DM modes can be added for chroma decoding, thereby increasing the number of available chroma encoding and decoding modes (from the luma block) for use by the video encoder 20 and video decoder 30. That is, according to these aspects of the invention, the video encoder 20 and video decoder 30 can have more DM options than a single option to inherit the decoding mode for the corresponding luma block. For example, according to the technique of the invention, the video encoder 20 and/or video decoder 30 can generate a candidate list containing DM intra-prediction modes for the chroma block based on the intra-prediction mode used in the corresponding luma block. While maintaining the same total number of possible chroma modes in the DM candidate list preserves decoding and bandwidth efficiency, the technique of the invention provides potential accuracy enhancement for the application of multiple DMs because the DMs offer better accuracy compared to the default modes used in the prior art.

In this example, the video encoder 20 can signal the chroma mode as currently described in JEM3.0. However, if the video encoder 20 selects the DM mode for chroma decoding of the chroma block, then the video encoder 20 can implement additional signal transmission. More specifically, according to this example, the video encoder 20 can encode and signal a flag indicating that the DM mode has been selected for encoding the chroma block. Based on the fact that the chroma block has been encoded in DM mode, the video encoder 20 can encode and signal an index value to indicate which mode in the candidate list is used as the DM mode. Based on the size of the candidate list, the video encoder 20 can encode and signal an index value between zero (0) and five (5). That is, the video encoder 20 can generate a candidate list of chroma prediction modes containing a total of six (6) candidates, resulting in a candidate list size of six (6).

Based on a received flag indicating that the encoded chroma block is encoded using a DM mode, the video decoder 30 can determine that the decoding mode for the chroma block is included in the candidate list. Subsequently, the video decoder 30 can receive and decode the index of the entry in the chroma mode candidate list. Based on the flag indicating that the encoded chroma block is encoded using a DM mode, and using the received index value for the encoded chroma block, the video decoder 30 can select a specific mode from the chroma mode candidate list for decoding the chroma block. In this way, in the example where a DM mode is selected for decoding the chroma block, the video encoder 20 and the video decoder 30 can increase the number of candidate modes available for encoding and decoding the chroma block. Based on the size of the candidate list, the video decoder 30 can decode index values between zero (0) and five (5). That is, the video decoder 30 can generate a candidate list of chroma prediction modes containing a total of six (6) candidates, resulting in a candidate list size of six (6).

In some instances, the video encoder 20 may first encode and signal a flag indicating whether the chroma block is encoded in a linear model (LM) mode. In these instances, the video encoder 20 may signal the flag (indicating whether the chroma block is LM encoded) followed by data indicating all DM candidates in the candidate list. According to this embodiment, the video decoder 30 may receive the encoded flag indicating whether the chroma block is LM encoded in the encoded video bitstream. The video decoder 30 may parse the data indicating all DM candidates in the candidate list from the position following the LM flag in the encoded video bitstream. Thus, it will be understood that, according to various embodiments of the invention, the video decoder 30 may construct a DM candidate list, or alternatively, may receive the entire DM candidate list in the encoded video bitstream. In either case, the video decoder 30 may use a signal-transmitted index to select the appropriate DM mode from the candidate list.

The video encoder 20 can also streamline the list of candidate DMs. Specifically, the video encoder 20 can determine whether two DMs included in the list are identical. If the video encoder 20 determines that multiple instances of a single DM (i.e., multiple identical DMs) are included in the candidate list, then the video encoder 20 can remove redundancy by removing all other instances of the same DM. In other words, the video encoder 20 can streamline the list so that only one instance of this identical DM remains in the candidate list.

In some embodiments of the DM candidate list-based technique of the present invention, the video encoder 20 may refine the DM candidates in the candidate list for one or more of the default modes. According to the refinement technique of the present invention, if the video encoder 20 determines that one of the default modes (e.g., the Kth mode in the default mode list) is the same as one of the DM modes in the DM candidate list, then the video encoder 20 may replace this DM mode in the candidate list with an alternative mode. In addition to replacing the refined DM mode in the candidate list, the video encoder 20 may also set the alternative mode to a mode with an index equal to ((maximum intra-frame mode index) - 1 - K). In some embodiments where the video encoder 20 signals data indicating all DM modes included in the candidate list, the video encoder 20 may signal data reflecting the refined DM candidate list.

In some instances where the video decoder 30 also performs DM candidate list construction, the video decoder 30 may also perform simplification to complete the DM candidate list. For example, if the video decoder 30 determines that one of the default modes (e.g., the Kth mode in the default mode list) is the same as one of the DM modes in the DM candidate list, then the video decoder 30 may replace this DM mode in the candidate list with an alternative mode. In addition to replacing the simplified DM mode in the candidate list, the video decoder 30 may also set the alternative mode to a mode with an index equal to ((maximum intra-frame mode index) - 1 - K).

By implementing one or more of the DM candidate list-based techniques described above, the video encoder 20 and video decoder 30 can increase the number of possible chroma prediction modes. The increased number of chroma modes obtainable via the DM candidate list-based techniques described above can improve decoding efficiency while maintaining accuracy. As mentioned above, in various instances, the video decoder 30 can receive the entire DM candidate list via the encoded video bitstream. Alternatively, the DM candidate list can be constructed and a prediction mode can be selected from the DM candidate list for a chroma block using the signal transmission index used. Because the video decoder 30 can receive an explicitly signaled DM candidate list, or alternatively construct a DM candidate list, the various DM candidate list-based techniques are described herein as being performed by the video encoder 20 and, where appropriate, by the video decoder 30.

In some instances, the video encoder 20 may fix the size of the DM candidate list (i.e., the total number of candidates included in the DM candidate list) within a specific range, such as within a pattern block, within a slice, within a picture, or within a sequence. In some such instances, if the video decoder 30 is configured to construct the DM candidate list and select candidates using the signal transmission index used, then the video decoder 30 may also fix the size of the DM candidate list (i.e., the total number of candidates included in the DM candidate list) within a specific range, such as within a pattern block, within a slice, within a picture, or within a sequence.

In some instances, the video encoder 20 may signal the size of the candidate list in a data structure containing metadata, which may be signaled out-of-band with respect to the corresponding encoded video data. As some non-limiting examples, the video encoder 20 may signal the size of the candidate list in any of the slice header, picture parameter set (PPS), or sequence parameter set (SPS). According to some instances, the video encoder 20 (and, where appropriate, the video decoder 30) may be configured to predefine the size of the candidate list such that the size of the candidate list is the same for all block sizes. Alternatively, the video encoder 20 (and, where appropriate, the video decoder 30) may be configured to predefine the size of the candidate list such that the size of the candidate list varies depending on the block size.

According to some examples, the video encoder 20 (and, where appropriate, the video decoder 30) may construct a DM candidate list to include (e.g., contain) up to three parts. In these examples, the three parts of the DM candidate list include each of the following: (i) a first part containing candidates for the lumen intra-prediction mode associated with a specific position relative to the corresponding lumen block; (ii) a second part containing candidates derived from a function of all lumen blocks within the corresponding lumen block, such as the most commonly used lumen intra-prediction mode as described in one of the examples above; and (iii) a third part containing candidates derived from a selected lumen intra-prediction mode with a specific offset of a mode index.

In one instance, the video encoder 20 (and, if applicable, the video decoder 30) may sequentially insert candidates from the first two parts into the DM candidate list until the total number of candidates equals the predefined list size (i.e., the predefined total number of DM modes). After a simplification process is performed on the modes included in the DM candidate list, if the size of the candidate list is still less than the predefined total number of DM modes, then the video encoder 20 (and, if applicable, the video decoder 30) may insert candidates from the third part of the list. In one such instance, the video encoder 20 (and, if applicable, the video decoder 30) may insert candidates from three parts (or two parts, depending on the result of simplification) into the candidate list in the order of the first part, followed by the second part, followed by the third part. In another alternative instance, the video encoder 20 (and, if applicable, the video decoder 30) may insert candidates from the second part before candidates from the first part. In yet another alternative instance, the video encoder 20 (and, where appropriate, the video decoder 30) may insert candidates from the second part into the candidates from the first part (e.g., by interleaving or interlacing the candidates from the first and second parts).

In some instances, the first part of the DM candidate list consists of patterns inherited from specific positions used for decoding the corresponding luma blocks. For example, the first part of the candidate list may contain patterns inherited from the following positions within the corresponding luma block: center, upper left, upper right, lower left, and lower right. That is, in this instance, the first part of the candidate list may contain patterns inherited from the four corners of the corresponding luma block. In one such instance, the video encoder 20 (and, where appropriate, the video decoder 30) may insert the patterns inherited from the four corner positions of the corresponding luma block into the DM candidate list in the following order: center, upper left, upper right, lower left, and lower right. In another such instance, the video encoder 20 (and, where appropriate, the video decoder 30) may insert the patterns inherited from the four corner positions of the corresponding luma block into the DM candidate list in the following order: center, upper left, lower right, lower left, and upper right. In other instances, the order may vary, and it should be understood that the order described above is a non-limiting example.

In one instance, the video encoder 20 (and, if applicable, the video decoder 30) may form a first part of the DM candidate list to include intra-prediction modes for all positions of the corresponding luma block. In this instance, a second part may be optional because the first part contains all intra-prediction modes for the corresponding luma block. Alternatively, the video encoder 20 (and, if applicable, the video decoder 30) may traverse all units within the corresponding luma block in a certain order. Alternatively or additionally, the video encoder 20 (and, if applicable, the video decoder 30) may add additional modes to the DM candidate list in a decreasing order based on their occurrence within the corresponding luma block.

In one instance, the video encoder 20 (and, if applicable, the video decoder 30) may apply an offset to one or more candidates already inserted into the list in order to form a third portion. Additionally, when forming the third portion, the video encoder 20 (and, if applicable, the video decoder 30) may further apply or perform a reduction of the inserted candidates. In an alternative instance, the video encoder 20 (and, if applicable, the video decoder 30) may form the third portion to include one or more intra-frame chroma modes from adjacent blocks.

According to some embodiments of the technology described herein, the video encoder 20 (and, depending, the video decoder 30) can adaptively change the size of the candidate list from CU to CU, from PU to PU, or from TU to TU. In one instance, the video encoder 20 (and, depending, the video decoder 30) can add only candidates from the first part, as described in the embodiments relating to the formation of a three-part DM candidate list. Alternatively, the video encoder 20 (and, depending, the video decoder 30) can add only candidates from the first and second parts to the DM candidate list. In some instances, the video encoder 20 (and, depending, the video decoder 30) can perform simplification to remove identical intra-prediction modes. III.

In an instance where the video encoder 20 reduces the DM candidate list, if the number of candidates in the final reduced DM candidate list is equal to 1, then the video encoder 20 may not need to signal the DM index. In some instances, the video encoder 20 (and, where appropriate, the video decoder 30) may use truncated unary binary representation to binaryize the DM index values in the DM candidate list. Alternatively, the video encoder 20 (and, where appropriate, the video decoder 30) may use unary binary representation to binaryize the DM index values in the DM candidate list.

In some instances, the video encoder 20 (and, depending on the case, the video decoder 30) may set the context model index to equal the binary index. Alternatively, the total number of context models used to decode the DM index value may be less than the maximum number of candidates. In this case, the video encoder 20 may set the context model index to equal min(K, binary index), where K represents a positive integer. Alternatively, the video encoder 20 may use the context model to encode only the first few bits and may use it in bypass mode to encode the remaining bits. In this instance, the video decoder 30 may use the context model to decode only the first few bits and may use it in bypass mode to decode the remaining bits.

Alternatively, the video encoder 20 (and, where appropriate, the video decoder 30) may determine the number of context-decoded bits based on one or more of the total number of DM candidates or the sizes of CU, PU, or TU. Alternatively, for the first M bits (e.g., M equals 1), the context modeling may further depend on the total number of DM candidates or the sizes of CU/PU/TU or the splitting information of the corresponding luma blocks in the final (e.g., reduced) DM candidate list.

In some instances, the video encoder 20 (and, where appropriate, the video decoder 30) may further reorder candidates in the candidate list before binaryization. In one instance, when the width of the CU/PU/TU is greater than the height of the CU/PU/TU, the reordering may be based on the difference in intra-prediction mode index between the actual intra-frame mode used for a candidate and the horizontal intra-prediction mode. The smaller the difference, the smaller the index of the candidate to be assigned to the DM candidate list. In another instance, when the height of the CU/PU/TU is greater than the width of the CU/PU/TU, the reordering may be based on the difference in intra-prediction mode index between the actual intra-frame mode used for a candidate and the vertical intra-prediction mode. Also in this instance, the smaller the difference, the smaller the index to be assigned to the candidate in the DM candidate list.

Alternatively, the video encoder 20 (and, where appropriate, the video decoder 30) may perform a reduction of all DM candidates in the list relative to the default mode. If one of the default modes (e.g., the Kth mode in the default mode list) is the same as one of the DM modes in the DM candidate list, then the video encoder 20 (and, where appropriate, the video decoder 30) may replace this DM mode in the DM candidate list with an alternative mode. In addition to replacing the reduced DM modes in the candidate list, the video encoder 20 (and, where appropriate, the video decoder 30) may also set the alternative mode to a mode with an index equal to ((maximum intra-frame mode index) - 1 - K).

According to some techniques of the present invention, the video encoder 20 and the video decoder 30 can unify the intra-frame prediction modes for luma and chroma. That is, for each chroma block, in addition to the linear model (LM) mode and other modes specific to the decoded chroma components, the video encoder 20 and/or the video decoder 30 can also select a prediction mode from a pool of available luma prediction modes. The pool of available luma prediction modes is described herein as containing a total of “N” prediction modes, where “N” represents a positive integer value. In some instances, the value of “N” is equal to sixty-seven (67), corresponding to 67 different available luma prediction modes.

In addition, regarding the encoding and signal transmission of chroma intra-prediction modes, the video encoder 20 can also signal a Most Probable Mode (MPM) flag and, depending on the value of the MPM flag, signal an MPM index (corresponding to the index of an MPM candidate in the MPM candidate list). For example, the video encoder 20 can construct an MPM candidate list by first adding one or more DM modes for the chroma block to the MPM candidate list. As mentioned above, the video encoder 20 can identify multiple DM modes for the chroma block. However, it should be understood that in some situations, the video encoder 20 can identify a single DM mode for the chroma block. After adding a DM mode to the MPM candidate list, the video encoder 20 can add other chroma modes from adjacent blocks to the MPM candidate list. Alternatively or additionally, the video encoder 20 may add a default mode, for example, by using the luminance MPM candidate list construction process described in “Neighbor based intra mostprobable modes list derivation” (JVET-C0055, Geneva, May 2016) by V. Seregin, X. Zhao, A. Said, M. Karczewicz (hereinafter, “Seregin”).

Alternatively, the video encoder 20 may construct a chroma-specific MPM candidate list in the same manner as for a luma-specific MPM candidate list. For example, the video encoder 20 may examine several adjacent blocks in the order described in Seregin. In these embodiments, the video encoder 20 may process LM mode and/or other chroma-specific intra-prediction modes in the same manner as it processes other intra-prediction modes. Furthermore, the video encoder 20 may streamline the MPM candidate list to remove redundancy resulting from the addition of the same intra-prediction mode from multiple sources.

In one example, the video encoder 20 may first signal a flag to indicate the use of one or more chroma-specific modes applicable only to the chroma components (e.g., the LM mode and/or other prediction modes used only for decoding chroma components). If the selected prediction mode is not a chroma-specific mode (i.e., the video encoder 20 sets the aforementioned flag to a disabled state), then the video encoder 20 may further signal an MPM flag. In this example implementation, when adding a chroma prediction mode inherited from a neighboring block to the MPM list, the video encoder 20 may disregard chroma-specific modes (e.g., the LM mode) if the chroma-specific mode is taken from a neighboring block.

Example use cases of this implementation are described below. Video encoder 20 can use LM mode to intra-frame predict chroma blocks, and therefore can signal an LM flag set to the enabled state. Based on the chroma block being encoded using the LM prediction mode, video encoder 20 can signal an MPM index indicating the position of the chroma block within the MPM candidate list. This example use case illustrates that video encoder 20 can use a one-bit flag to first provide video decoder 30 with an indication of whether the prediction mode for the chroma block is a candidate in the MPM candidate list. Only if the prediction mode for the chroma block is a candidate from the MPM candidate list can video encoder 20 signal an index to indicate to video decoder 30 which mode in the MPM candidate list is used to predict the chroma block. In this way, video encoder 20 can save bandwidth by first using a one-bit flag and then determining whether to signal an index value based on the flag value.

The decoder aspect of the above-described technology is discussed below. The video decoder 30 can receive an MPM flag in the encoded video bitstream. If the MPM flag value is set to the enabled state, the video decoder 30 can also receive an MPM index for the relevant chroma block, which corresponds to the index of a specific MPM candidate in the MPM candidate list. For example, the video decoder 30 can construct the MPM candidate list by first adding one or more DM modes for the chroma block to the MPM candidate list. As mentioned above, the video decoder 30 can recognize multiple DM modes for the reconstruction of the chroma block. However, it should be understood that in some situations, the video decoder 30 can recognize a single DM mode for the chroma block. After adding a DM mode to the MPM candidate list, the video decoder 30 can add other chroma modes from adjacent blocks to the MPM candidate list. Alternatively or additionally, the video decoder 30 can add a default mode, for example, by using the luma MPM candidate list construction process described in Seregin.

Alternatively, the video decoder 30 may construct a chroma-specific MPM candidate list in the same manner as for a luma-specific MPM candidate list. For example, the video decoder 30 may examine several adjacent blocks in the order described in Seregin. In these embodiments, the video decoder 30 may process LM mode and/or other chroma-specific intra-prediction modes in the same manner as it processes other intra-prediction modes. Furthermore, the video decoder 30 may streamline the MPM candidate list to remove redundancy resulting from the addition of the same intra-prediction mode from multiple sources.

In one example, the video encoder 20 may first signal a flag to indicate the use of one or more chroma-specific modes applicable only to the chroma components (e.g., the LM mode and/or other prediction modes used only for decoding chroma components). If the selected prediction mode is not a chroma-specific mode (i.e., the video decoder 30 determines that the flag is set to a disabled state), then the video decoder 30 may further receive an MPM flag. In this example implementation, when adding a chroma prediction mode inherited from a neighboring block to the MPM list, the video decoder 30 may disregard chroma-specific modes (e.g., the LM mode) if the chroma-specific mode is taken from a neighboring block.

Example use cases of this implementation are described below. Video decoder 30 can receive an LM flag set to the enabled state and can therefore reconstruct the chroma block using LM mode intra-frame prediction. Based on the chroma block being encoded using an LM prediction mode, video decoder 30 can receive an MPM index indicating its position within the MPM candidate list for the chroma block. This example use case illustrates that video decoder 30 can use a one-bit flag to first determine whether the prediction mode for the chroma block is actually a candidate in the MPM candidate list. If the prediction mode is not a candidate from the MPM candidate list, then video decoder 30 avoids needing video encoder 20 to signal an indication of which mode in the MPM candidate list is used to predict the chroma block. In this way, video decoder 30 can save bandwidth by reducing the number of instances where video encoder 20 needs to signal the index value, which is more bandwidth-intensive than signaling a one-bit flag.

In some instances, in addition to the LM mode, the video encoder 20 and/or video decoder 30 may add other chroma-specific or chroma-specific intra-prediction modes to the MPM list, and add the remaining intra-prediction modes as the default modes of the list. Alternatively, the video encoder 20 may first signal an MPM flag, and when constructing the MPM list, the video encoder 20 and/or video decoder 30 may always consider the chroma prediction modes of adjacent blocks, regardless of whether the adjacent blocks are predicted using the LM mode. In another instance, if the LM mode is not added to the MPM list, the video encoder 20 and/or video decoder 30 may add the LM mode as the first default mode. In another instance, the video encoder 20 and/or video decoder 30 may use only the LM and modes from the MPM candidate list, and may remove the default mode along with it. In some instances, the video encoder 20 (and, where appropriate, the video decoder 30) may add an existing default mode only if the total number of added default modes is less than a predetermined integer value represented by "K". In one such instance, K is set to the value four (4).

In some instances, when only one DM is allowed, instead of obtaining the intra-prediction mode of luminance from the upper left corner of the corresponding luminance block, the video encoder 20 and/or video decoder 30 may use one or more of the following rules to select the luminance intra-prediction mode as the DM mode. In one instance of this rule, the luminance intra-prediction mode is the most frequently used mode within the corresponding luminance block. In one instance, based on a certain scan order, the video encoder 20 and/or video decoder 30 may traverse the intra-prediction modes of each unit within the corresponding luminance block and record the occurrence count of the existing luminance prediction mode. The video encoder 20 and/or video decoder 30 may select the mode with the highest occurrence count. That is, the video encoder 20 and/or video decoder 30 may select the luminance intra-prediction mode that covers the largest size (i.e., area) of the corresponding luminance block. When two prediction modes have the same usage in the corresponding luminance block, the video encoder 20 and/or video decoder 30 may select the prediction mode detected first based on the scan order. Here, a unit is defined as the minimum PU/TU size for luminance/chrominance intra-prediction. In some instances, the scanning order can be a raster/Z-shaped/diagonal/Z-shaped scanning order or a decoding order.

Alternatively, the video encoder 20 and/or video decoder 30 may begin scanning from the center of the luma block and traverse to the boundary in a certain order. Alternatively or additionally, the scan/unit may depend on the PU/TU size. Alternatively, based on a certain scan order, the video encoder 20 and/or video decoder 30 may traverse the intra-frame prediction modes of each PU/TU/CU within the corresponding luma block and record the number of occurrences of the recorded existing luma prediction modes. The video encoder 20 and/or video decoder 30 may select the mode with the maximum number of occurrences. When two modes have the same usage in the luma block, the video encoder 20 and/or video decoder 30 may select the prediction mode that appears first (i.e., is detected first) based on the scan order. In some instances, the scan order may be a raster/Z-shaped/diagonal/Z-shaped scan order or a decoding order. Alternatively, the scan may depend on the PU/TU size.

In another alternative example, for the instance described above with respect to a single permitted DM mode, if the video encoder 20 and/or video decoder 30 determine that two or more modes have an equal number of occurrences in the corresponding luma block, then the video encoder 20 and/or video decoder 30 may select one of the modes that have an equal number of occurrences in the luma block. This selection may depend on the mode index and/or PU/TU size of these multiple luma modes. Alternatively, for a specific block size, such as a block size greater than 32×32, the video encoder 20 and/or video decoder 30 may evaluate only a portion (e.g., a subset) of the luma intra-prediction modes for the corresponding luma block according to this rule based on a single DM.

As another example of this rule regarding a single DM mode scenario, the video encoder 20 and/or video decoder 30 may select a luma intra-frame prediction mode associated with the center position of the corresponding luma block. In one instance, the video encoder 20 and/or video decoder 30 may define the center position based on a coordinate tuple (2x+W-1, 2y+H-1) for the 4:2:0 color format. Alternatively, the video encoder 20 and/or video decoder 30 may define the center position as follows:

- If both W and H are equal to 2, then the video encoder 20 and/or the video decoder 30 can use position (2x, 2y) as the center position.

- Otherwise, if H equals 2, then the video encoder 20 and/or the video decoder 30 can use the position (2x+(2*W/4/2-1)*4,2y) as the center position.

Otherwise, if W equals 2, then the video encoder 20 and/or the video decoder 30 can use the position (2x, 2y + (2*H/4/2-1)*4) as the center position.

Otherwise (for example, if H and W are not equal to 4), then use (2x+(2*W/4/2-1)*4,2y+(2*H/4/2-1)*4) as the center position.

According to some embodiments of the technology of the present invention, instead of using the same default mode for all blocks, the video encoder 20 and/or video decoder 30 may treat the mode derived from the corresponding luma block as the default mode. In one embodiment, the total number of default modes is increased to include more modes derived from the corresponding luma block. In another embodiment, when the total number of added default modes is less than K (in a non-limiting embodiment, K is set to 4), only existing default modes are added.

Figures 12A and 12B illustrate adjacent block selection for adaptive sorting of chroma prediction modes according to one or more aspects of the present invention. In some embodiments of the technology according to the present invention, the video encoder 20 and/or video decoder 30 may apply adaptive sorting of chroma modes such that the order depends on the chroma mode of adjacent blocks. In one embodiment, the video encoder 20 and/or video decoder 30 applies adaptive sorting only to specific modes, such as DM mode and/or LM mode. In another embodiment, there are five adjacent blocks, as depicted in Figure 12A. Alternatively, the video encoder 20 and/or video decoder 30 may use only two adjacent blocks, for example, A1 and B1 shown in Figure 12A, or the upper block (A) and left block (L) shown in Figure 12B. In one embodiment, when all available adjacent intra-decoded blocks are decoded in LM mode, the video encoder 20 and/or video decoder 30 may place LM mode before DM mode. Alternatively, when at least one of the adjacent intra-frame decoded blocks is decoded in LM mode, the video encoder 20 and/or video decoder 30 may place the LM mode before the DM mode.

According to some embodiments of the present invention, the video encoder 20 and/or the video decoder 30 may use luma information to reorder chroma syntax values before entropy decoding. In one embodiment, the NSST index of the luma block may be used to update the decoding order of the chroma NSST index. In this case, the video encoder 20 and/or the video decoder 30 may first encode/decode a binary indicating whether the index of the chroma block is the same as the NSST index of the corresponding luma block. In another embodiment, the video encoder 20 and/or the video decoder 30 may use the adaptive multiple transform (AMT) index of the luma block to update the decoding order of the chroma AMT index. In this case, the video encoder 20 and/or the video decoder 30 may first encode/decode a binary indicating whether the index of the chroma block is the same as the AMT index of the corresponding luma block. The video encoder 20 and/or the video decoder 30 may use another (e.g., similar) approach for any other syntax, wherein the method may be applicable to both luma and chroma components, and the index/pattern may be different for the luma and chroma components.

According to some embodiments of the invention, the video encoder 20 and/or the video decoder 30 can derive multiple sets of LM parameters for a chroma block, such that the derive is based on the intra-prediction mode of the corresponding luma block. In one embodiment, the video encoder 20 and/or the video decoder 30 can derive at most K sets of parameters, for example, where “K” represents an integer value. In one embodiment, “K” is set to a value of two (2). In another embodiment, the video encoder 20 and/or the video decoder 30 can classify adjacent luma/chroma samples into K sets based on the intra-prediction mode of samples located in the corresponding luma block. The video encoder 20 and/or the video decoder 30 can classify luma samples within the corresponding luma block into K sets based on the intra-prediction mode of samples located in the corresponding luma block. In another embodiment, when two intra-prediction modes are considered to be “far apart,” for example, when the absolute value of the mode index is greater than a threshold, the video encoder 20 and/or the video decoder 30 can treat corresponding sub-blocks and adjacent samples as using different parameters.

According to some embodiments of the present invention, the video encoder 20 and/or the video decoder 30 may use a composite DM mode to encode/decode the current chroma block. According to the composite DM mode of the present invention, the video encoder 20 may generate a prediction block using a weighted sum of prediction blocks generated from two or more identified intra-prediction modes. The video encoder 20 may identify two or more intra-prediction modes for encoding co-located luma blocks, for encoding adjacent chroma blocks, or for encoding adjacent blocks of corresponding luma blocks. The video encoder may then generate a prediction block for each of the identified intra-prediction modes and may derive a weighted sum of the two or more generated prediction blocks as the prediction block for this composite DM mode.

In one instance, the weights of the prediction blocks used to generate this composite DM mode depend on the area size of each identified intra-prediction mode applied to the corresponding luma block. Alternatively, the weights of the prediction blocks for each identified intra-prediction mode may depend on the position of the current pixel and whether the current identified intra-prediction mode covers the current pixel. In another alternative, the weights are the same for each identified intra-prediction mode. In yet another alternative, the video encoder 20 and/or the video decoder 30 may still utilize a predefined set of weights. In yet another alternative, or additionally, the video encoder 20 may signal an index of the weights for each CTU/CU/PU/TU. When a default mode (such as the non-DM and non-LM modes shown in Table 7.1) is signaled, if the default mode has already been identified for generating the composite DM mode, the video encoder 20 may replace the default mode with other intra-prediction modes that have not been identified for generating the composite DM mode.

Figures 13A and 13B are conceptual diagrams illustrating an example of how video encoder 20 and video decoder 30 can be used to select block positions of chroma intra-prediction modes according to the multiple DM mode selection technique described above. An example implementation of multiple DM mode selection for chroma decoding is described below. As described above, according to aspects of the invention, video encoder 20 (and, where appropriate, video decoder 30) can perform DM mode selection. That is, in some instances, video encoder 20 can explicitly transmit a list of DM candidates by signaling, thereby eliminating the need for video decoder 30 to also form a list of DM candidates. In other instances, video encoder 20 can transmit only the index of the selected candidate from the list of DM candidates by signaling, thereby enabling video decoder 30 to select a candidate from a list of DM candidates also formed by video decoder 30.

Figure 13A illustrates the prediction modes used in the sub-blocks of the luma component (luma block 202). Figure 13B illustrates the luma mode inheritance for chroma block 204 according to HEVC technology. As shown, according to HEVC technology, the prediction mode (i.e., mode L(1)) from the upper left sub-block of luma block 202 is inherited relative to the left region of chroma block 204. As shown in Figure 13A, (e.g., by video encoder 20, and, where applicable, video decoder 30) the luma modes for the sub-blocks located at the center (C0), upper left (TL), upper right (TR), lower left (BL), and lower right (BR) are obtained. The modes are represented by the acronyms DMC, DMTL, DMTR, DMBL, and DMBR. In some alternatives, video encoder 20 (and, where applicable, video decoder 30) may replace the C0 selection with the selection of the mode used at positions C1 and/or C2 and/or C3. Additionally, the video encoder 20 (and, if applicable, the video decoder 30) can add a luminance mode covering most of the area of luminance block 202 as an additional DM mode to the DM candidate list. The luminance mode covering the largest area of luminance block 202 is represented by the acronym "DMM".

The video encoder 20 (and, where appropriate, the video decoder 30) may use one or more techniques discussed below to construct the DM candidate list. Several candidates (denoted by “N”) from the candidate group, including DMC, DMTL, DMTR, DMBL, and DMBL, may be added to the DM candidate list in a predetermined order. In one instance, “N” is set to six (6) and the order may be as follows: DMC, DMM, DMTL, DMTR, DMBL, DMBR. In an alternative example, “N” is set to five (5) and the order may be as follows: DMC, DMTL, DMTR, DMBL, DMBR. When forming the candidate list, before adding each such candidate to the DM candidate list, the video encoder 20 (and, where appropriate, the video decoder 30) may reduce each candidate relative to all candidates or a subset (e.g., a proper subset) of previously added candidates. While two example orders have been discussed above, it should be understood that, according to aspects of the invention, the video encoder 20 (and, where appropriate, the video decoder 30) may also use various other orders. Assuming the total number of DM modes in the candidate list is “M” (where “M” is a positive integer), and the total number of default modes is represented by “F”, then a specific candidate in the DM candidate list is represented by DM_{_i}. In this notation, the subscript “i” represents an integer value ranging from 0 to M-1.

The video encoder 20 (and, if applicable, the video decoder 30) can use applied simplification in both DM candidates and default modes. That is, when forming the list of DM candidates, the video encoder 20 (and, if applicable, the video decoder 30) can simplify the DM candidates relative to the default modes. In an alternative example, for each DM _{_i}, the video encoder 20 (and, if applicable, the video decoder 30) can compare DM _{_i} with each of the default modes. If any default mode is found to be identical to DM_{_i}, then the video encoder 20 (and, if applicable, the video decoder 30) can replace the first such default mode (which was found to be identical to DM_{_i}) with an alternative mode. For example, the video encoder 20 (and, if applicable, the video decoder 30) can replace the simplified default mode with a mode having an index value equal to (K-1-i), where “K” is the total number of luminance prediction modes for the corresponding luma block. Example pseudocode for these operations is given below:

For example, the default modes could be: Mode 0 (planar), Mode 50 (vertical), Mode 18 (horizontal), and Mode 1 (DC), and the DM candidate list is {Mode 0, Mode 63, Mode 50, Mode 1}. After the simplification process, the default modes are replaced by the following set: {Mode 66, Mode 64, Mode 18, Mode 63}. In another alternative example, the video encoder 20 (and, if applicable, the video decoder 30) can apply full simplification, where each default mode is simplified relative to all DM modes. That is, for each default mode, the default mode is compared with all DM modes. If a step-by-step comparison indicates that one of the DM modes is the same as the default mode currently under examination, then the default mode is replaced by the last non-DM mode. Example pseudocode for this instance is given below:

The video encoder 20 can implement various aspects of the present invention based on multiple DM modes to implement the signal transmission of chroma modes. The video encoder 20 can encode chroma modes according to a process comprising the following parts: As part, the video encoder 20 can encode and signal a one-bit flag to indicate the use of only any of the prediction modes applicable to the chroma components (e.g., LM, which is specific to chroma encoding). If the chroma block is encoded according to this chroma-specific mode (thereby causing the video encoder 20 to set the flag to an enabled state), then the video encoder 20 can additionally encode and signal an index of the specific mode.

Additionally, the video encoder 20 can encode and signal a flag to indicate the use of a mode derived from the corresponding luma block. That is, if the video encoder 20 selects a prediction mode for encoding the chroma block based on the prediction mode used for the corresponding luma block, then the video encoder 20 can set the flag to an enabled state. Subsequently, if the chroma block is encoded using a prediction mode inherited from the corresponding luma block, then the video encoder 20 can additionally encode and signal an index of the mode selected from the corresponding luma block.

If the video encoder 20 determines that the chroma block is encoded neither according to a chroma-specific prediction mode nor according to a prediction mode derived from the luma block, then the video encoder 20 can encode and signal information identifying the remaining mode. The video encoder 20 can implement some/options of the chroma encoding listed above in different orders. Examples of different orders are given in Tables 7.3 and 7.4 or Table 8 below.

Table 7.3 - Specifications of Chroma Intra-Frame Prediction Modes and Associated Names

Table 7.4 - Binary strings for each chroma mode

Table 8 - Binary strings for each chroma mode

As described above, aspects of the present invention relate to the unification of luma and chroma modes. Example embodiments of the unification of luma and chroma modes are described below. The total allowed number of most probable mode (MPM) candidates is denoted by N _mpm below. The video encoder 20 and/or video decoder 30 can construct a mode list of chroma intra-frame modes to include the following portions:

-LM mode; and

-MPM mode.

The MPM mode portion may include a DM candidate list and a chroma mode portion. The video encoder 20 (and, if applicable, the video decoder 30) may use the same techniques described above regarding multiple DM modes to form the DM candidate list portion, which represents a unified candidate list. Regarding the chroma mode portion of the MPM mode, the video encoder 20 (and, if applicable, the video decoder 30) may derive the chroma mode from adjacent blocks of the currently decoded chroma block. For example, to derive the chroma mode from adjacent blocks, the video encoder 20 (and, if applicable, the video decoder 30) may further utilize the MPM construction process used for the luma mode. If the total number of MPM candidates is still less than N _mpm after performing the list construction process described above, then the video encoder 20 (and, if applicable, the video decoder 30) may perform various steps according to JVET-C0055 referenced above.

For example, if the total number of MPM candidates is less than N _mpm after performing the list construction process described above, then the video encoder 20 (and, depending on the case, the video decoder 30) can add the following modes: left (L), top (A), plane, DC, bottom left (BL), top right (AR), and top left (AL). If the MPM candidate list is still incomplete (i.e., if the total number of MPM candidates is less than N _mpm ), then the video encoder 20 (and, depending on the case, the video decoder 30) can add -1 and +1 to the already included angle modes. If the MPM list is still incomplete (i.e., the total number of MPM candidates is less than N _mpm ), then the video encoder 20 (and, depending on the case, the video decoder 30) can add default modes, namely, vertical, horizontal, 2, and diagonal modes.

The non-MPM modes recognizable by video encoder 20 and/or video decoder 30 include any remaining intra-predictive modes not included in the MPM candidate list construction process described above. The difference from the luma-based MPM list construction process described above (e.g., in the section referencing JVET-C0055) is that when a candidate is added, the added candidate is not an LM mode. Alternatively or additionally, planar and DC modes may be added after all spatially adjacent modes. Alternatively, video encoder 20 and/or video decoder 30 may implement one or more other MPM list construction techniques to replace the technique of JVET-C0055.

Regarding the unification of luminance and chrominance modes, the video encoder 20 can implement various chrominance mode signal transmission techniques of the present invention. The video encoder 20 can encode chrominance modes according to a process comprising the following parts: As part, the video encoder 20 can encode and signal a flag to indicate the use of only any of the predictive modes applicable to the chrominance components (e.g., LM, which is specific to chrominance encoding). If the chrominance block is encoded according to this chrominance-specific mode (thereby causing the video encoder 20 to set the flag to an enabled state), then the video encoder 20 can additionally encode and signal an index of the specific mode.

Additionally, the video encoder 20 can encode and signal a flag to indicate the use of a mode included in the MPM candidate list. That is, if the video encoder 20 selects a predictive mode for encoding chroma blocks, and the selected predictive mode is included in the MPM candidate list, then the video encoder 20 can set the flag to an enabled state. Subsequently, if the chroma blocks are encoded using a predictive mode included in the MPM candidate list, then the video encoder 20 can additionally encode and signal an index of the mode, indicating the mode's position in the MPM candidate list.

If the video encoder 20 determines that the chroma block is not encoded according to either a chroma-specific prediction mode or a prediction mode included in the MPM candidate list, then the video encoder 20 can encode and signal information identifying the remaining mode. The video encoder 20 can implement some/options of the chroma encoding listed above in different orders. Examples of different orders are given in Table 8.1 or Table 9 below.

Table 8.1 - Binary strings for each chroma mode

If the chroma intra-frame mode list only contains the LM and MPM portions (similar to the luma MPM, which contains multiple DM modes and modes from spatial neighbors), then the video encoder 20 can implement the chroma mode signal transmission in another modified manner, as shown in Table 9 below:

Table 9

In another alternative, the video encoder 20 (and, if applicable, the video decoder 30) may always add a default mode (e.g., planar, DC, horizontal, vertical mode) to the MPM candidate list. In one instance, N _mpm candidates for the MPM candidate list may be constructed first using one or more of the techniques described above. Then, the missing modes of the default mode may replace the last one or more MPM candidates.

Figure 14 is a flowchart illustrating an example process 220 executable by the processing circuitry of the video decoder 30 according to an aspect of the present invention. Process 220 may begin when the processing circuitry of the video decoder 30 performs the following operations: determining a plurality of derived modes (DMs) that can be used to predict luminance blocks of video data, and also to predict chrominance blocks of video data, the chrominance blocks corresponding to the luminance blocks (222). The video decoder 30 may form a candidate list of prediction modes for the chrominance blocks, the candidate list containing one or more DMs that can be used to predict the chrominance blocks (224). In some non-limiting embodiments, the processing circuitry of the video decoder 30 may receive data in the encoded video bitstream indicating each corresponding DM in the candidate list, and reconstruct the received data indicating each corresponding DM in the candidate list, thereby forming the candidate list. In other embodiments, the processing circuitry of the video decoder 30 may construct the candidate list.

The processing circuitry of video decoder 30 can determine to decode the chroma block using any of one or more DMs from the candidate list (226). In some non-limiting instances, the processing circuitry of video decoder 30 can receive a one-bit flag in the encoded video bitstream indicating that the chroma block is encoded using one of the DMs. Based on the determination to decode the chroma block using any of one or more DMs from the candidate list, the processing circuitry of video decoder 30 can decode an indication (228) identifying the selected DM from the candidate list to be used for decoding the chroma block. For example, the processing circuitry of video decoder 30 can reconstruct data (received in the encoded video bitstream) indicating the index value of the selected DM in the candidate list. Subsequently, the processing circuitry of video decoder 30 can decode the chroma block according to the selected DM (230). In various instances, video data containing the luma block and the chroma block can be stored in the memory of video decoder 30.

In some instances, one or more DMs included in the candidate list may include one or more of the following: a first prediction mode associated with the center position of the corresponding luma block; a second prediction mode associated with the upper left position of the corresponding luma block; a third prediction mode associated with the upper right position of the corresponding luma block; a fourth prediction mode associated with the lower left position of the corresponding luma block; or a fifth prediction mode associated with the lower right position of the corresponding luma block. In some instances, the candidate list may further include one or more chroma intra-frame prediction modes different from each of the one or more DMs. In some such instances, each of the chroma intra-frame prediction modes corresponds to a mode used to predict adjacent chroma blocks of the chroma block. In some instances, at least one corresponding chroma intra-frame prediction mode in the candidate list is a chroma-specific prediction mode used only for predicting chroma data.

Figure 15 is a flowchart illustrating an example process 240 executable by the processing circuitry of the video encoder 20 according to an aspect of the present invention. Process 240 may begin when the processing circuitry of the video encoder 20 performs the following operations: determining a plurality of derived modes (DMs) that can be used to predict luminance blocks of video data, and also to predict chrominance blocks of video data, the chrominance blocks corresponding to the luminance blocks (242). In various instances, video data containing luminance and chrominance blocks may be stored in the memory of the video encoder 20. The video encoder 20 may form a candidate list of prediction modes for chrominance blocks, the candidate list containing one or more DMs (244) among a plurality of DMs that can be used to predict chrominance blocks.

The processing circuitry of video encoder 20 can determine to encode a chroma block using any DM from one or more DMs in a candidate list (246). Based on the determination to encode the chroma block using any DM from one or more DMs in the candidate list, the processing circuitry of video encoder 20 can encode an indication (248) identifying a selected DM from the candidate list to be used for decoding the chroma block. For example, the processing circuitry of video encoder 20 can encode data indicating the index value of the position of the selected DM in the candidate list and signal the encoded data in the encoded video bitstream. Subsequently, the processing circuitry of video encoder 20 can encode the chroma block according to the selected DM (250). In some instances, the processing circuitry of video encoder 20 can signal a one-bit flag in the encoded video bitstream indicating whether the chroma block is encoded using a linear model (LM) mode. In these instances, the processing circuitry of video encoder 20 can signal data in the encoded video bitstream indicating each corresponding DM in one or more DMs in the candidate list.

In some instances, one or more DMs included in the candidate list may include one or more of the following: a first prediction mode associated with the center position of the corresponding luma block; a second prediction mode associated with the upper left position of the corresponding luma block; a third prediction mode associated with the upper right position of the corresponding luma block; a fourth prediction mode associated with the lower left position of the corresponding luma block; or a fifth prediction mode associated with the lower right position of the corresponding luma block. In some instances, the candidate list may further include one or more chroma intra-frame prediction modes different from each of the one or more DMs. In some such instances, each of the chroma intra-frame prediction modes corresponds to a mode used to predict adjacent chroma blocks of the chroma block. In some instances, at least one corresponding chroma intra-frame prediction mode in the candidate list is a chroma-specific prediction mode used only for predicting chroma data. In some instances, the processing circuitry of the video encoder 20 may determine that at least two of the one or more DMs are identical, and may include only one of the at least two identical DMs in the candidate list.

Figure 16 is a flowchart illustrating an example process 260 executable by the processing circuitry of the video decoder 30 according to an aspect of the present invention. Process 260 may begin when the processing circuitry of the video decoder 30 performs the following operation: forming a most probable mode (MPM) candidate list for chroma blocks of video data stored in the memory of the video decoder 30, such that the MPM candidate list contains one or more derived modes (DMs) associated with luminance blocks of the video data associated with the chroma blocks, and a plurality of luminance prediction modes (262) that can be used to decode the luminance components of the video data. In some instances, the processing circuitry of the video decoder 30 may add one or more DMs to the MPM candidate list, and may add one or more chroma modes inherited from adjacent chroma blocks in the MPM candidate list after the positions of all one or more DMs appearing in the MPM candidate list.

In some instances, the processing circuitry of the video decoder 30 may omit any additional instances of the LM mode from the MPM candidate list in response to the determination of one or more adjacent chroma blocks used to predict chroma blocks. In some instances, the processing circuitry of the video decoder 30 may receive a one-bit flag in the encoded video bitstream indicating whether a chroma block is encoded using the LM mode. In one scenario, the processing circuitry of the video decoder 30 may determine that the received one-bit flag is set to a deactivated state, receive an MPM index corresponding to a specific mode in the MPM candidate list, and select a specific mode corresponding to the received MPM index based on the received one-bit flag being set to a deactivated state. In another scenario, the processing circuitry of the video decoder 30 may determine that the received one-bit flag is set to an enabled state, and select an LM mode from the MPM candidate list based on the received one-bit flag being set to an enabled state.

In some instances, the processing circuitry of the video decoder 30 may determine whether the number of default modes associated with a chroma block meets a predetermined threshold. Based on the determination that the number of default modes meets the predetermined threshold, the processing circuitry of the video decoder 30 may add each default mode to the MPM candidate list and may omit all default modes from the MPM candidate list. The processing circuitry of the video decoder 30 may select a mode from the MPM candidate list (264). Subsequently, the processing circuitry of the video decoder 30 may decode the chroma block according to the mode selected from the MPM candidate list (266).

In some instances, to form an MPM candidate list, the processing circuitry of the video decoder 30 may add one or more DMs to the MPM candidate list, and may add one or more chroma modes inherited from the chroma block to the MPM candidate list after all positions of the one or DMs appearing in the MPM candidate list. In some instances, to form an MPM candidate list, the processing circuitry of the video decoder 30 may add one or more linear model (LM) modes to the MPM candidate list. In one such instance, the processing circuitry of the video decoder 30 may determine that one or more LM modes include a first instance of a first mode and one or more additional instances of the first LM mode, and may omit one or more additional instances of the LM mode from the MPM candidate list in response to the determination that the first LM mode is used to predict one or more adjacent chroma blocks of the chroma block.

In some instances, the processing circuitry of the video decoder 30 may receive a one-bit flag in the encoded video bitstream indicating whether a chroma block is encoded using an LM mode, wherein mode selection from the MPM candidate list is based on the value of the one-bit flag. In some such instances, the processing circuitry of the video decoder 30 may determine that one or more LM modes comprise multiple LM modes, and may determine that the received one-bit flag is set to an enabled state. In some such instances, the processing circuitry of the video decoder 30 may receive an LM index corresponding to the position of a specific LM mode among multiple LM modes in the MPM candidate list, and may select a specific LM mode corresponding to the received LM index for decoding the chroma block based on the received one-bit flag being set to an enabled state. In some instances, to select a mode from the MPM candidate list, the processing circuitry of the video decoder 30 may determine that the received one-bit flag is set to a disabled state, may receive an MPM index corresponding to a specific mode in the MPM candidate list, and may select a specific mode corresponding to the received MPM index based on the received one-bit flag being set to a disabled state.

In some instances, the processing circuitry of the video decoder 30 may determine whether the number of default modes associated with a chroma block meets a predetermined threshold. In these instances, the processing circuitry of the video decoder 30 may perform one of the following: (i) adding each default mode in the default modes to the MPM candidate list based on the determination that the number of default modes does not meet the predetermined threshold; or (ii) omitting all default modes from the MPM candidate list based on the determination that the number of default modes meets the predetermined threshold.

Figure 17 is a flowchart illustrating an example process 280 executable by the processing circuitry of the video encoder 20 according to an aspect of the present invention. Process 280 may begin when the processing circuitry of the video encoder 20 performs the following operation: forming a most probable mode (MPM) candidate list for chroma blocks of video data stored in the memory of the video encoder 20, such that the MPM candidate list includes linear model (LM) modes, one or more derived modes (DMs) associated with luminance blocks of the video data associated with the chroma blocks, and multiple luminance prediction modes (282) that can be used to decode the luminance blocks. In some instances, the processing circuitry of the video encoder 20 may add one or more DMs to the MPM candidate list, and may add one or more chroma modes inherited from adjacent chroma blocks to positions in the MPM candidate list after all positions of one or more DMs appearing in the MPM candidate list.

In some instances, the processing circuitry of the video encoder 20 may omit any additional instances of the LM mode from the MPM candidate list in response to the determination that the LM mode is used to predict one or more adjacent chroma blocks. In some instances, the processing circuitry of the video encoder 20 may signal a one-bit flag in the encoded video bitstream indicating whether the chroma block is encoded using the LM mode. In one scenario, the processing circuitry of the video encoder 20 may set a one-bit flag to a deactivated state based on the determination that the chroma block is not encoded using the LM mode. In this scenario, based on the determination that the chroma block is not encoded using the LM mode and that the chroma block is encoded using a specific mode from the MPM candidate list, the processing circuitry of the video encoder 20 may signal an MPM index corresponding to the specific mode in the MPM candidate list in the encoded video bitstream. In another scenario, the processing circuitry of the video encoder 20 may set a one-bit flag to an enabled state based on the determination that the chroma block is encoded using the LM mode.

In some instances, the processing circuitry of the video encoder 20 may determine whether the number of default modes associated with a chroma block meets a predetermined threshold. Based on the determination that the number of default modes meets the predetermined threshold, the processing circuitry of the video encoder 20 may add each default mode to the MPM candidate list and may omit all default modes from the MPM candidate list. The processing circuitry of the video encoder 20 may select a mode (284) from the MPM candidate list. Subsequently, the processing circuitry of the video encoder 20 may encode the chroma block according to the mode selected from the MPM candidate list.

In some instances, to form an MPM candidate list, the processing circuitry of the video encoder 20 may add one or more linear model (LM) modes to the MPM candidate list. In some instances, the processing circuitry of the video encoder 20 may signal a flag in the encoded video bitstream indicating whether a chroma block is encoded using any of the one or more LM modes in the MPM candidate list. In some instances, the processing circuitry of the video encoder 20 may set a flag to a deactivated state based on the determination that the chroma block is not encoded using any LM mode in the candidate list, and may signal an MPM index corresponding to the specific mode in the MPM candidate list in the encoded video bitstream based on the determination that the chroma block is encoded using a specific mode in the MPM candidate list. In some instances, the processing circuitry of the video encoder 20 may set a flag to an enabled state based on the determination that the chroma block is encoded using a specific LM mode in the one or more LM modes in the MPM candidate list.

In some instances, the processing circuitry of the video encoder 20 may determine whether the number of default modes associated with a chroma block meets a predetermined threshold. Subsequently, the processing circuitry of the video encoder 20 may perform one of the following: (i) adding each default mode in the default modes to the MPM candidate list based on the determination that the number of default modes does not meet the predetermined threshold; or (ii) omitting all default modes from the MPM candidate list based on the determination that the number of default modes meets the predetermined threshold.

It should be recognized that, depending on the instance, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, combined, or may be omitted entirely (e.g., not all described actions or events are necessary to practice the techniques). Furthermore, in some instances, actions or events may be performed simultaneously rather than sequentially, for example, via multithreading, interrupt handling, or multiple processors.

In one or more instances, the described functionality may be implemented in hardware, software, solid-state, or any combination thereof. If implemented in software, the functionality may be stored on or transmitted via a computer-readable medium as one or more instructions or codes, and executed by a hardware-based processing unit. The computer-readable medium may include: a computer-readable storage medium corresponding to a tangible medium such as a data storage medium; or a communication medium containing, for example, any medium facilitating the transfer of a computer program from one place to another according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the techniques described herein. Computer program products may contain computer-readable media.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that is accessible to a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technology (e.g., infrared, radio, and microwave), then coaxial cable, optical fiber, twisted pair, DSL, or wireless technology (e.g., infrared, radio, and microwave) is included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. The combination of the above should also be included in the scope of computer-readable media.

The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" can refer to any of the above-described structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, the techniques can be fully implemented within one or more circuit or logic elements.

The techniques of this invention can be implemented using various devices or apparatuses, including wireless handsets, integrated circuits (ICs), or collections of ICs (e.g., chipsets). Various components, modules, or units are described in this invention to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. More precisely, as described above, various units can be combined within a codec hardware unit, or provided through a collection of interoperable hardware units (comprising one or more processors as described above) combined with suitable software and/or solid-state drives.

Various examples have been described. These and other examples are within the scope of the appended claims.

Claims (18)

1. A method for decoding video data, the method comprising: The most probable pattern (MPM) candidate list for the chroma blocks of the video data is formed, at least in part, through the following operations: One or more derived modes DM associated with the luminance block of the video data, the luminance block corresponding to the chrominance block, and multiple luminance prediction modes that can be used to decode the luminance components of the video data are added to the MPM candidate list. Add one or more linear model (LM) patterns to the MPM candidate list; Determine whether the one or more LM patterns include a first instance of the first LM pattern and one or more additional instances of the first LM pattern; and In response to the determination of one or more adjacent chroma blocks used to predict the first LM mode, the one or more additional instances of the LM mode are omitted from the MPM candidate list; a mode is selected from the MPM candidate list; and The chroma block is decoded according to the pattern selected from the MPM candidate list. 2. The method of claim 1, wherein forming the MPM candidate list comprises: One or more chroma modes that inherit from the chroma block are added at positions following all the positions of the one or more derived modes DM that appear in the MPM candidate list. 3. The method of claim 1, wherein decoding the chroma block includes decoding the chroma block, the method further comprising receiving in the encoded video bitstream a one-bit flag indicating whether the chroma block is encoded using a specific LM mode among the one or more LM modes, wherein the mode is selected from the MPM candidate list based on the value of the one-bit flag. 4. The method of claim 3, further comprising: It is determined that the one or more LM patterns contain multiple LM patterns; It is confirmed that the received flag has been set to the enabled state; Receive the LM index corresponding to the position of the specific LM mode among the plurality of LM modes in the MPM candidate list; and Based on the received flag being set to the enabled state, the specific LM mode corresponding to the received LM index is selected for decoding the chroma block. 5. The method of claim 3, wherein selecting the mode from the MPM candidate list comprises: determining that one received flag is set to a deactivated state; Receive the MPM index corresponding to a specific pattern in the MPM candidate list; and Based on the received flag being set to the disabled state, the specific mode corresponding to the received MPM index is selected. 6. The method of claim 1, wherein decoding the chroma block comprises encoding the chroma block, the method further comprising: In the encoded video bitstream, a flag is transmitted via signaling to indicate whether the chroma block is encoded using any of the one or more LM modes in the MPM candidate list. 7. The method of claim 6, further comprising: Based on the determination that the chroma block was not encoded using any LM mode from the MPM candidate list, the flag bit is set to a disabled state; and Based on the determination that the chroma block is not encoded using any LM mode of the MPM candidate list and based on the determination that the chroma block is encoded using a specific mode of the MPM candidate list, the MPM index corresponding to the specific mode of the MPM candidate list is signaled in the encoded video bitstream. 8. The method of claim 6, further comprising: Based on the determination that the chroma block is encoded using a specific LM mode from one or more LM modes in the MPM candidate list, the one-bit flag is set to the enabled state. 9. The method of claim 1, further comprising: Determine whether the number of default modes associated with the chroma block meets a predetermined threshold; and Perform one of the following: Based on the determination that the number of default patterns does not meet the predetermined threshold, each default pattern in the default patterns is added to the MPM candidate list, or Based on the determination that the number of default patterns meets the predetermined threshold, all default patterns are omitted from the MPM candidate list. 10. An apparatus for decoding video data, comprising: A memory configured to store video data; and The processing circuitry communicating with the memory is configured to perform the following operations: For the chroma blocks of the video data stored in the memory device, a list of most probable mode MPM candidates is formed, wherein, in order to form the list of MPM candidates, the processing circuitry is configured to perform the following operations: One or more derived modes DM associated with the luminance block of the video data, the luminance block corresponding to the chrominance block, and multiple luminance prediction modes that can be used to decode the luminance components of the video data are added to the MPM candidate list. Add one or more linear model (LM) patterns to the MPM candidate list; Determine whether the one or more LM patterns include a first instance of the first LM pattern and one or more additional instances of the first LM pattern; and In response to the determination of one or more adjacent chroma blocks used to predict the first LM mode, the one or more additional instances of the LM mode are omitted from the MPM candidate list; a mode is selected from the MPM candidate list; and The chroma block is decoded according to the pattern selected from the MPM candidate list. 11. The apparatus of claim 10, wherein, in order to form the MPM candidate list, the processing circuitry is further configured to perform the following operations: One or more chroma modes that inherit from the chroma block are added at positions following all the positions of the one or more derived modes DM that appear in the MPM candidate list. 12. The apparatus of claim 10, wherein, in order to decode the chroma block, the processing circuitry is configured to decode the chroma block, wherein the processing circuitry is further configured to receive in the encoded video bitstream a one-bit flag indicating whether the chroma block is encoded using a specific LM mode among the one or more LM modes, and wherein, in order to select the mode from the MPM candidate list, the processing circuitry is configured to select the mode based on the value of the one-bit flag. 13. The apparatus of claim 12, wherein the processing circuitry is further configured to perform the following operation: determining that the one or more LM modes comprise a plurality of LM modes; It is confirmed that the received flag has been set to the enabled state; Receive the LM index corresponding to the position of the specific LM mode in the plurality of LM modes in the MPM candidate list; and Based on the received flag being set to the enabled state, the specific LM mode corresponding to the received LM index is selected for decoding the chroma block. 14. The apparatus of claim 12, wherein in order to select the mode from the MPM candidate list, the processing circuitry is configured to perform the following operations: It is determined that the received flag has been set to a disabled state; Receive the MPM index corresponding to a specific pattern in the MPM candidate list; and Based on the received flag being set to the disabled state, the specific mode corresponding to the received MPM index is selected. 15. The apparatus of claim 10, wherein, in order to decode the chroma block, the processing circuitry is configured to encode the chroma block, and wherein the processing circuitry is further configured to signal in the encoded video bitstream a one-bit flag indicating whether the chroma block is encoded using any of the one or more LM modes. 16. The apparatus of claim 15, wherein the processing circuitry is further configured to perform the following operations: Based on the determination that the chroma block is not encoded using any of the one or more LM modes, the flag bit is set to a disabled state; and Based on the determination that the chroma block was not encoded using any of the one or more LM modes and based on the determination that the chroma block was encoded using a specific mode of the MPM candidate list, the MPM index corresponding to the specific mode of the MPM candidate list is signaled in the encoded video bitstream. 17. The apparatus of claim 15, wherein the processing circuitry is further configured to perform the following operations: Based on the determination of encoding using a specific LM mode among the one or more LM modes for the chroma block, the bit flag is set to the enabled state. 18. The apparatus of claim 10, wherein the processing circuitry is further configured to perform the following operations: Determine whether the number of default modes associated with the chroma block meets a predetermined threshold; Based on the determination that the number of default patterns does not meet the predetermined threshold, each default pattern in the default patterns is added to the MPM candidate list; and Based on the determination that the number of default patterns meets the predetermined threshold, all default patterns are omitted from the MPM candidate list.