HK1205841B - Video coding with improved random access point picture behaviors - Google Patents

Description

Video decoding with improved random access point picture behavior

This application claims the benefit of U.S. Provisional Application No. 61/703,695, filed September 20, 2012, the entire contents of which are incorporated herein by reference.

Technical Field

The present disclosure relates generally to processing video data, and more particularly to random access pictures for use in video data.

Background Art

Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital camcorders, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10, "Advanced Video Coding (AVC)," the currently developing High Efficiency Video Coding (HEVC) standard, and extensions of such standards. By implementing such video coding techniques, video devices can more efficiently transmit, receive, encode, decode, and/or store digital video information.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video frame or a portion of a video frame) may be partitioned into video blocks (which may also be referred to as treeblocks), coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames.

Spatial prediction or temporal prediction results in a predictive block for the block to be coded. Residual data represents the pixel differences between the original block to be coded and the predictive block. Inter-coded blocks are encoded based on a motion vector pointing to a block of reference samples forming the predictive block and the residual data indicating the difference between the coded block and the predictive block. Intra-coded blocks are encoded based on an intra-coding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, resulting in residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients, and entropy coding can be applied to achieve even more compression.

Summary of the Invention

In general, this disclosure describes techniques for providing improved support for random access point (RAP) pictures, including clean random access (CRA) pictures and broken link access (BLA) pictures, in video coding. In some cases, RAP pictures may alternatively be referred to as intra random access point (IRAP) pictures. Specifically, this disclosure describes techniques for selecting coded picture buffer (CPB) parameters that define a CPB for a CRA or BLA picture in a video bitstream of a video coding device. Either a default set or an alternative set of CPB parameters can be used to define the CPB. If the default set is used when an alternative set should be selected, the CPB may overflow.

In one example, the disclosure is directed to a method of processing video data, the method comprising receiving a bitstream representing a plurality of pictures, the plurality of pictures including one or more of CRA pictures or BLA pictures; and receiving a message indicating whether an alternative set of CPB parameters is to be used for at least one of the CRA pictures or the BLA pictures. The method further comprises setting a variable defined to indicate the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the received message; and selecting the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the variable for the picture.

In another example, the present disclosure is directed to a video decoding device for processing video data, the device comprising a CPB configured to store the video data; and one or more processors configured to receive a bitstream representing a plurality of pictures, the plurality of pictures including one or more of CRA pictures or BLA pictures; receive a message indicating whether to use an alternative set of CPB parameters for at least one of the CRA pictures or the BLA pictures; set a variable defined to indicate the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the received message; and select the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the variable for the picture.

In other examples, the present disclosure is directed to a video decoding device for processing video data, the device comprising means for receiving a bitstream representing a plurality of pictures, the plurality of pictures including one or more of CRA pictures or BLA pictures; means for receiving a message indicating whether an alternative set of CPB parameters is to be used for at least one of the CRA pictures or the BLA pictures; means for setting a variable defined to indicate the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the received message; and means for selecting the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the variable for the picture.

In an additional example, the present disclosure is directed to a computer-readable medium comprising instructions for processing video data, the instructions, when executed, causing one or more processors to receive a bitstream representing a plurality of pictures, the plurality of pictures including one or more of CRA pictures or BLA pictures; receive a message indicating whether an alternative set of CPB parameters is to be used for at least one of the CRA pictures or the BLA pictures; set a variable defined to indicate the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the received message; and select the set of CPB parameters for the one of the CRA pictures or the BLA pictures based on the variable for the picture.

The details of one or more examples are set forth in the following drawings and the description. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a block diagram illustrating an example video encoding and decoding system that may utilize the techniques described in this disclosure.

2 is a block diagram illustrating an example video encoder that may implement the techniques described in this disclosure.

3 is a block diagram illustrating an example video decoder that may implement the techniques described in this disclosure.

4 is a block diagram illustrating an example destination device configured to operate according to a hypothetical reference decoder (HRD).

5 is a flowchart illustrating an example operation of selecting a set of coded picture buffer (CPB) parameters based on a variable that indicates the set of CPB parameters for a particular random access point (RAP) picture in a bitstream.

6 is a flow diagram illustrating example operations of setting a network abstraction layer (NAL) unit type for a particular RAP picture based on a variable that indicates a set of CPB parameters for the picture.

7 is a flowchart illustrating an example operation of selecting a set of CPB parameters for a particular RAP picture based on the NAL unit type for the picture and a variable that indicates the set of CPB parameters for the picture.

8 is a flowchart illustrating an example operation of selecting a set of CPB parameters based on a variable defined to indicate a network abstraction layer (NAL) unit type for a particular RAP picture in a bitstream.

9 is a block diagram illustrating an example collection of devices forming part of a network.

DETAILED DESCRIPTION

This disclosure describes techniques for providing improved support for random access point (RAP) pictures, including clean random access (CRA) pictures and broken link access (BLA) pictures, in video coding. In some cases, RAP pictures may alternatively be referred to as intra random access point (IRAP) pictures. Specifically, this disclosure describes techniques for selecting coded picture buffer (CPB) parameters, which are used to define the CPB for CRA and BLA pictures in a video bitstream of a video coding device. The hypothetical reference decoder (HRD) relies on HRD parameters that include buffering period information and picture timing information. The buffering period information defines CPB parameters, namely, initial CPB removal delay and initial CPB removal delay offset. Either a default set or an alternative set of CPB parameters can be used to define the CPB based on the type of picture used to initialize the HRD. If the default set is used when an alternative set should be selected, the CPB that conforms to the HRD in the video coding device may overflow.

According to the techniques, a video coding device receives a bitstream representing a plurality of pictures, the plurality of pictures including one or more CRA pictures or BLA pictures, and further receives a message indicating whether an alternative set of CPB parameters is used for each of the CRA pictures or BLA pictures. The message may be received from an external device, such as a processing device included in a streaming server, an intermediate network element, or another network entity.

The video coding device sets a variable based on the received message, the variable being defined to indicate a set of CPB parameters for a given one of a CRA picture or a BLA picture. The video coding device then selects a set of CPB parameters for the picture based on the variable for the given one of the CRA picture or the BLA picture. The selected set of CPB parameters is applied to the CPB included in the video encoder or the video decoder to ensure that the CPB will not overflow during video coding. In some cases, the video coding device may set a network abstraction layer (NAL) unit type for the given one of the CRA picture or the BLA picture. The video coding device may set the NAL unit type for the picture to the signaled one, or the video coding device may set the NAL unit type based on the variable for the picture. The video coding device may select a set of CPB parameters for the given picture based on the NAL unit type for the picture and the variable.

FIG1 is a block diagram illustrating an example video encoding and decoding system 10 that may utilize the techniques described in this disclosure. As shown in FIG1 , system 10 includes a source device 12 that provides encoded video data to be decoded at a later time by a destination device 14. In detail, source device 12 provides the video data to destination device 14 via a computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephones such as so-called "smart" phones, so-called "smart" tablets, televisions, cameras, display devices, digital media players, digital game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

Destination device 14 may receive the encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 may comprise any type of medium or device capable of moving the encoded video data from source device 12 to destination device 14. In one example, computer-readable medium 16 may comprise a communication medium to enable source device 12 to transmit the encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard (e.g., a wireless communication protocol) and transmitted to destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network such as the Internet). The communication medium may include routers, switches, base stations, or any other equipment that may be used to facilitate communication from source device 12 to destination device 14.

In some examples, the encoded data may be output from output interface 22 to a storage device. Similarly, the encoded data may be accessed from the storage device via the input interface. The storage device may include any of a variety of distributed or locally accessible data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In other examples, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by source device 12. Destination device 14 may access the stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to destination device 14. Example file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data via any standard data connection, including an Internet connection. This data connection may include a wireless channel suitable for accessing encoded video data stored on a file server (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

The techniques of this disclosure are not necessarily limited to wireless applications or settings. The techniques may be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. In other examples, the source and destination devices may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18 (e.g., an external camera). Similarly, destination device 14 may interface with an external display device rather than including an integrated display device.

The illustrated system 10 of FIG. 1 is merely an example. The techniques of this disclosure may be performed by any digital video encoding and/or decoding device. Although typically performed by a video encoding device, the techniques may also be performed by a video encoder/decoder, commonly referred to as a "codec." Furthermore, the techniques of this disclosure may also be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of such decoding devices in which source device 12 generates coded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate in a substantially symmetrical manner, such that each of devices 12, 14 includes video encoding and decoding components. Thus, system 10 may support one-way or two-way video transmission between video devices 12, 14, for example, for video streaming, video playback, video broadcasting, or video telephony.

Video source 18 of source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, and/or a video feed interface for receiving video from a video content provider. As another alternative, video source 18 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, as mentioned above, the techniques described in this disclosure may be generally applicable to video coding and may be applied to wireless and/or wired applications. In each case, the captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output by output interface 22 onto computer-readable medium 16.

Computer-readable medium 16 may include transitory media, such as wireless broadcasts or wired network transmissions, or storage media (i.e., non-transitory storage media), such as a hard drive, a flash drive, an optical disc, a digital video disc, a Blu-ray disc, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from source device 12 and provide the encoded video data to destination device 14, such as via a network transmission. Similarly, a computing device at a media production facility (e.g., a disc imprinting facility) may receive encoded video data from source device 12 and produce a disc containing the encoded video data. Thus, in various examples, computer-readable medium 16 may be understood to include one or more computer-readable media in various forms.

Input interface 28 of destination device 14 receives information from computer-readable medium 16. The information of computer-readable medium 16 may include syntax information defined by video encoder 20, including syntax elements describing characteristics and/or processing of blocks and other coded units (e.g., GOPs), that is also used by video decoder 30. Display device 32 displays the decoded video data to a user and may comprise any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

Video encoder 20 and video decoder 30 may operate according to a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC Test Model (HM). Alternatively, video encoder 20 and video decoder 30 may operate according to other proprietary or industry standards, such as the ITU-T H.264 standard, or extensions of such standards, also known as MPEG-4 Part 10 (Advanced Video Coding (AVC)). However, the techniques of this disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although not shown in FIG. 1 , in some aspects, video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams. If applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol or other protocols, such as the User Datagram Protocol (UDP).

The ITU-T H.264/MPEG-4 (AVC) standard was developed by the ITU-T Video Coding Expert Group (VCEG) in conjunction with the ISO/IEC Motion Picture Expert Group (MPEG) as a collective partnership known as the Joint Video Team (JVT). In some aspects, the techniques described in this disclosure may be applied to devices that generally conform to the H.264 standard. The H.264 standard is described in ITU-T Recommendation H.264 (Advanced Video Coding for General Audiovisual Services), published in March 2005 by the ITU-T study group, which may be referred to herein as the H.264 standard, the H.264 specification, or the H.264/AVC standard or specification. The Joint Video Team (JVT) continues to work on extensions to H.264/MPEG-4 AVC.

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, the device may store instructions for the software in a suitable, non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in the respective device.

The JCT-VC is working on the development of the HEVC standard. The HEVC standardization effort is based on an evolving model of a video coding device called the HEVC Test Model (HM). The HM assumes several additional capabilities of video coding devices compared to existing devices based on, for example, ITU-T H.264/AVC. For example, while H.264 provides nine intra-frame prediction coding modes, the HM can provide up to 33 intra-frame prediction coding modes.

In general, the HM's working model describes that a video frame or picture can be divided into a sequence of treeblocks, or largest coding units (LCUs), that include both luma and chroma samples. Syntax data within the bitstream can define the size of an LCU, which is a largest coding unit in terms of the number of pixels. A slice comprises a number of consecutive treeblocks in coding order. A video frame or picture can be partitioned into one or more slices. Each treeblock can be split into coding units (CUs) according to a quadtree. In general, a quadtree data structure includes one node per CU, with the root node corresponding to the treeblock. If a CU is split into four sub-CUs, the node corresponding to the CU includes four leaf nodes, each of which corresponds to one of the sub-CUs.

Each node in the quadtree data structure may provide syntax data for the corresponding CU. For example, a node in the quadtree may include a split flag, indicating whether the CU corresponding to the node is split into sub-CUs. Syntax elements for a CU may be defined recursively, and the syntax elements for a CU may depend on whether the CU is split into sub-CUs. If a CU is not further split, it is referred to as a leaf-CU. In the present invention, the four sub-CUs of a leaf-CU are referred to as leaf-CUs even if there is no explicit splitting of the original leaf-CU. For example, if a CU of size 16×16 is not further split, the four 8×8 sub-CUs are also referred to as leaf-CUs, even though the 16×16 CU has never been split.

CUs have uses similar to macroblocks of the H.264 standard, except that CUs do not have size distinctions. For example, a treeblock can be split into four child nodes (also called sub-CUs), and each child node can be a parent node and split into another four child nodes. The final unsplit child node, called a leaf node of the quadtree, includes a coding node, which is also called a leaf CU. Syntax data associated with the coded bitstream can define the maximum number of times a treeblock can be split (which is called the maximum CU depth) and can also define the minimum size of the coding node. Therefore, the bitstream can also define a minimum coding unit (SCU). This disclosure uses the term "block" to refer to any of a CU, PU, or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., macroblocks and their sub-blocks in H.264/AVC).

A CU comprises a coding node and prediction units (PUs) and transform units (TUs) associated with the coding node. The size of the CU corresponds to the size of the coding node and must be square in shape. The size of a CU can range from 8×8 pixels to the size of a treeblock with a maximum of 64×64 pixels or larger. Each CU may contain one or more PUs and one or more TUs. Syntax data associated with a CU may describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may differ depending on whether the CU is encoded in skip or direct mode, encoded in intra-prediction mode, or encoded in inter-prediction mode. A PU may be partitioned into non-square shapes. Syntax data associated with a CU may also describe, for example, the partitioning of the CU into one or more TUs according to a quadtree. The shape of a TU may be square or non-square (for example, rectangular).

The HEVC standard allows for transforms based on TUs, which can be different for different CUs. The size of a TU is typically set based on the size of the PU within a given CU defined for a partitioned LCU, but this may not always be the case. A TU is typically the same size as a PU, or smaller than a PU. In some examples, a quadtree structure called a "residual quadtree" (RQT) can be used to subdivide the residual samples corresponding to a CU into smaller units. The leaf nodes of the RQT can be referred to as transform units (TUs). Pixel difference values associated with a TU can be transformed to produce transform coefficients, which can be quantized.

A leaf-CU may include one or more prediction units (PUs). Generally, a PU represents a spatial region corresponding to all or a portion of the corresponding CU and may include data used to retrieve reference samples for the PU. In addition, a PU includes data related to prediction. For example, when the PU is intra-mode encoded, the data for the PU may be included in a residual quadtree (RQT), which may include data describing the intra-prediction mode of the TU corresponding to the PU. As another example, when the PU is inter-mode encoded, the PU may include data defining one or more motion vectors for the PU. The data defining the motion vector of the PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., quarter-pixel precision or eighth-pixel precision), the reference picture to which the motion vector points, and/or the reference picture list for the motion vector (e.g., List 0, List 1, or List C).

A leaf-CU with one or more PUs may also include one or more transform units (TUs). The transform units may be specified using an RQT (also known as a TU quadtree structure), as discussed above. For example, a split flag may indicate whether a leaf-CU is split into four transform units. Each transform unit may then be further split into additional sub-TUs. When a TU is not further split, it may be referred to as a leaf-TU. Generally speaking, for intra coding, all leaf-TUs belonging to a leaf-CU share the same intra prediction mode. That is, the same intra prediction mode is typically applied to calculate prediction values for all TUs of a leaf-CU. For intra coding, the video encoder may calculate the residual value for each leaf-TU using the intra prediction mode as the difference between the portion of the CU corresponding to the TU and the original block. A TU is not necessarily limited to the size of a PU. Thus, a TU may be larger or smaller than a PU. For intra coding, a PU may be co-located with a corresponding leaf-TU of the same CU. In some examples, the maximum size of a leaf-TU may correspond to the size of the corresponding leaf-CU.

Furthermore, the TUs of a leaf-CU may also be associated with corresponding quadtree data structures called residual quadtrees (RQTs). That is, a leaf-CU may include a quadtree that indicates how the leaf-CU is partitioned into TUs. The root node of a TU quadtree typically corresponds to a leaf-CU, while the root node of a CU quadtree typically corresponds to a treeblock (or LCU). TUs that are not split in an RQT are referred to as leaf-TUs. In general, unless otherwise indicated, this disclosure uses the terms CU and TU to refer to a leaf-CU and leaf-TU, respectively.

A video sequence typically includes a series of video frames or pictures. A group of pictures (GOP) typically includes a series of one or more of the video pictures. A GOP may include syntax data describing the number of pictures included in the GOP in a header of the GOP, a header of one or more of the pictures, or elsewhere. Each slice of a picture may include slice syntax data describing the coding mode of the respective slice. Video encoder 20 typically operates on video blocks within individual video slices to encode the video data. A video block may correspond to a coding node within a CU. A video block may have a fixed or varying size and may differ in size according to a specified coding standard.

As an example, the HM supports prediction with various PU sizes. Assuming a particular CU is 2N×2N in size, the HM supports intra prediction with PU sizes of 2N×2N or N×N, and inter prediction with symmetric PU sizes of 2N×2N, 2N×N, N×2N, or N×N. The HM also supports asymmetric partitioning for inter prediction with PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N. In asymmetric partitioning, the CU is unpartitioned in one direction and partitioned into 25% and 75% in the other direction. The portion of the CU corresponding to the 25% partition is indicated by an "n" followed by an indication of "up," "down," "left," or "right." Thus, for example, "2N×nU" refers to a 2N×2N CU that is partitioned horizontally into a 2N×0.5N PU at the top and a 2N×1.5N PU at the bottom.

In this disclosure, "NxN" and "N by N" may be used interchangeably to refer to the pixel dimensions of a video block in terms of vertical and horizontal dimensions, e.g., 16x16 pixels or 16 by 16 pixels. Generally, a 16x16 block will have 16 pixels in the vertical direction (y=16) and 16 pixels in the horizontal direction (x=16). Similarly, an NxN block typically has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. The pixels in a block may be arranged in rows and columns. Furthermore, a block need not necessarily have the same number of pixels in the horizontal direction as in the vertical direction. For example, a block may comprise NxM pixels, where M is not necessarily equal to N.

After intra-predictive or inter-predictive coding of the PUs of a CU, video encoder 20 may calculate residual data for the TUs of the CU. A PU may include syntax data that describes a method or mode of generating predictive pixel data in the spatial domain (also called the pixel domain), and a TU may include coefficients in the transform domain after applying a transform, such as a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform, to the residual video data. The residual data may correspond to pixel differences between pixels of an unencoded picture and prediction values corresponding to the PU. Video encoder 20 may form TUs including the residual data for the CU, and then transform the TUs to generate transform coefficients for the CU.

After performing any transforms to generate transform coefficients, video encoder 20 may perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the coefficients, thereby providing further compression. The quantization process may reduce the bit depth associated with some or all of the coefficients. For example, an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

After quantization, the video encoder may scan the transform coefficients, generating a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) coefficients at the front of the array and lower energy (and therefore higher frequency) coefficients at the back of the array. In some examples, video encoder 20 may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector that can be entropy encoded. In other examples, video encoder 20 may perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 20 may entropy encode the one-dimensional vector, for example, according to context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method. Video encoder 20 may also entropy encode syntax elements associated with the encoded video data for use by video decoder 30 in decoding the video data.

To perform CABAC, video encoder 20 may assign context within a context model to a symbol to be transmitted. The context may relate to, for example, whether neighboring values of a symbol are non-zero. To perform CAVLC, video encoder 20 may select a variable length code for the symbol to be transmitted. Codewords in VLC may be constructed so that relatively shorter codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, using VLC may achieve bit savings (compared to, for example, using equal-length codewords for each symbol to be transmitted). Probability decisions may be based on the context assigned to the symbol.

Video encoder 20 may further send syntax data (e.g., block-based syntax data, frame-based syntax data, and GOP-based syntax data), e.g., in a frame header, block header, slice header, or GOP header, to video decoder 30. The GOP syntax data may describe the number of frames in the corresponding GOP, and the frame syntax data may indicate the encoding/prediction mode used to encode the corresponding frame.

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder or decoder circuits, as applicable, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined video encoder/decoder (CODEC). A device including video encoder 20 and/or video decoder 30 may comprise an integrated circuit, a microprocessor, and/or a wireless communication device (e.g., a cellular telephone).

Video coding standards may include specifications for video buffering models. In AVC and HEVC, the buffering model is called a Hypothetical Reference Decoder (HRD), which includes buffering models for both the coded picture buffer (CPB) and the decoded picture buffer (DPB) included in video encoder 20 and/or video decoder 30, with the CPB and DPB behaviors mathematically specified. The HRD directly imposes constraints on different timings, buffer sizes, and bit rates, and indirectly on bitstream characteristics and statistics. The complete set of HRD parameters includes five basic parameters: initial CPB removal delay, CPB size, bit rate, initial DPB output delay, and DPB size. In AVC and HEVC, bitstream conformance and decoder conformance are specified as part of the HRD specification. Although HRD is named for a type of decoder, it is generally required on the encoder side to ensure bitstream conformance (i.e., the bitstream generated by the encoder conforms to the requirements of the decoder), but is generally not required on the decoder side.

In the AVC and HEVC HRD models, decoding or CPB removal is based on access units, and picture decoding is assumed to be instantaneous. In practice, if a conforming decoder strictly follows the decoding time signaled in a picture timing supplemental enhancement information (SEI) message to start decoding an access unit, the earliest possible time to output a particular decoded picture is equal to the decoding time of that particular picture plus the time required to decode that particular picture. Unlike the AVC and HEVC HRD models, the time required to decode a picture in the real world is not equal to zero. As used in this disclosure, the terms "instantaneous" and "instantaneously" may refer to any duration that can be assumed to be instantaneous in one or more coding models or idealized aspects of any one or more coding models, with the understanding that this may be different from "instantaneous" in a physical or literal sense. For example, for the purposes of this disclosure, a function or process may be considered nominally "instantaneous" if it occurs at or within a practical tolerance of the assumed or idealized earliest possible time to perform the function or process. In some examples, syntax and variable names as used herein may be understood according to their meaning within the HEVC model.

The following description of example hypothetical reference decoder (HRD) operations, example operations of a coded picture buffer, example timing of bitstream arrival, example timing of decoding unit removal, example decoding of a decoding unit, example operations of a decoded picture buffer, example removal of a picture from a decoded picture buffer, example picture output, and example current decoded picture marking and storage is provided to illustrate an example of a video encoder 20 and/or video decoder 30 that may be configured to, among other functions, store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for one or more decoding units, remove a decoding unit from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding unit. In other examples, the operations may be defined or performed differently. In this manner, video encoder 20 and/or video decoder 30 may be configured to operate according to various examples of HRD operations described below.

The HRD may be initialized at any of the buffering period supplemental enhancement information (SEI) messages. Prior to initialization, the CPB may be empty. After initialization, the HRD may not be initialized again by a subsequent buffering period SEI message. The access unit associated with the buffering period SEI message that initializes the CPB may be referred to as access unit 0. The decoded picture buffer may contain picture storage buffers. Each of the picture storage buffers may contain decoded pictures marked as "used for reference" or held for future output. Prior to initialization, the DPB may be empty.

An HRD (e.g., video encoder 20 and/or video decoder 30) may operate as follows. A hypothetical stream scheduler (HSS) may deliver data associated with decoding units that flow into the CPB according to a specified arrival schedule. In one example, data associated with each decoding unit may be instantaneously removed and decoded by a transient decoding process at the CPB removal time. Each decoded picture may be placed in the DPB. Decoded pictures may be removed from the DPB at a later time, either at the DPB output time or when they are no longer needed for inter-frame prediction reference.

The HRD relies on HRD parameters, including the CPB parameters for the initial CPB removal delay and the initial CPB removal delay offset. In some cases, the HRD parameters may be determined based on the type of picture used to initialize the HRD. In the case of random access, the HRD may be initialized with a random access point (RAP) picture, such as a clean random access (CRA) picture or a broken link access (BLA) picture. In some cases, a RAP picture may alternatively be referred to as an intra random access point (IRAP) picture. For example, when the HRD is initialized with a BLA picture that does not have an associated non-decodable preceding picture in the bitstream, also known as a marked-for-discard (TFD) picture or a random access skipped preceding (RASL) picture, an alternative set of CPB parameters may be used. Otherwise, a default set of CPB parameters is used for the HRD. If the default set of CPB parameters is used when an alternative set should have been selected, the CPB may overflow.

In some examples, a given CRA picture or BLA picture may have an associated TFD picture in the original bitstream, and the TFD picture may be removed from the original bitstream by an external device. The external device may include a processing device included in a streaming server, an intermediate network element, or another network entity. However, the external device may not be able to change the signaled type of a given CRA picture or BLA picture to reflect the removal of the associated TFD picture. In this case, a default set of CPB parameters may be selected based on the type of the CRA picture or BLA picture signaled in the original bitstream. This situation may result in a CPB overflow because the TFD picture has been removed by the external device, such that the picture no longer has an associated TFD picture, and an alternative set of CPB parameters should be used for HRD.

This disclosure describes techniques for selecting CPB parameters used to define a CPB for a CRA picture or a BLA picture in a video bitstream for video encoder 20 and/or video decoder 30. According to the techniques, video decoder 30 receives a bitstream representing a plurality of pictures, the plurality of pictures including one or more CRA pictures or BLA pictures, and also receives a message indicating whether to use an alternative set of CPB parameters for at least one of the CRA pictures or the BLA pictures. The message may be received from an external device, such as a processing device included in a streaming server, an intermediate network element, or another network entity.

Based on the received message, video decoder 30 sets a variable defined to indicate a set of CPB parameters for a given one of a CRA picture or a BLA picture. Video decoder 30 then selects a set of CPB parameters for the picture based on the variable for the given one of the CRA picture or the BLA picture. In some cases, video decoder 30 may set a network abstraction layer (NAL) unit type for the given one of the CRA picture or the BLA picture and may select a set of CPB parameters for the picture based on the NAL unit type and the variable for the given picture.

The selected set of CPB parameters is applied to the CPB included in video decoder 30 to ensure that the CPB will not overflow during video decoding. Video encoder 20 may be configured to perform similar operations and apply the selected set of CPB parameters to the CPB included in video encoder 20 to ensure that the CPB included in video encoder 20 will not overflow during video encoding, and that the CPB included in video decoder 30 will not overflow when receiving an encoded bitstream generated by video encoder 20.

FIG2 is a block diagram illustrating an example of a video encoder 20 that may implement the techniques described in this disclosure. Video encoder 20 may perform intra- and inter-coding of video blocks within a video slice. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or pictures of a video sequence. Intra-mode (I-mode) may refer to any of several spatially-based coding modes. Inter-mode (e.g., unidirectional prediction (P-mode) or bidirectional prediction (B-mode)) may refer to any of several temporally-based coding modes.

As shown in FIG2 , video encoder 20 receives a current video block within a video frame to be encoded. In the example of FIG2 , video encoder 20 includes a mode select unit 40, a summer 50, a transform processing unit 52, a quantization unit 54, an entropy encoding unit 56, a decoded picture buffer (DPB) 64, and a coded picture buffer (CPB) 66. Mode select unit 40, in turn, includes a motion compensation unit 44, a motion estimation unit 42, an intra-prediction processing unit 46, and a partitioning unit 48. For video block reconstruction, video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and summer 62. A deblocking filter (not shown in FIG2 ) may also be included to filter block boundaries, thereby removing blocking artifacts from the reconstructed video. If necessary, the deblocking filter will typically filter the output of summer 62. In addition to the deblocking filter, additional filters (in-loop or post-loop) may also be used. Such filters are not shown for simplicity, but if desired, such filters may filter the output of summer 50 (as in-loop filters).

During the encoding process, video encoder 20 receives a video frame or slice to be coded. The frame or slice may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive coding of the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-prediction processing unit 46 may alternatively perform intra-predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction. Video encoder 20 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

Furthermore, partition unit 48 may partition blocks of video data into sub-blocks based on evaluation of previous partitioning schemes in previous coding passes. For example, partition unit 48 may initially partition a frame or slice into LCUs and, based on rate-distortion analysis (e.g., rate-distortion optimization), partition each of the LCUs into sub-CUs. Mode select unit 40 may further generate a quadtree data structure indicating the partitioning of the LCUs into sub-CUs. A leaf node CU of the quadtree may include one or more PUs and one or more TUs.

Mode select unit 40 may select one of the coding modes (intra or inter) (e.g., based on the error result) and provide the resulting intra- or inter-coded block to summer 50 to generate residual block data and to summer 62 to reconstruct the encoded block for use as a reference frame. Mode select unit 40 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy encoding unit 56.

Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are described separately for conceptual purposes. Motion estimation, performed by motion estimation unit 42, is the process of generating motion vectors, which estimate the motion of video blocks. For example, a motion vector may indicate the displacement of a PU of a video block within a current video frame or picture relative to a predictive block within a reference frame (or other coded unit) relative to the current block being coded within the current frame (or other coded unit). A predictive block is a block that is found to closely match the block to be coded in terms of pixel difference, which can be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. In some examples, video encoder 20 may calculate values for sub-integer pixel positions of a reference picture stored in DPB 64. For example, video encoder 20 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform motion searches relative to full-pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

Motion compensation performed by motion compensation unit 44 may involve extracting or generating a predictive block based on the motion vector determined by motion estimation unit 42. Again, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists. Summer 50 forms a residual video block by subtracting the pixel values of the predictive block from the pixel values of the current video block being coded, forming pixel difference values, as discussed below. In general, motion estimation unit 42 performs motion estimation for the luma component, and motion compensation unit 44 uses the motion vector calculated based on the luma component for both chroma and luma components. Mode select unit 40 may also generate syntax elements associated with video blocks and video slices for use by video decoder 30 when decoding video blocks of the video slices.

As described above, intra-prediction processing unit 46 may intra-predict a current block, as an alternative to the inter-prediction performed by motion estimation unit 42 and motion compensation unit 44. In particular, intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode the current block. In some examples, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode select unit 40) may select an appropriate intra-prediction mode to use from among the tested modes.

For example, intra-prediction processing unit 46 may calculate rate-distortion values using a rate-distortion analysis for various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between an encoded block and the original, unencoded block that was encoded to produce the encoded block, as well as the bit rate (i.e., bit count) used to produce the encoded block. Intra-prediction processing unit 46 may calculate a ratio based on the distortion and rate for each encoded block to determine which intra-prediction mode exhibits the best rate-distortion value for that block.

After selecting an intra-prediction mode for a block, intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Video encoder 20 may include configuration data in the transmitted bitstream, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables), definitions of encoding contexts for various blocks and the most probable intra-prediction mode to be used for each of the contexts, the intra-prediction mode index tables, and indications of the modified intra-prediction mode index tables.

Video encoder 20 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being coded. Summer 50 represents the one or more components that perform this subtraction operation. Transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, producing a video block comprising residual transform coefficient values. Transform processing unit 52 may perform other transforms conceptually similar to the DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. In any case, transform processing unit 52 applies a transform to the residual block, producing a block of residual transform coefficients. The transform may convert the residual information from the pixel value domain to a transform domain (e.g., the frequency domain). Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

After quantization, entropy encoding unit 56 entropy codes the quantized transform coefficients. For example, entropy encoding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. In the case of context-based entropy coding, the context may be based on neighboring blocks. After entropy coding by entropy encoding unit 56, the encoded bitstream may be buffered or stored, more or less temporarily, in CPB 66, transmitted to another device (e.g., video decoder 30), or archived for later transmission or retrieval.

Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, for later use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the frames of DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reconstructed video block for storage in DPB 64. The reconstructed video block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for inter-coding a block in a subsequent video frame.

DPB 64 may be or may be included in a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). DPB 64 may operate according to any combination of the example coded picture buffer and/or decoded picture buffer behaviors described in this disclosure. For example, video encoder 20 may be configured to operate according to a hypothetical reference decoder (HRD). In this case, DPB 64 included in video encoder 20 may be defined by HRD parameters including CPB parameters and DPB parameters according to the HRD's buffering model.

Similarly, CPB 66 may be or may be included in a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). Although shown as forming part of video encoder 20, in some examples, CPB 66 may form part of a device, unit, or module external to video encoder 20. For example, CPB 66 may form part of a stream scheduler unit (e.g., a delivery scheduler or a virtual stream scheduler (HSS)) external to video encoder 20. In the event that video encoder 20 is configured to operate according to an HRD, CPB 66 included in video encoder 20 may be defined by HRD parameters including CPB parameters for initial CPB removal delay and offset according to the buffering model of the HRD.

According to the techniques of this disclosure, video encoder 20 may apply either a default set or an alternative set of CPB parameters to CPB 66 to ensure that CPB 66 does not overflow during encoding of video data and that the CPB included in video decoder 30 does not overflow when receiving an encoded bitstream generated by video encoder 20. If the default set is used when an alternative set should be selected, CPB 66 included in video encoder 20 or the CPB included in video decoder 30 may overflow. Selection of appropriate CPB parameters is primarily a concern when random access point (RAP) pictures, such as clean random access (CRA) pictures or broken link access (BLA) pictures, are used to initialize HRDs. Thus, the techniques may provide improved support for RAP pictures in video coding.

Video encoder 20 may be configured to receive a bitstream representing a plurality of pictures, the plurality of pictures including one or more CRA pictures or BLA pictures, and also receive a message indicating whether an alternative set of CPB parameters is used for at least one of the CRA pictures or the BLA pictures. In some cases, the bitstream may be received directly from the encoding portion of video encoder 20 (e.g., entropy encoding unit 56 or CPB 66) at the decoding portion of video encoder 20 (i.e., inverse quantization unit 58 and inverse transform processing unit 60). The message may be received from an external device, such as a processing device included in a streaming server, an intermediate network element, or another network entity.

Based on the received message, video encoder 20 sets a variable defined to indicate a set of CPB parameters for a given one of a CRA picture or a BLA picture. Video encoder 20 then selects a set of CPB parameters for the given one of the CRA picture or the BLA picture based on the variable for the picture. Video encoder 20 applies the selected set of CPB parameters to CPB 66 included in video encoder 20 to ensure that CPB 66 will not overflow during video encoding and to ensure that the CPB included in video decoder 30 will not overflow when receiving the encoded bitstream generated by video encoder 20. In some cases, video encoder 20 may set a network abstraction layer (NAL) unit type for the given one of the CRA picture or the BLA picture and may select a set of CPB parameters for the given picture based on the NAL unit type and the variable for the picture. The CPB parameter selection process for RAP pictures is described in more detail with respect to video decoder 30 of FIG.

3 is a block diagram illustrating an example of a video decoder 30 that may implement the techniques described in this disclosure. In the example of FIG3 , video decoder 30 includes an entropy decoding unit 70, a prediction processing unit 71 including a motion compensation unit 72 and an intra-prediction processing unit 74, an inverse quantization unit 76, an inverse transform processing unit 78, a summer 80, a coded picture buffer (CPB) 68, and a decoded picture buffer (DPB) 82. In some examples, video decoder 30 may perform a decoding pass that is generally reciprocal to the encoding pass described with respect to video encoder 20 from FIG2 .

During the decoding process, video decoder 30 receives an encoded video bitstream from video encoder 20, which represents video blocks and associated syntax elements of an encoded video slice. Video decoder 30 may receive the encoded video bitstream from network entity 29. Network entity 29 may be, for example, a streaming server, a media-aware network element (MANE), a video editor/joiner, an intermediate network element, or other such device configured to implement one or more of the techniques described above. Network entity 29 may include external devices configured to perform the techniques of this disclosure. As described above, some of the techniques described in this disclosure may be implemented by network entity 29 before network entity 29 transmits the encoded video bitstream to video decoder 30. In some video decoding systems, network entity 29 and video decoder 30 may be parts of separate devices, while in other cases, the functionality described with respect to network entity 29 may be performed by the same device that includes video decoder 30.

The bitstream may be buffered or stored more or less temporarily in CPB 68 prior to entropy decoding by entropy decoding unit 70. Entropy decoding unit 70 of video decoder 30 then entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive syntax elements at the video slice level and/or the video block level.

When the video slice is coded as an intra-coded (I) slice, intra-prediction processing unit 74 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When the video frame is coded as an inter-coded (i.e., B or P) slice, motion compensation unit 72 generates predictive blocks for the video block of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 70. The predictive blocks may be generated according to one of the reference pictures within one of the reference picture lists. Video decoder 30 may construct the reference frame lists (List 0 and List 1) using default construction techniques based on the reference pictures stored in DPB 82.

Motion compensation unit 72 determines prediction information for video blocks of the current video slice by parsing motion vectors and other syntax elements, and uses the prediction information to generate predictive blocks for the current video block being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode (e.g., intra or inter prediction) used to code the video blocks of the video slice, the inter-prediction slice type (e.g., B slice or P slice), construction information for one or more of the slice's reference picture lists, motion vectors for each inter-coded video block of the slice, the inter-prediction state for each inter-coded video block of the slice, and other information to decode the video blocks in the current video slice.

Motion compensation unit 72 may also perform interpolation based on interpolation filters. Motion compensation unit 72 may calculate interpolated values for sub-integer pixels of a reference block using interpolation filters as used by video encoder 20 during encoding of the video block. In this case, motion compensation unit 72 may determine the interpolation filters used by video encoder 20 from received syntax elements and use the interpolation filters to produce a predictive block.

Inverse quantization unit 76 inverse quantizes (i.e., dequantizes) the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 70. The inverse quantization process may include determining the degree of quantization, and likewise the degree of inverse quantization, that should be applied using a quantization parameter QP _Y for each video block in the video slice calculated by video decoder 30. Inverse transform processing unit 78 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to produce residual blocks in the pixel domain.

After motion compensation unit 72 generates a predictive block for the current video block based on the motion vectors and other syntax elements, video decoder 30 forms a decoded video block by summing the residual block from inverse transform unit 78 with the corresponding predictive block generated by motion compensation unit 72. Summer 90 represents the one or more components that perform this summing operation. If desired, a deblocking filter may also be applied to filter the decoded blocks to remove blocking artifacts. Other loop filters (in the decoding loop or after the decoding loop) may also be used to smooth pixel transitions or otherwise improve video quality. The decoded video blocks in a given frame or picture are then stored in DPB 82, which stores reference pictures used for subsequent motion compensation. DPB 82 also stores the decoded video for later reproduction on a display device, such as display device 32 of FIG. 1 .

DPB 82 may be, or may be included in, a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). DPB 82 may operate according to any combination of the example coded picture buffer and/or decoded picture buffer behaviors described in this disclosure. For example, video decoder 30 may be configured to operate according to a hypothetical reference decoder (HRD). In this case, video decoder 30 may decode HRD parameters (including CPB parameters and DPB parameters) that define DPB 82 according to the HRD's buffering model.

Similarly, CPB 68 may be or may be included in a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). Although shown as forming part of video decoder 30, in some examples, CPB 68 may form part of a device, unit, or module external to video decoder 30. For example, CPB 68 may form part of a stream scheduler unit (e.g., a delivery scheduler or a virtual stream scheduler (HSS)) external to video decoder 30. In the event that video decoder 30 is configured to operate according to an HRD, video decoder 30 may decode HRD parameters, including CPB parameters for initial CPB removal delay and offset, used to define CPB 68 according to the buffering model of the HRD.

According to the techniques of this disclosure, video decoder 30 may apply either a default set or an alternative set of CPB parameters to CPB 68 to ensure that CPB 68 does not overflow during decoding of video data. If the default set is used when an alternative set should be selected, CPB 68 included in a video decoder configured to operate according to an HRD may overflow. The selection of appropriate CPB parameters is primarily a concern when random access point (RAP) pictures, such as clean random access (CRA) pictures or broken link access (BLA) pictures, are used to initialize the HRD. Thus, the techniques may provide improved support for RAP pictures in video coding.

Video decoder 30 receives a bitstream representing a plurality of pictures, the plurality of pictures including one or more CRA pictures or BLA pictures, and also receives a message indicating whether an alternative set of CPB parameters is to be used for at least one of the CRA pictures or the BLA pictures. The message may be received from network entity 29 or another external device, such as a processing device included in a streaming server or an intermediate network element.

Based on the received message, video decoder 30 sets a variable defined to indicate a set of CPB parameters for a given one of a CRA picture or a BLA picture. The video coding device then selects a set of CPB parameters for the given one of the CRA picture or the BLA picture based on the variable for the picture. Video decoder 30 applies the selected set of CPB parameters to CPB 68 to ensure that CPB 68 will not overflow during video decoding. In some cases, video decoder 30 may set a network abstraction layer (NAL) unit type for the given one of the CRA picture or the BLA picture. Video decoder 30 may set the NAL unit type for the picture to the signaled one, or may set the NAL unit type based on the variable for the picture. Video decoder 30 may then select a set of CPB parameters for the given picture based on the NAL unit type for the picture and the variable.

In general, this disclosure describes techniques that provide improved support for RAP pictures, including improved methods for selecting HRD parameters for RAP pictures and treating CRA pictures as BLA pictures. As described above, video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including its scalable video coding (SVC) and multi-view video coding (MVC) extensions. In addition, there is a new video coding standard, High Efficiency Video Coding (HEVC), being developed by the Joint Collaborative Team on Video Coding (JCT-VC) of the ITU-T Video Coding Expert Group (VCEG) and the ISO/IEC Motion Picture Expert Group (MPEG). A recent working draft (WD) of HEVC (hereinafter referred to as HEVC WD8) is described in document JCTVC-J1003_d7, Bross et al., “High Efficiency Video Coding (HEVC) Text Specification Draft 8,” of the 10th Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Stockholm, Sweden, July 11-20, 2012, available as of September 20, 2012 at http://phenix.int-evry.fr/jct/doc_end_user/documents/10_Stockholm/wg11/JCTVC-J1003-v8.zip.

Random access refers to starting the decoding of a video bitstream from a coded picture that is not the first coded picture in the bitstream. Random access to the bitstream is required in many video applications such as broadcasting and streaming, for example, for users to tune to a program at any time, switch between different channels, jump to a specific part of the video, or switch to a different bitstream for stream adaptation of bit rate, frame rate, spatial resolution, and the like. This feature is enabled by inserting random access pictures or random access points into the video bitstream many times at regular intervals.

Bitstream splicing refers to the concatenation of two or more bitstreams or portions thereof. For example, a first bitstream may be appended to a second bitstream, possibly with some modifications to one or both of the bitstreams to produce a spliced bitstream. The first coded picture in the second bitstream is also referred to as the splice point. Thus, pictures after the splice point in the spliced bitstream originate from the second bitstream, while pictures before the splice point in the spliced bitstream originate from the first bitstream.

The bitstream splicing is performed by a bitstream splicer. Bitstream splicers are often lightweight and much less intelligent than encoders. For example, a bitstream splicer may not be equipped with entropy decoding and encoding capabilities. Bitstream switching can be used in adaptive streaming environments. A bitstream switching operation at a picture in the bitstream being switched to is actually a bitstream splicing operation, where the splicing point is the bitstream switching point (i.e., the first picture from the bitstream being switched to).

Instantaneous decoding refresh (IDR) pictures, as specified in AVC or HEVC, can be used for random access. However, since pictures following an IDR picture in decoding order cannot use pictures decoded before the IDR picture as references, bitstreams that rely on IDR pictures for random access can have significantly lower coding efficiency. To improve coding efficiency, the concept of clean random access (CRA) pictures was introduced in HEVC to allow pictures following a CRA picture in decoding order but preceding the CRA picture in output order to use pictures decoded before the CRA picture as reference pictures.

A picture that follows a CRA picture in decoding order but precedes the CRA picture in output order is called a preceding picture associated with the CRA picture or a preceding picture of the CRA picture. A preceding picture of a CRA picture is correctly decodable if decoding starts from an IDR or CRA picture that precedes the current CRA picture. A preceding picture of a CRA picture may be non-decodable when random access occurs from the current CRA picture. Therefore, preceding pictures are typically discarded during random access decoding. To prevent error propagation from reference pictures that may be unavailable (depending on where decoding starts), all pictures that follow a CRA picture in both decoding and output order should not use any picture (including the preceding picture) that precedes the CRA picture in either decoding or output order as a reference picture.

Following the introduction of CRA pictures, HEVC further introduced the concept of broken link access (BLA) pictures, which are based on the concept of CRA pictures. BLA pictures typically originate from a bitstream spliced at the location of a CRA picture, and in the spliced bitstream, the splice-point CRA picture is changed to a BLA picture. IDR pictures, CRA pictures, and BLA pictures are collectively referred to as random access point (RAP) pictures or intra random access point (IRAP) pictures.

The main differences between BLA pictures and CRA pictures are discussed below. For a CRA picture, if decoding starts with a RAP picture that precedes the CRA picture in decoding order, the associated preceding picture is correctly decodable; and when random access occurs from a CRA picture (i.e., when decoding starts with a CRA picture, or in other words, when the CRA picture is the first picture in the bitstream), the associated preceding picture may not be correctly decodable. For a BLA picture, the associated preceding picture may not be decodable in all cases, even when decoding starts with a RAP picture that precedes the BLA picture in decoding order.

For a particular CRA or BLA picture, some of the associated preceding pictures are correctly decodable even when the CRA or BLA picture is the first picture in the bitstream. Such preceding pictures are referred to as decodable preceding pictures (DLPs), and other preceding pictures are referred to as non-decodable preceding pictures (NLPs). In some cases, DLPs are alternatively referred to as random access decodable preceding (RADL) pictures. In HEVC WD8, NLPs are referred to as marked as discarded (TFD) pictures. In other cases, NLPs are alternatively referred to as random access skipped preceding (RASL) pictures. For the purposes of the present invention, the terms "non-decodable preceding pictures", "TFD pictures" and "RASL pictures" are used interchangeably.

In HEVC WD8, the Hypothetical Reference Decoder (HRD) is specified in Annex C. The HRD relies on HRD parameters (which may be provided in the bitstream in the hrd_parameters() syntax structure contained in the Video Parameter Set (VPS) and/or Sequence Parameter Set (SPS), a buffering period supplemental enhancement information (SEI) message, and a picture timing SEI message. The buffering period SEI message primarily includes CPB parameters, namely the initial coded picture buffer (CPB) removal delay and the initial CPB removal delay offset. Two sets of CPB parameters are provided: a default set, signaled by the syntax elements initial_cpb_removal_delay[] and initial_cpb_removal_delay_offset[], and an alternative set, signaled by the syntax elements initial_alt_cpb_removal_delay[] and initial_alt_cpb_removal_delay_offset[].

When sub_pic_cpb_params_present_flag is equal to 0 and rap_cpb_params_present_flag is equal to 1, the following applies. When an HRD is initialized with a BLA picture that does not have an associated TFD picture in the bitstream, video decoder 30 uses an alternative set of CPB parameters to define the CPB 68. A BLA picture that does not have an associated non-decodable preceding picture has a nal_unit_type that indicates a BLA picture with a decodable preceding picture (e.g., BLA_W_DLP) or a BLA picture with no preceding picture (e.g., BLA_N_LP). If the default set is used instead, the CPB may overflow. When an HRD is initialized with a CRA picture or a BLA picture that has an associated TFD picture, video decoder 30 uses the default set of CPB parameters to define the CPB 68. A BLA picture that has an associated TFD picture has a nal_unit_type that indicates a BLA picture with a non-decodable preceding picture (e.g., BLA_W_TFD). This situation is reflected in the following text in subclause C.2.1 of HEVC WD8:

The variables InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set as follows.

- If any of the following conditions is true, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_alt_cpb_removal_delay[SchedSelIdx] and initial_alt_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message:

- Access unit 0 is a BLA access unit of which the coded picture has nal_unit_type equal to BLA_W_DLP or BLA_N_LP, and the value of rap_cpb_params_present_flag of the associated buffering period SEI message is equal to 1;

-SubPicCpbFlag is equal to 1.

Otherwise, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message.

As can be seen above, the selection of which set of CPB parameters to use for a given picture may be based on the value of the picture's nal_unit_type.

HEVC WD8 also includes the following text in subclause 8.1 for handling CRA pictures as BLA pictures.

When the current picture is a CRA picture, the following applies.

- If some external device not specified in this specification is available to set the variable HandleCraAsBlaFlag to a value, then HandleCraAsBlaFlag is set to the value provided by the external device.

- Otherwise, set the value of HandleCraAsBlaFlag to 0.

When HandleCraAsBlaFlag is equal to 1, the following applies during the parsing and decoding process of each coded slice NAL unit:

- Set the value of nal_unit_type to BLA_W_TFD.

- Set the value of no_output_of_prior_pics_flag to 1.

In HEVC WD8, a CRA picture has nal_unit_type equal to CRA_NUT in the NAL unit header of its coded slices, and it may have associated TFD pictures and DLP pictures.

The following problems are associated with existing methods for selecting CPB parameters for CRA pictures, BLA pictures, and CRA pictures treated as BLA pictures. The first problem is associated with selecting CPB parameters for CRA pictures and BLA pictures. CRA pictures may have associated TFD pictures. When a CRA picture has an associated TFD picture in the original bitstream, but the associated TFD picture is discarded by the streaming server or an intermediate network element, in order to enable selection of an appropriate set (i.e., an alternative set) of CPB parameters, network entity 29 or another external device must change the CRA picture to a BLA picture before sending the CRA picture to video decoder 30. However, network entity 29 may be unable to do this. In such cases, selection of an appropriate set of initial CPB removal delays and offsets may fail, which may result in overflow of CPB 68; or TFD picture discarding may fail, which may result in wasted bandwidth or lower video quality.

A second issue is associated with treating a CRA picture as a BLA picture. A CRA picture may have an associated TFD picture. When a CRA picture has an associated TFD picture in the original bitstream, but the associated TFD picture is discarded by network entity 29 or another external device (e.g., a processing device included in a streaming server or an intermediate network element), the external device instructs the CRA picture to be treated as a BLA picture. As specified in HEVC WD8, video decoder 30 then sets the value of nal_unit_type to indicate a BLA picture with a non-decodable preceding picture (e.g., BLA_W_TFD), which results in the use of the default set of CPB parameters, and therefore CPB 68 may overflow.

The techniques of this disclosure provide improved RAP picture behavior that can eliminate or avoid the problems described above. According to the techniques, variables are defined outside the scope of the video coding specification, and their values can be set by network entity 29 or another external device (e.g., a processing device included in a streaming server, an intermediate network element, or another network entity). In one example, the variables can specify whether to use an alternative set of CPB parameters and what NAL unit type to use when handling CRA pictures as BLA pictures. In another example, the variables can specify a NAL unit type value to be used for a particular picture, from which the default set or the alternative set of CPB parameters can be derived.

In the following sections, the above techniques are described in more detail. Underlining may indicate additions relative to HEVC WD8 and may indicate deletions relative to HEVC WD8.

In one example, video decoder 30 receives a bitstream representing a plurality of pictures, the plurality of pictures including one or more CRA pictures or BLA pictures. Video decoder 30 also receives a message from network entity 29 indicating whether an alternative set of CPB parameters is to be used for at least one of the CRA pictures or the BLA pictures. Video decoder 30 sets a variable based on the received message, the variable being defined to indicate a set of CPB parameters to be used for a given one of the CRA pictures or the BLA pictures. Video decoder 30 then selects a set of CPB parameters to be used for the given one of the CRA pictures or the BLA pictures based on the picture-specific variable.

According to this example, a variable UseAltCpbParamsFlag may be defined for each BLA or CRA picture. The value of this variable is set to 0 or 1 by network entity 29 or some other external device. If this external device is not available, video decoder 30 may set the value of the variable to 0.

In this case, the text in subclause 8.1 of HEVC WD8 cited above may be replaced by the following:

When the current picture is a BLA picture with nal_unit_type equal to BLA_W_TFD or a CRA picture, The following situations apply.

- If some external device not specified in this specification is available to set the variable UseAltCpbParamsFlag to a value, UseAltCpbParamsFlag is set to the value provided by the external device.

-Otherwise, set the value of UseAltCpbParamsFlag to 0.

When the current picture is a CRA picture, the following applies.

- If some external device not specified in this specification is available to set the variable HandleCraAsBlaFlag to a value, then HandleCraAsBlaFlag is set to the value provided by the external device.

- Otherwise, set the value of HandleCraAsBlaFlag to 0.

When the current picture is a CRA picture and HandleCraAsBlaFlag is equal to 1, during the parsing and decoding process for each coded slice NAL unit, the following applies, and the CRA picture is treated as a BLA picture and the CRA access unit is treated as a BLA access unit:

- If UseAltCpbParamsFlag is equal to 0, set the value of nal_unit_type to BLA_W_TFD. Otherwise, set the value of nal_unit_type to BLA_W_DLP.

- Set the value of no_output_of_prior_pics_flag to 1.

Additionally, the text in subclause C.2.1 of HEVC WD8 cited above may be replaced by the following:

The variables InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set as follows.

- If one of the following conditions is true, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_alt_cpb_removal_delay[SchedSelIdx] and initial_alt_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message:

- Access unit 0 is a BLA access unit of which the coded picture has nal_unit_type equal to BLA_W_DLP or BLA_N_LP, and the value of rap_cpb_params_present_flag of the associated buffering period SEI message is equal to 1;

- Access unit 0 is a BLA access unit of a coded picture with nal_unit_type equal to BLA_W_TFD or It is a CRA access unit, UseAltCpbParamsFlag is equal to 1, and the rap_cpb_ The value of params_present_flag is equal to 1;

-SubPicCpbFlag is equal to 1.

Otherwise, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message.

Network entity 29 or another external device configured to set the value of UseAltCpbParamsFlag may function as follows. Network entity 29 may send a message to video decoder 30 or to a receiver containing video decoder 30. The message may indicate that a particular BLA or CRA picture has an associated TFD picture but the associated TFD picture is discarded and, therefore, an alternative set of CPB parameters should be used. Upon receiving this message, video decoder 30 may set the value of UseAltCpbParamsFlag for the particular BLA or CRA picture to 1. If the particular BLA or CRA does not have a TFD picture, or it has a TFD picture that is not discarded, then no message needs to be sent, or a message may be sent to direct video decoder 30 to set the value of UseAltCpbParamsFlag for the particular BLA or CRA picture to 0.

In some cases, video decoder 30 may set a network abstraction layer (NAL) unit type for a given one of a CRA picture or a BLA picture, and may select a set of CPB parameters for a given picture based on the NAL unit type and variables for the picture. As another example, instead of using only one NAL unit type (e.g., CRA_NUT) that indicates a general CRA picture, the techniques of this disclosure allow the use of three different NAL unit types that respectively indicate a CRA picture with a non-decodable preceding picture (e.g., CRA_W_TFD), a CRA picture with a decodable preceding picture (e.g., CRA_W_DLP), and a CRA picture with no preceding picture (e.g., CRA_N_LP). In this case, Table 7-1 in HEVC WD8 and the notes below the table are changed as shown below.

Table 7-1 - NAL unit type codes and NAL unit type classes

NOTE 3 - A CRA picture with nal_unit_type equal to CRA_W_TFD may have an associated TFD picture or an associated DLP picture, or both, present in the bitstream. A CRA picture with nal_unit_type equal to CRA_W_DLP does not have an associated TFD picture present in the bitstream, but may have an associated DLP picture in the bitstream. A CRA picture with nal_unit_type equal to CRA_N_LP does not have an associated preceding picture present in the bitstream.

NOTE 4 - A BLA picture with nal_unit_type equal to BLA_W_TFD may have an associated TFD picture or an associated DLP picture, or both, present in the bitstream. A BLA picture with nal_unit_type equal to BLA_W_DLP does not have an associated TFD picture present in the bitstream, but may have an associated DLP picture in the bitstream. A BLA picture with nal_unit_type equal to BLA_N_LP does not have an associated preceding picture present in the bitstream.

NOTE 5 - An IDR picture with nal_unit_type equal to IDR_N_LP does not have an associated preceding picture present in the bitstream. An IDR picture with nal_unit_type equal to IDR_W_DLP does not have an associated TFD picture present in the bitstream, but may have an associated DLP picture in the bitstream.

In addition, similar to the first example above, a variable UseAltCpbParamsFlag is defined for each BLA or CRA picture. The value of this variable is set to 0 or 1 by network entity 29 or another external device. If this external device is not available, video decoder 30 may set the value of the variable to 0.

In this case, the text in subclause 8.1 of HEVC WD8 cited above may be replaced by the following:

When the current picture is a BLA picture with nal_unit_type equal to BLA_W_TFD or a CRA_ When using a CRA picture with nal_unit_type of W_TFD, the following applies.

- If some external device not specified in this specification is available to set the variable UseAltCpbParamsFlag to a value, UseAltCpbParamsFlag is set to the value provided by the external device.

-Otherwise, set the value of UseAltCpbParamsFlag to 0.

When the current picture is a CRA picture, the following applies.

- If some external device not specified in this specification is available to set the variable HandleCraAsBlaFlag to a value, then HandleCraAsBlaFlag is set to the value provided by the external device.

- Otherwise, set the value of HandleCraAsBlaFlag to 0.

When the current picture is a CRA picture and HandleCraAsBlaFlag is equal to 1, during the parsing and decoding process for each decoded slice NAL unit, the following applies, and the CRA picture is treated as a BLA picture and the CRA access unit is treated as a BLA access unit:

- If the value of nal_unit_type is equal to CRA_W_TFD, then the value of nal_unit_type is set to BLA_W_TFD. Otherwise, if the value of nal_unit_type is equal to CRA_W_DLP, then the value of nal_unit_type is set to BLA_W_DLP. Otherwise, the value of nal_unit_type is set to BLA_N_LP.

- Set the value of no_output_of_prior_pics_flag to 1.

Additionally, the text in subclause C.2.1 of HEVC WD8 cited above may be replaced by the following:

The variables InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set as follows.

- If one of the following conditions is true, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_alt_cpb_removal_delay[SchedSelIdx] and initial_alt_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message:

- Access unit 0 is a BLA access unit of which the coded picture has nal_unit_type equal to BLA_W_DLP or BLA_N_LP, and the value of rap_cpb_params_present_flag of the associated buffering period SEI message is equal to 1;

- Access unit 0 is a CRA access unit whose coded picture has nal_unit_type equal to CRA_W_DLP or VRA_N_LP , and the value of rap_cpb_params_present_flag of the associated buffering period SEI message is equal to 1;

- Access unit 0 is a BLA access unit whose coded picture has nal_unit_type equal to BLA_W_TFD or a CRA access unit whose coded picture has nal_unit_type equal to CRA_W_TFD, UseAltCpbParamsFlag is equal to 1, and the value of rap_cpb_params_present_flag of the associated buffering period SEI message is equal to 1;

-SubPicCpbFlag is equal to 1.

Otherwise, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message.

Network entity 29 or another external device configured to set the value of UseAltCpbParamsFlag may function as follows. Network entity 29 may send a message to video decoder 30 or a receiver containing video decoder 30. The message may indicate that a particular BLA or CRA picture has an associated TFD picture but the associated TFD picture is discarded and, therefore, an alternative set of CPB parameters should be used. Upon receiving this message, video decoder 30 may set the value of UseAltCpbParamsFlag for the particular BLA or CRA picture to 1. If the particular BLA or CRA does not have a TFD picture, or if it has a TFD picture but it is not discarded, then no message needs to be sent, or a message may be sent to direct video decoder 30 to set the value of UseAltCpbParamsFlag for the particular BLA or CRA picture to 0.

In another example, video decoder 30 receives a bitstream representing a plurality of pictures, the plurality of pictures including one or more CRA pictures or BLA pictures, and also receives a message from network entity 29 indicating a NAL unit type for at least one of the CRA pictures or BLA pictures. Video decoder 30 sets a variable based on the received message, the variable being defined to indicate the NAL unit type for a given one of the CRA pictures or BLA pictures. Video decoder 30 then sets the NAL unit type for the given one of the CRA pictures or BLA pictures and selects a set of CPB parameters for the given picture based on the NAL unit type.

According to this example, a variable UseThisNalUnitType may be defined for each CRA or BLA picture. The value of this variable is set by network entity 29 or some other external device. If such an external device is not available, video decoder 30 may set the value of the variable to the nal_unit_type of the CRA or BLA picture. In some examples, possible values for this variable are CRA_NUT, BLA_W_TFD, BLA_W_DLP, and BLA_N_LP. In other examples, possible values for this variable may include other nal_unit_types configured to indicate a normal CRA picture, a BLA picture with a non-decodable preceding picture, a BLA picture with a decodable preceding picture, and a BLA picture with no preceding picture.

In this case, the text in subclause 8.1 of HEVC WD8 cited above may be replaced by the following:

When the current picture is a BLA or CRA picture, the following applies.

- If some external device not specified in this specification is available to set the variable UseThisNalUnitType to a value, then UseThisNalUnitType is set to the value provided by the external device. For BLA pictures with nal_unit_type equal to BLA_N_LP, the external device may only set UseThisNalUnitType to BLA_N_LP; for BLA pictures with nal_unit_type equal to BLA_W_DLP, the external device may only set UseThisNalUnitType to BLA_W_DLP or BLA_N_LP; for BLA pictures with nal_unit_type equal to BLA_W_TFD, the external device may only set UseThisNalUnitType to one of BLA_W_TFD, BLA_W_DLP and BLA_N_LP; for BLA pictures, the external device should never set UseThisNalUnitType to indicate CRA pictures or any other picture type; for CRA pictures, the external device may set UseThisNalUnitType to one of CRA_NUT, BLA_W_TFD, BLA_W_DLP and BLA_N_LP instead of any other value.

- Otherwise, set the value of UseThisNalUnitType to the nal_unit_type of the current picture.

When the current picture is a CRA or BLA picture, during the parsing and decoding process for each coded slice NAL unit, the following applies:

- The value of nal_unit_type is set to UseThisNalUnitType, and the current picture or access unit is considered to be a CRA or BLA picture or access unit according to the value of nal_unit_type being equal to UseThisNalUnitType.

- If the current picture is a CRA picture before the above step and has been changed to a BLA picture, the value of no_output_of_prior_pics_flag is set to 1.

The text in subclause C.2.1 of HEVC WD8 cited above does not need to be changed.

As another example, instead of using only one NAL unit type (e.g., CRA_NUT) that indicates a general CRA picture, the techniques of this disclosure allow the use of three different NAL unit types that respectively indicate a CRA picture with a non-decodable preceding picture (e.g., CRA_W_TFD), a CRA picture with a decodable preceding picture (e.g., CRA_W_DLP), and a CRA picture with no preceding picture (e.g., CRA_N_LP). In this case, Table 7-1 in HEVC WD8 and the notes below the table are changed as described above.

Furthermore, similar to the second example described above, a variable, UseThisNalUnitType, is defined for each CRA or BLA picture. The value of this variable is set by network entity 29 or another external device. If this external device is not available, video decoder 30 may set the value of the variable to the nal_unit_type of the CRA or BLA picture. In some examples, possible values for this variable are CRA_W_TFD, CRA_W_DLP, CRA_N_LP, BLA_W_TFD, BLA_W_DLP, and BLA_N_LP. In other examples, possible values for this variable may include other nal_unit_types configured to indicate a CRA picture with a non-decodable preceding picture, a CRA picture with a decodable preceding picture, a CRA picture with no preceding picture, a BLA picture with a non-decodable preceding picture, a BLA picture with a decodable preceding picture, and a BLA picture with no preceding picture.

In this case, the text in subclause 8.1 of HEVC WD8 cited above may be replaced by the following:

When the current picture is a BLA or CRA picture, the following applies.

- If some external device not specified in this specification is available to set the variable UseThisNalUnitType to a value, then UseThisNalUnitType is set to the value provided by the external device.

For BLA pictures with nal_unit_type equal to BLA_N_LP, the external device may only set UseThisNalUnitType to BLA_N_LP; for BLA pictures with nal_unit_type equal to BLA_W_DLP, the external device may only set UseThisNalUnitType to BLA_W_DLP or BLA_N_LP; for BLA pictures with nal_unit_type equal to BLA_W_TFD, the external device may only set UseThisNalUnitType to one of BLA_W_TFD, BLA_W_DLP and BLA_N_LP; for BLA pictures, the external device should never set UseThisNalUnitType to indicate CRA pictures or any other picture type.

For CRA pictures with nal_unit_type equal to CRA_N_LP, the external device may only set UseThisNalUnitType to CRA_N_LP or BLA_N_LP; for CRA pictures with nal_unit_type equal to CRA_W_DLP, the external device may only set UseThisNalUnitType to CRA_W_DLP, CRA_N_LP, BLA_W_DLP, or BLA_N_LP; for CRA pictures with nal_unit_type equal to CRA_W_TFD, the external device may only set UseThisNalUnitType to CRA_W_TFD, CRA_W_DLP, CRA_N_LP, BLA_W_TFD, BLA_W_DLP, or BLA_N_LP.

- Otherwise, set the value of UseThisNalUnitType to the nal_unit_type of the current picture.

When the current picture is a CRA or BLA picture, during the parsing and decoding process for each coded slice NAL unit, the following applies:

- The value of nal_unit_type is set to UseThisNalUnitType, and the current picture or access unit is considered to be a CRA or BLA picture or access unit according to the value of nal_unit_type being equal to UseThisNalUnitType.

- If the current picture is a CRA picture before the above step and has been changed to a BLA picture, the value of no_output_of_prior_pics_flag is set to 1.

Additionally, the text in subclause C.2.1 of HEVC WD8 cited above may be replaced by the following:

The variables InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set as follows.

- If one of the following conditions is true, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_alt_cpb_removal_delay[SchedSelIdx] and initial_alt_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message:

- Access unit 0 is a BLA access unit of which the coded picture has nal_unit_type equal to BLA_W_DLP or BLA_N_LP, and the value of rap_cpb_params_present_flag of the associated buffering period SEI message is equal to 1;

- Access unit 0 is a CRA access unit whose coded picture has nal_unit_type equal to CRA_W_DLP or CRA_N_LP , and the value of rap_cpb_params_present_flag of the associated buffering period SEI message is equal to 1;

-SubPicCpbFlag is equal to 1.

Otherwise, InitCpbRemovalDelay[SchedSelIdx] and InitCpbRemovalDelayOffset[SchedSelIdx] are set to the values of the corresponding initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx], respectively, of the associated buffering period SEI message.

FIG4 is a block diagram illustrating an example destination device 100 configured to operate according to a hypothetical reference decoder (HRD). In this example, destination device 100 includes an input interface 102, a stream scheduler 104, a coded picture buffer (CPB) 106, a video decoder 108, a decoded picture buffer (DPB) 110, a rendering unit 112, and an output interface 114. Destination device 100 may substantially correspond to destination device 14 from FIG1. Input interface 102 may include any input interface capable of receiving a coded bitstream of video data, and may substantially correspond to input interface 28 from FIG1. For example, input interface 102 may include a receiver, a modem, a network interface such as a wired or wireless interface, a memory or storage interface, a drive for reading data from a disk (e.g., an optical drive interface or a magnetic media interface), or other interface components.

Input interface 102 may receive a coded bitstream including video data and provide the bitstream to stream scheduler 104. Stream scheduler 104 extracts video data units (e.g., access units and/or decoding units) from the bitstream and stores the extracted units in CPB 106. In this manner, stream scheduler 104 represents an example implementation of a hypothetical stream scheduler (HSS). CPB 106 may substantially conform to CPB 68 from FIG. 3 , except that, as shown in FIG. 4 , CPB 106 is separate from video decoder 108. In different examples, CPB 106 may be separate from video decoder 108 or integrated as part of video decoder 108.

The video decoder 108 includes a DPB 110. The video decoder 108 may substantially conform to the video decoder 30 from Figures 1 and 3. The DPB 110 may substantially conform to the DPB 82 from Figure 3. Thus, the video decoder 108 may decode the decoding units of the CPB 106. Furthermore, the video decoder 108 may output decoded pictures from the DPB 110. The video decoder 108 may pass the output pictures to a rendering unit 112. The rendering unit 112 may crop the pictures and then pass the cropped pictures to an output interface 114. The output interface 114 may, in turn, provide the cropped pictures to a display device, which may substantially conform to the display device 32 from Figure 1.

A display device may form part of destination device 100 and may be communicatively coupled to destination device 100. For example, the display device may comprise a screen, touch screen, projector, or other display unit that is integrated with destination device 100, or may comprise a separate display such as a television, monitor, projector, touch screen, or other device that is communicatively coupled to destination device 100. The communicative coupling may include, for example, a wired or wireless coupling via a coaxial cable, a composite video cable, a component video cable, a High-Definition Multimedia Interface (HDMI) cable, radio frequency broadcast, or other wired or wireless coupling.

FIG5 is a flowchart illustrating an example operation of selecting a set of coded picture buffer (CPB) parameters based on a variable that indicates the set of CPB parameters for a particular random access point (RAP) picture in a bitstream. The illustrated operation is described with respect to video decoder 30 from FIG3 , including CPB 68. In other examples, similar operations may be performed by video encoder 20 from FIG2 , including CPB 66, destination device 100 from FIG4 , including CPB 106 and video decoder 108, or other devices including a video encoder or video decoder having a CPB configured to operate according to HRD operation.

Video decoder 30 receives a bitstream including one or more CRA pictures or BLA pictures (120). Along with the bitstream, video decoder 30 also receives a message indicating whether an alternative set of CPB parameters is used for a particular one of the CRA or BLA pictures (122). More specifically, video decoder 30 may receive a message from an external device, such as network entity 29, that is capable of discarding a TFD picture associated with a particular picture and is also capable of notifying video decoder 30 that the TFD picture has been discarded.

For example, when a particular picture has a TFD picture in the original bitstream output from video encoder 20, and the TFD picture has been discarded by an external device, the message received by video decoder 30 indicates that an alternative set of CPB parameters is used for the particular picture. As another example, when the particular picture does not have a TFD picture in the original bitstream output from video encoder 20, or the particular picture has a TFD picture in the original bitstream, and the TFD picture has not been discarded by an external device, the message received by video decoder 30 does not indicate that an alternative set of CPB parameters is used for the particular picture. In this case, either a default set or an alternative set of CPB parameters may be used for either a CRA picture or a BLA picture based on the NAL unit type of the picture.

Video decoder 30 sets a variable (e.g., UseAltCpbParamsFlag) based on the received message, the variable being defined to indicate a set of CPB parameters for a particular picture (124). For example, when the received message indicates an alternative set of CPB parameters for a particular picture, video decoder 30 may set UseAltCpbParamsFlag to 1. Conversely, when the received message does not explicitly indicate an alternative set of CPB parameters for a particular picture, video decoder 30 may set UseAltCpbParamsFlag to 0. In some cases, video decoder 30 may not receive a message for at least one of a CRA picture or a BLA picture. Video decoder 30 may then set UseAltCpbParamsFlag to 0.

Video decoder 30 then sets the NAL unit type for the particular picture (126). In some cases, video decoder 30 may set the NAL unit type for the particular picture as signaled in the bitstream. In other cases, video decoder 30 may set the NAL unit type for the particular picture based at least in part on a variable for the picture. The NAL unit type selection operation is described in more detail below with respect to FIG. 6. Video decoder 30 selects a default set or an alternative set of CPB parameters for the particular picture based on the NAL unit type and the variable for the particular picture (128). In detail, video decoder 30 selects a default set of CPB parameters for one or more NAL unit types when the variable does not indicate an alternative set of CPB parameters; and selects an alternative set of CPB parameters for one or more NAL unit types when the variable indicates an alternative set of CPB parameters and for one or more different NAL unit types. The CPB parameter set selection operation is described in more detail below with respect to FIG.

FIG6 is a flowchart illustrating example operations for setting a network abstraction layer (NAL) unit type for a particular RAP picture based on a variable indicating a set of CPB parameters for the picture. The illustrated operations are described with respect to video decoder 30 from FIG3 including CPB 68. In other examples, similar operations may be performed by video encoder 20 from FIG2 including CPB 66, destination device 100 from FIG4 including CPB 106 and video decoder 108, or other devices including a video encoder or video decoder having a CPB configured to operate according to HRD operations.

Video decoder 30 receives a bitstream including one or more CRA pictures or BLA pictures (150). Video decoder 30 receives a message indicating whether to use an alternative set of CPB parameters for a particular one of the CRA pictures or BLA pictures (152). Video decoder 30 sets a variable defined to indicate a set of CPB parameters for the particular picture based on the received message (154).

When the particular picture is a BLA picture (the No branch of 156), video decoder 30 sets the NAL unit type for the particular BLA picture as signaled in the bitstream (158). When the particular picture is a CRA picture (the Yes branch of 156) and when the CRA picture is not handled as a BLA picture (the No branch of 160), video decoder 30 also sets the NAL unit type for the particular CRA picture as signaled in the bitstream (158).

Conventionally, when a CRA picture is handled as a BLA picture, the NAL unit type for the CRA picture is set to indicate a BLA picture with a non-decodable preceding picture (e.g., BLA_W_TFD), which results in the selection of a default set of CPB parameters for the picture. In some cases, a picture may not have an associated TFD picture, and using the default set of CPB parameters may result in overflow of the CPB. According to the techniques of this disclosure, when a particular picture is a CRA picture (the YES branch of 156 ) and the CRA picture is handled as a BLA picture (the YES branch of 160 ), video decoder 30 sets the NAL unit type for the particular picture based on a variable for the particular CRA picture.

For example, when the variable does not explicitly indicate an alternative set of CPB parameters (the NO branch of 162), video decoder 30 sets the NAL unit type for the particular picture to indicate a BLA picture with a non-decodable preceding picture (e.g., BLA_W_TFD), which indicates that the particular picture has an associated TFD picture (164). In this case, the default set of CPB parameters will be appropriately selected for the particular picture. When the variable indicates an alternative set of CPB parameters (the YES branch of 162), video decoder 30 sets the NAL unit type for the particular picture to indicate a BLA picture with a decodable preceding picture (e.g., BLA_W_DLP), which indicates that the particular picture does not have an associated TFD picture (166). In this case, the alternative set of CPB parameters will be appropriately selected for the particular picture. In this way, the techniques ensure that the CPB of the video decoder will not overflow due to the use of inappropriate CPB parameters.

FIG7 is a flowchart illustrating an example operation of selecting a set of CPB parameters for a particular RAP picture based on the NAL unit type for the picture and a variable indicating the set of CPB parameters for the picture. The illustrated operation is described with respect to video decoder 30 from FIG3 including CPB 68. In other examples, similar operations may be performed by video encoder 20 from FIG2 including CPB 66, destination device 100 from FIG4 including CPB 106 and video decoder 108, or other devices including a video encoder or video decoder having a CPB configured to operate according to HRD operation.

Video decoder 30 receives a bitstream that includes one or more CRA pictures or BLA pictures (170). Video decoder 30 receives a message indicating whether an alternative set of CPB parameters is used for a particular one of the CRA pictures or BLA pictures (172). Video decoder 30 sets a variable based on the received message, the variable being defined to indicate the set of CPB parameters for the particular picture (174). Video decoder 30 then sets a NAL unit type for the particular picture (176). As described above with respect to FIG. 6 , video decoder 30 may set the NAL unit type for the particular picture as signaled in the bitstream, or may set the NAL unit type for the particular picture based on a variable for the picture.

When a particular picture is a BLA picture with a NAL unit type indicating a BLA picture with decodable preceding pictures (e.g., BLA_W_DLP) or a BLA picture with no preceding pictures (e.g., BLA_N_LP), which indicates that the particular picture does not have an associated TFD picture (the yes branch of 178), video decoder 30 selects an alternative set of CPB parameters for the particular picture based on the NAL unit type (180). Conventionally, the default set of CPB parameters is for any CRA picture or BLA picture with an associated TFD picture (e.g., BLA_W_TFD). However, in some cases, the TFD picture associated with the particular picture in the original bitstream may be discarded before the bitstream reaches the video decoder. The video decoder then uses the default CPB parameters based on the NAL unit type even when the picture no longer has an associated TFD picture, which may result in an overflow of the CPB.

According to the techniques of this disclosure, when a particular picture is a CRA picture or a BLA picture (which indicates that the particular picture has an associated TFD picture) having a NAL unit type (e.g., BLA_W_TFD) that indicates a BLA picture with a non-decodable preceding picture (the YES branch of 182), video decoder 30 selects a set of CPB parameters for the particular picture based on a variable for the particular picture. For example, when the variable does not explicitly indicate an alternative set of CPB parameters (the NO branch of 184), video decoder 30 selects a default set of CPB parameters for the particular picture based on the variable (186). When the variable indicates an alternative set of CPB parameters (the YES branch of 184), video decoder 30 selects an alternative set of CPB parameters for the particular picture based on the variable (188). In this way, the techniques ensure that the CPB of the video decoder does not overflow due to the use of inappropriate CPB parameters.

FIG8 is a flowchart illustrating an example operation of selecting a set of CPB parameters based on a variable defined to indicate a network abstraction layer (NAL) unit type for a particular RAP picture in a bitstream. The illustrated operation is described with respect to video decoder 30 from FIG3 including CPB 68. In other examples, similar operations may be performed by video encoder 20 from FIG2 including CPB 66, destination device 100 from FIG4 including CPB 106 and video decoder 108, or other devices including a video encoder or video decoder having a CPB configured to operate according to HRD operation.

Video decoder 30 receives a bitstream (190) that includes one or more CRA pictures or BLA pictures. Along with the bitstream, video decoder 30 also receives a message that indicates the NAL unit type for a particular one of the CRA or BLA pictures (192). More specifically, video decoder 30 may receive a message from an external device, such as network entity 29, that is capable of discarding a TFD picture associated with a particular picture and that is also capable of notifying video decoder 30 that the TFD picture has been discarded.

For example, when a specific picture has a TFD picture in the original bitstream output from video encoder 20, and the TFD picture has been discarded by an external device, the message received by video decoder 30 may indicate a NAL unit type for the specific picture, the NAL unit type indicating a BLA picture with a decodable preceding picture (e.g., BLA_W_DLP), or a BLA picture with no preceding picture (e.g., BLA_N_LP). As another example, when the specific picture has a TFD picture in the original bitstream and the TFD picture has not been discarded by an external device, the message received by video decoder 30 may indicate a NAL unit type for one of a CRA picture or a BLA picture, the NAL unit type indicating a BLA picture with a non-decodable preceding picture (e.g., BLA_W_TFD).

Video decoder 30 sets a variable (e.g., UseThisNalUnitType) based on the received message, the variable being defined to indicate a NAL unit type for a particular picture (194). For example, video decoder 30 may set UseThisNalUnitType equal to the NAL unit type indicated by the received message for the particular picture. In some cases, video decoder 30 may not receive a message for at least one of a CRA picture or a BLA picture. Video decoder 30 may then set UseThisNalUnitType equal to the NAL unit type signaled in the bitstream for the particular picture. Video decoder 30 sets the NAL unit type for the particular picture based on the variable (196). Video decoder 30 then selects a default set or an alternative set of CPB parameters for the particular picture based on the NAL unit type for the particular picture (198).

9 is a block diagram illustrating an example set of devices that form part of network 200. In this example, network 200 includes routing devices 204A, 204B (routing devices 204) and transcoding device 206. Routing devices 204 and transcoding device 206 are intended to represent a small number of devices that may form part of network 200. Other network devices, such as switches, hubs, gateways, firewalls, bridges, and other such devices, may also be included within network 200. Furthermore, additional network devices may be provided along the network path between server device 202 and client device 208. In some examples, server device 202 may correspond to source device 12 of FIG. 1, while client device 208 may correspond to destination device 14 of FIG. 1.

Generally speaking, routing device 204 implements one or more routing protocols to exchange network data via network 200. In some instances, routing device 204 may be configured to perform proxy or caching operations. Therefore, in some instances, routing device 204 may be referred to as a proxy device. Generally speaking, routing device 204 executes routing protocols to discover routes through network 200. By executing such routing protocols, routing device 204B may discover a network route from itself via routing device 204A to server device 202.

The techniques of this disclosure can be implemented by network devices such as routing device 204 and transcoding device 206, but can also be implemented by client device 208. In this manner, routing device 204, transcoding device 206, and client device 208 represent examples of devices configured to perform the techniques of this disclosure, including those recited in the claims section of this disclosure. Furthermore, the devices of FIG. 1 , the encoder shown in FIG. 2 , and the decoder shown in FIG. 3 are also exemplary devices that can be configured to perform the techniques of this disclosure, including those recited in the claims section of this disclosure.

It should be appreciated that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary to practice the techniques). Furthermore, in some examples, actions or events may be performed simultaneously rather than sequentially, for example, via multithreading, interrupt processing, or multiple processors.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media (which corresponds to tangible media such as data storage media) or communication media, which includes, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) non-transitory, tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and/or data structures for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may be properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (e.g., infrared, radio, and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (e.g., infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but rather refer to non-transitory, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Alternatively, the techniques may be fully implemented in one or more circuits or logic components.

The techniques of this disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Specifically, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) in conjunction with appropriate software and/or firmware.

Various embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims (33)

1. A method for processing video data, the method comprising: Receive a bitstream representing multiple images, the multiple images including one or more Clean Random Access (CRA) images or one or more Broken Link Access (BLA) images, the one or more CRA images having an associated NAL cell type indicating that the CRA image has a TFD image as an associated discarded image, the one or more BLA images having an NAL cell type indicating that the BLA image has an associated TFD image, wherein the TFD image is a non-decodeable front image; Receive a message from an external device indicating whether the TFD image of at least one of the one or more CRA images or the one or more BLA images is discarded by the external device, and therefore whether an alternative set of decoded image buffer CPB parameters is used for the one or more CRA images or the one or more BLA images. Based on the received message, a variable is defined to indicate a set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images, such that the variable indicates the alternative set of CPB parameters when the TFD image is discarded by the external device; and when the variable indicates the alternative set of CPB parameters, the alternative set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images is selected, or when the variable does not indicate the alternative set of CPB parameters, a default set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images is selected. 2. The method of claim 1, further comprising initializing the HRD using at least one of the one or more CRA images or the one or more BLA images and associated hypothetical reference decoder HRD parameters, wherein the HRD parameters comprise the selected set of CPB parameters. 3. The method of claim 1, further comprising setting a Network Abstraction Layer (NAL) unit type for at least one of the one or more CRA images or the one or more BLA images, wherein selecting one of the default set or the alternative set of CPB parameters comprises selecting one of the default set or the alternative set of CPB parameters for the one or more CRA images or the one or more BLA images based on the NAL unit type and the value of the variable. 4. The method of claim 3, wherein the one or more CRA images or at least one of the one or more BLA images includes a CRA image treated as a BLA image, and wherein setting the NAL unit type includes setting the NAL unit type for the CRA image treated as the BLA image based on the value of the variable. 5. The method of claim 4, wherein setting the NAL unit type for the CRA image processed as the BLA image includes: Based on the values of the variables indicating the use of the CPB parameters, the NAL unit type for the CRA image treated as the BLA image is set to indicate the BLA image having an associated decodeable preceding image; and Based on the values of the variables in the alternative set that do not indicate the use of CPB parameters, the NAL cell type for the CRA image disposed of as the BLA image is set to indicate the BLA image with the associated TFD image. 6. The method of claim 3, wherein at least one of the one or more CRA images or the one or more BLA images includes a CRA image, and wherein setting the NAL unit type includes setting the NAL unit type for the CRA image to indicate a general CRA image. 7. The method of claim 3, wherein at least one of the one or more CRA images or the one or more BLA images includes a CRA image, and wherein setting the NAL unit type includes setting the NAL unit type for the CRA image to indicate one of a CRA image with an associated non-decorable front image, a CRA image with an associated decorable front image, or a CRA image without a front image. 8. The method of claim 1, wherein at least one of the one or more CRA images or the one or more BLA images does not have an associated non-decodeable pre-image in the original bitstream, or has an associated non-decodeable pre-image in the original bitstream and the associated non-decodeable pre-image has not been discarded by the external device, and wherein the value of the variable does not indicate the alternative set of CPB parameters used for at least one of the one or more CRA images or the one or more BLA images. 9. The method of claim 1, further comprising: Based on not receiving messages indicating whether to use the alternative set of CPB parameters for the one or more CRA images or the other of the one or more BLA images, the value of the variable is set to not indicate the use of the alternative set of CPB parameters for the one or more CRA images or the other of the one or more BLA images; and The default set of CPB parameters for the other of the one or more CRA images or the one or more BLA images is selected based on the value of the variable. 10. The method of claim 1, wherein each of the default set of CPB parameters and the alternative set of CPB parameters includes an initial CPB removal delay and an initial CPB removal delay offset. 11. The method of claim 1, further comprising applying the selected set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images to a CPB contained in a video decoding device to ensure that the CPB does not overflow during decoding of the video data. 12. The method of claim 1, further comprising applying the selected set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images to a first CPB included in the video encoding apparatus to ensure that the first CPB included in the video encoding apparatus does not overflow during encoding the video data, and to ensure that a second CPB included in the video decoding apparatus does not overflow upon receiving the encoded bitstream generated by the video encoding apparatus. 13. A video decoding apparatus for processing video data, the apparatus comprising: The decoded image buffer CPB is configured to store video data; and One or more processors, configured to: Receive a bitstream representing a plurality of images comprising one or more Clean Random Access (CRA) images or one or more Broken Link Access (BLA) images, wherein the one or more CRA images have NAL cell types indicating that the CRA images have associated TFD images marked as discarded, and the one or more BLA images have NAL cell types indicating that the BLA images have associated TFD images, wherein the TFD images are non-decodeable front images; Receive a message from an external device indicating whether the TFD image of at least one of the one or more CRA images or the one or more BLA images is discarded by the external device, and therefore whether an alternative set of CPB parameters is used for at least one of the one or more CRA images or the one or more BLA images. Based on the received message, a variable is set, defined to indicate a set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images, such that the variable indicates the alternative set of CPB parameters when the TFD image is discarded by the external device; and When the variable indicates the alternative set of CPB parameters, the alternative set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images is selected; or when the variable does not indicate the alternative set of CPB parameters, the default set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images is selected. 14. The video decoding apparatus of claim 13, wherein the one or more processors are further configured to initialize an HRD using at least one of the one or more CRA images or the one or more BLA images and associated hypothetical reference decoder HRD parameters, wherein the HRD parameters include the selected set of CPB parameters for the images. 15. The video decoding apparatus of claim 13, wherein the one or more processors are further configured to set a Network Abstraction Layer (NAL) unit type for at least one of the one or more CRA images or the one or more BLA images, and to select, based on the NAL unit type and the value of the variable, one of the default set or the alternative set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images. 16. The video decoding apparatus of claim 15, wherein the one or more CRA images or at least one of the one or more BLA images includes a CRA image treated as a BLA image, and wherein the one or more processors are further configured to set the NAL unit type for the CRA image treated as the BLA image based on the value of the variable. 17. The video decoding apparatus according to claim 16, wherein: Based on the values of the variables indicating the use of the CPB parameters, the one or more processors are further configured to set the NAL unit type for the CRA image treated as the BLA image to indicate a BLA image with an associated decodeable preceding image; and Based on the values of the variables in the alternative set that do not indicate the use of CPB parameters, the one or more processors are further configured to set the NAL unit type for the CRA image disposed of as the BLA image to indicate the BLA image with an associated TFD image. 18. The video decoding apparatus of claim 15, wherein at least one of the one or more CRA images or the one or more BLA images comprises a CRA image, and wherein the one or more processors are further configured to set the NAL unit type for the CRA image to indicate a general CRA image. 19. The video decoding apparatus of claim 15, wherein at least one of the one or more CRA images or the one or more BLA images comprises a CRA image, and wherein the one or more processors are further configured to set the NAL unit type for the CRA image to indicate one of a CRA image having an associated non-decoding front image, a CRA image having an associated decoding front image, or a CRA image without a front image. 20. The video decoding apparatus of claim 13, wherein at least one of the one or more CRA images or the one or more BLA images does not have an associated non-decodeable pre-image in the original bitstream, or has an associated non-decodeable pre-image in the original bitstream and the associated non-decodeable pre-image has not been discarded by the external device, and wherein the value of the variable does not indicate the alternative set of CPB parameters used for at least one of the one or more CRA images or the one or more BLA images. 21. The video decoding apparatus of claim 13, wherein the one or more processors are further configured to: Based on not receiving messages indicating whether to use the alternative set of CPB parameters for the one or more CRA images or the other of the one or more BLA images, the value of the variable is set to not indicate the use of the alternative set of CPB parameters for the one or more CRA images or the other of the one or more BLA images; and The default set of CPB parameters for the other of the one or more CRA images or the one or more BLA images is selected based on the value of the variable. 22. The video decoding apparatus of claim 13, wherein each of the default set of CPB parameters and the alternative set of CPB parameters includes an initial CPB removal delay and an initial CPB removal delay offset. 23. The video decoding apparatus of claim 13, wherein the one or more processors are further configured to apply the selected set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images to the CPB contained in the video decoding apparatus to ensure that the CPB does not overflow during decoding of the video data. 24. The video decoding apparatus of claim 13, wherein the one or more processors are further configured to apply the selected set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images to a first CPB included in the video encoding apparatus to ensure that the first CPB included in the video encoding apparatus does not overflow during encoding the video data, and to ensure that a second CPB included in the video decoding apparatus does not overflow upon receiving the encoded bitstream generated by the video encoding apparatus. 25. A video decoding apparatus for processing video data, the apparatus comprising: A means for receiving a bitstream representing a plurality of images, the plurality of images comprising one or more Clean Random Access (CRA) images or one or more Broken Link Access (BLA) images, the one or more CRA images having NAL cell types indicating that the CRA image has an associated TFD image marked as discarded, and the one or more BLA images having NAL cell types indicating that the BLA image has an associated TFD image, wherein the TFD image is a non-decodeable front image; A means for receiving a message from an external device, the message indicating whether the TFD image of at least one of the one or more CRA images or the one or more BLA images is discarded by the external device, and therefore whether an alternative set of decoded image buffer CPB parameters is used for at least one of the one or more CRA images or the one or more BLA images. A means for setting variables based on received messages, the variables being defined to indicate a set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images, such that the variables indicate the alternative set of CPB parameters when the TFD image is discarded by the external device; and A means for selecting the alternative set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images when the variable indicates the alternative set of CPB parameters, or for selecting the default set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images when the variable does not indicate the alternative set of CPB parameters. 26. The video decoding apparatus of claim 25, further comprising means for initializing an HRD using at least one of the one or more CRA images or the one or more BLA images and associated hypothetical reference decoder HRD parameters, wherein the HRD parameters comprise the selected set of CPB parameters. 27. The video decoding apparatus of claim 25, further comprising means for setting a Network Abstraction Layer (NAL) unit type for at least one of the one or more CRA images or the one or more BLA images, and means for selecting, based on the NAL unit type and the value of the variable, one of the default set or the alternative set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images. 28. The video decoding apparatus of claim 27, wherein the one or more CRA images or at least one of the one or more BLA images includes a CRA image treated as a BLA image, the video decoding apparatus further comprising means for setting the NAL unit type for the CRA image treated as the BLA image based on the value of the variable. 29. The video decoding apparatus of claim 25, further comprising means for applying the selected set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images to a CPB contained in the video decoding apparatus to ensure that the CPB does not overflow during decoding of the video data. 30. The video decoding apparatus of claim 25, further comprising means for applying the selected set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images to a first CPB included in the video encoding apparatus to ensure that the first CPB included in the video encoding apparatus does not overflow during encoding the video data and to ensure that a second CPB included in the video decoding apparatus does not overflow upon receiving the encoded bitstream generated by the video encoding apparatus. 31. A non-transitory computer-readable medium storing instructions for processing video data, said instructions, when executed, causing one or more processors to: Receive a bitstream representing multiple images, the multiple images including one or more Clean Random Access (CRA) images or one or more Broken Link Access (BLA) images, the one or more CRA images having an associated NAL cell type indicating that the CRA image has a TFD image as an associated discarded image, the one or more BLA images having an NAL cell type indicating that the BLA image has an associated TFD image, wherein the TFD image is a non-decodeable front image; Receive a message from an external device indicating whether the TFD image of at least one of the one or more CRA images or the one or more BLA images is discarded by the external device, and therefore whether an alternative set of decoded image buffer CPB parameters is used for at least one of the one or more CRA images or the one or more BLA images. Based on the received message, a variable is set, defined to indicate a set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images, such that the variable indicates the alternative set of CPB parameters when the TFD image is discarded by the external device; and When the variable indicates the alternative set of CPB parameters, the alternative set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images is selected; or when the variable does not indicate the alternative set of CPB parameters, the default set of CPB parameters for at least one of the one or more CRA images or the one or more BLA images is selected. 32. The non-transitory computer-readable medium of claim 31, wherein the instructions cause the one or more processors to set a Network Abstraction Layer (NAL) unit type for at least one of the one or more CRA images or the one or more BLA images, and to select, based on the NAL unit type and the value of the variable, one of the default set or the alternative set of CPB parameters for the one or more CRA images or the one or more BLA images. 33. The non-transitory computer-readable medium of claim 32, wherein the one or more CRA images or at least one of the one or more BLA images includes a CRA image treated as a BLA image, and wherein the instructions cause the one or more processors to set the NAL unit type for the CRA image treated as the BLA image based on the value of the variable.