HK1203009B - Low-delay video buffering in video coding - Google Patents

Description

Low-latency video buffering in video decoding

This application claims the benefit of U.S. Provisional Application No. 61/620,266, filed April 4, 2012, and U.S. Provisional Application No. 61/641,063, filed May 1, 2012, each of which is incorporated herein by reference in its entirety.

Technical Field

The present invention relates to video decoding.

Background Art

Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), portable or desktop computers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radiotelephones, video teleconferencing devices, and the like. Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including its scalable video coding (SVC) and multi-view video coding (MVC) extensions. In addition, High Efficiency Video Coding (HEVC) is a video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). A recent draft of the upcoming HEVC standard, referred to as “HEVC Working Draft 6” or “HEVC WD6,” is described in document JCTVC-H1003, “High efficiency video coding (HEVC) text specification draft 6,” by Bross et al. (Joint Collaborative Team on Video Coding (JCT-VC) of ITU-TSG16 WP3 and ISO/IEC JTC1/SC29/WG11, 8th Meeting: San Jose, California, USA, February 2012), which was available for download as of May 1, 2012, from http://phenix.int-evry.fr/jct/doc_end_user/documents/8San%20Jose/wg11/JCTVC-H1003-v22.zip.

Video compression techniques perform spatial prediction and/or temporal prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video frame or slice may be partitioned into macroblocks. Each macroblock may be further partitioned. Macroblocks in an intra-coded (I) frame or slice are encoded using spatial prediction with respect to neighboring macroblocks. Macroblocks in an inter-coded (P or B) frame or slice may use spatial prediction with respect to neighboring macroblocks in the same frame or slice, or temporal prediction with respect to other reference frames.

Summary of the Invention

In general, this disclosure describes various techniques for achieving reduced codec latency in an interoperable manner. In one example, these techniques can be implemented via a general sub-picture-based coded picture buffer (CPB) behavior.

In one example, a method of decoding video data includes storing one or more decoding units of the video data in a coded picture buffer (CPB). The method further includes obtaining respective buffer removal times for the one or more decoding units. The method further includes removing the decoding units from the CPB based on the obtained buffer removal times for each of the decoding units. The method further includes determining whether the CPB operates at an access unit level or a sub-picture level. The method further includes decoding video data corresponding to the removed decoding units. If the CPB operates at the access unit level, decoding the video data includes decoding the access units included in the decoding units. If the CPB operates at the sub-picture level, decoding the video data includes decoding a subset of the access units included in the decoding units.

In another example, a device for decoding video data is configured to store one or more decoding units of the video data in a coded picture buffer (CPB). The device is further configured to obtain respective buffer removal times for the one or more decoding units. The device is further configured to remove the decoding units from the CPB based on the obtained buffer removal times for each of the decoding units. The device is further configured to determine whether the CPB operates at an access unit level or a sub-picture level. The device is further configured to decode video data corresponding to the removed decoding units. If the CPB operates at an access unit level, decoding the video data includes decoding the access units included in the decoding unit. If the CPB operates at a sub-picture level, decoding the video data includes decoding a subset of the access units included in the decoding unit.

In another example, an apparatus for decoding video data includes means for storing one or more decoding units of video data in a coded picture buffer (CPB). The apparatus further includes means for obtaining respective buffer removal times for the one or more decoding units. The apparatus further includes means for removing the decoding units from the CPB according to the obtained buffer removal times for each of the decoding units. The apparatus further includes means for determining whether the CPB operates at an access unit level or a sub-picture level. The apparatus further includes means for decoding video data corresponding to the removed decoding units. If the CPB operates at an access unit level, decoding the video data includes decoding the access units included in the decoding unit. If the CPB operates at a sub-picture level, decoding the video data includes decoding a subset of the access units included in the decoding unit.

In another example, a computer-readable storage medium includes instructions stored thereon that, when executed, cause a processor to store one or more decoding units of video data in a coded picture buffer (CPB). The instructions further cause the processor to obtain respective buffer removal times for the one or more decoding units. The instructions further cause the processor to remove the decoding units from the CPB based on the obtained buffer removal times for each of the decoding units. The instructions further cause the processor to determine whether the CPB operates at an access unit level or a sub-picture level. The instructions further cause the processor to decode video data corresponding to the removed decoding units. If the CPB operates at an access unit level, decoding the video data includes decoding the access units included in the decoding units. If the CPB operates at a sub-picture level, decoding the video data includes decoding a subset of the access units included in the decoding units.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a block diagram illustrating an example video encoding and decoding system that may utilize techniques for deblocking edges between video blocks, according to the techniques of this disclosure.

2 is a block diagram illustrating an example of a video encoder that may implement techniques for deblocking edges between video blocks, according to the techniques of this disclosure.

3 is a block diagram illustrating an example of a video decoder that decodes an encoded video sequence, according to the techniques of this disclosure.

4 is a block diagram illustrating an example destination device that may implement any or all of the techniques of this disclosure.

5 is a flowchart illustrating an example method that includes a decoding unit removing video data from a picture buffer according to an obtained buffer removal time, according to the techniques of this disclosure.

6 is a flowchart illustrating another example method that includes a decoding unit removing video data from a picture buffer according to an obtained buffer removal time, according to the techniques of this disclosure.

7 is a flowchart illustrating another example method of processing video data, including outputting a cropped picture in a boosting process, according to the techniques of this disclosure.

DETAILED DESCRIPTION

Video applications can include local playback, streaming, broadcast/multicast, and conversation applications. Conversational applications can include video telephony and video conferencing and are also referred to as low-latency applications. Conversational applications require relatively low end-to-end latency across the entire system—that is, the delay between when a video frame is captured and when it is displayed. Typically, acceptable end-to-end latency for conversational applications should be less than 400 milliseconds (ms), with an end-to-end latency of approximately 150 ms considered excellent. Each processing step can contribute to the overall end-to-end latency, such as capture latency, pre-processing latency, encoding latency, transmission latency, receive buffering latency (for de-jittering), decoding latency, decoded picture output latency, post-processing latency, and display latency. Therefore, codec latency (encoding latency, decoding latency, and decoded picture output latency) should generally be minimized in conversational applications. In particular, the coding structure should ensure that the decoding order of pictures is identical to the output order, resulting in zero decoded picture output latency.

Video coding standards may include specifications for video buffering models. In AVC and HEVC, the buffering model is called the Hypothetical Reference Decoder (HRD), which includes buffering models for both the coded picture buffer (CPB) and the decoded picture buffer (DPB), with CPB and DPB behaviors mathematically specified. The HRD directly imposes constraints on different timings, buffer sizes, and bit rates, and indirectly on bitstream characteristics and statistics. The complete set of HRD parameters includes five basic parameters: initial CPB removal delay, CPB size, bit rate, initial DPB output delay, and DPB size.

In AVC and HEVC, bitstream conformance and decoder conformance are specified as part of the HRD specification. Although HRD is named as a type of decoder, HRD is generally required at the encoder to ensure bitstream conformance, but is generally not required at the decoder. Two types of bitstream or HRD conformance are specified: Type I and Type II. Similarly, two types of decoder conformance are specified: output timing decoder conformance and output order decoder conformance.

In the AVC and HEVC HRD models, decoding or CPB removal is based on access units, and picture decoding is assumed to be instantaneous. In practice, if a conforming decoder strictly adheres to the decoding time signaled, for example, in a picture timing supplemental enhancement information (SEI) message to begin decoding an access unit, then the earliest possible time to output a particular decoded picture is equal to the decoding time of that particular picture plus the time required to decode that particular picture. Unlike the AVC and HEVC HRD models, the time required to decode a picture in the real world is not zero. As used throughout this disclosure, the terms "instantaneous" and "instantaneously" may refer to any duration that can be assumed to be instantaneous in one or more coding models or idealized aspects of any one or more coding models, with the understanding that this may differ from being "instantaneous" in a physical or textual sense. For example, for the purposes of this disclosure, a function or process may be considered nominally "instantaneous" if it occurs within or at a practical margin of the assumed or idealized earliest possible time for the function or process to be performed. In some examples, syntax and variable names as used herein may be understood according to their meaning within the HEVC model.

Sub-picture based CPB behavior was proposed in Kazui et al., "Enhancement on operation of coded picture buffer" (Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16WP3 and ISO/IEC JTC1/SC29/WG11, 7th Meeting: Geneva, CH 21-30, November 2011, JCTVC-G188 (available at http://phenix.int-evry.fr/jct/doc_end_user/documents/7_Geneva/wg11/JCTVC-G188-v2.zip) to achieve a coding delay of less than one picture period in an interoperable manner. The JCTVC-G188 method can be summarized as follows: a picture can be evenly divided into M treeblock groups, i.e., the first M treeblocks in the treeblock raster scan of the picture belong to a first treeblock group, the second M treeblocks in the treeblock raster scan of the picture belong to a second treeblock group, and so on. The value M may be signaled in the buffering period SEI message. This value may be used to derive the CPB removal time (i.e., decoding time) for each treeblock group. In this sense, the JCTVC-G188 CPB behavior is based on sub-pictures, where each sub-picture is a treeblock group. In some examples, a sub-picture may correspond to one or more slices, one or more waves (for wavefront partitioning of a picture), or one or more image blocks. In this approach of JCTVC-G188, it is assumed that the access unit level CPB removal time is signaled as usual (using the picture timing SEI message), and within each access unit, the CPB removal time for the treeblock group is assumed to linearly or evenly divide the interval from the CPB removal time of the previous access unit to the CPB removal time of the current access unit.

This approach of JCTVC-G188 further implies the following assumptions or bitstream requirements: (1) within each picture, each tree block group is encoded in a manner that requires the same amount of decoding time (not only in the HRD model, but also for real-world decoders), where the decoded data of the first tree block group is considered to be included in all non-VCL NAL units in the same access unit and before the first VCL (Video Coding Layer) NAL (Network Abstraction Layer) unit; (2) within each picture, the number of bits used for each tree block group is exactly the same, where the decoded data of the first tree block group is considered to be included in all non-VCL NAL units in the same access unit and before the first VCL NAL unit.

Existing methods for specifying sub-picture-based CPB behavior are associated with at least the following problems: (1) The requirement that the amount of coded data for each treeblock group in a coded picture is exactly the same is difficult to achieve with balanced coding performance (where treeblock groups with areas of more detailed texture or motion activity in the picture may use more bits). (2) When more than one treeblock group is included in a slice, there may be no easy way to split the coded bits of treeblocks belonging to different treeblock groups and send the bits separately at the encoder side and remove the bits separately from the CPB (i.e., decode the bits separately).

To address the above issues, the present invention describes a general design for supporting sub-picture-based CPB behavior with various alternatives. In some instances, features of the sub-picture-based CPB technology of the present invention may include aspects of the following techniques: (1) Each sub-picture may include several coding blocks of a coded picture that are consecutive in decoding order. A coding block may be exactly the same as a tree block, or a subset of a tree block; (2) The coding of sub-pictures and the allocation of bits to different sub-pictures in a picture may be performed as usual, without assuming or requiring that each sub-picture in a picture (i.e., a set of tree blocks) be coded with the same amount of bits. Therefore, the CPB removal time for each sub-picture may be signaled in the bitstream, rather than being derived from the signaled picture-level CPB removal time; (3) When more than one sub-picture is included in a slice, byte alignment may be applied at the end of each sub-picture, in contrast to the byte alignment used for image blocks in HEVC WD6, for example. Furthermore, the entry point of each sub-picture (except the first sub-picture in a coded picture) can be signaled, in contrast to the byte alignment of tiles used, for example, in HEVC WD6. For example, the received signaled value may indicate the byte alignment of at least one of the sub-pictures within a larger set of video data, such as a slice, tile, or frame. Each of features (1) to (3) can be applied independently or in combination with the other features.

In one example, HRD operations including sub-picture-based CPB behavior can be summarized as follows: When sub-picture-based CPB behavior is signaled to be in use (e.g., via sequence-level signaling of the syntax element sub_pic_cpb_flag equal to 1), CPB removal or decoding is based on sub-pictures, or equivalently, on decoding units, which can be access units or subsets of access units. In other words, whenever a decoding unit (whether an access unit or a subset of an access unit) is removed from the CPB for decoding, the removal time of the decoding unit from the CPB can be derived from the signaled initial CPB removal delay and the signaled CPB removal delay for the decoding unit. CPB underflow is specified as the condition that, for any value of m, the nominal CPB removal time t _r,n (m) of decoding unit m is less than the final CPB removal time t _af (m) of decoding unit m. In one example, when the syntax element low_delay_hrd_flag is equal to 0, the CPB is required to never underflow.

In one example, the DPB output and removal process may still operate at the picture level or access unit level, i.e., whenever an entire decoded picture is output or removed from the DPB. Removal of a decoded picture from the DPB may occur instantaneously at the CPB removal time of the first decoding unit of access unit n (containing the current picture).

1 is a block diagram illustrating an example video encoding and decoding system 10 that may utilize techniques for, among other things, storing one or more decoding units of video data in a picture buffer; obtaining respective buffer removal times for the one or more decoding units; removing decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units; and decoding video data corresponding to the removed decoding units.

As shown in FIG1 , system 10 includes a source device 12 that transmits encoded video to a destination device 14 via a communication channel 16. Source device 12 and destination device 14 may comprise any of a wide range of devices. In some cases, source device 12 and destination device 14 may comprise wireless communication devices, such as wireless cell phones, so-called cellular or satellite radiotelephones, or any wireless device that can communicate video information via communication channel 16, in which case communication channel 16 is wireless. However, the techniques of this disclosure are not necessarily limited to wireless applications or settings. For example, these techniques may be applied to over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet video transmissions, encoded digital video encoded onto storage media, or other scenarios. Thus, communication channel 16 may comprise any combination of wireless, wired, or storage media suitable for the transmission or storage of encoded video data.

Alternatively, the encoded data may be output from transmitter 24 to storage device 34. Similarly, the encoded data may be accessed from storage device 34 by receiver 26. Storage device 34 may include any of a variety of distributed or locally accessible data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, storage device 34 may correspond to a file server, a virtual server, a data center, a redundant network of data centers, or another intermediate storage device that can hold the encoded video generated by source device 12. Destination device 14 may access the stored video data from storage device 34 via streaming or downloading. A file server implementation of storage device 34, or a portion thereof, may be any type of server capable of storing encoded video data and transmitting that encoded video data to destination device 14. Example file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data over any standard data connection, including an Internet connection. Such a connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a remote or non-local storage device 34. The transmission of the encoded video data from storage device 34 may be a streaming transmission, a download transmission, or a combination of both.

In the example of FIG1 , source device 12 includes a video source 18, a video encoder 20, a modulator/demodulator (modem) 22, and a transmitter 24. Destination device 14 includes a receiver 26, a modem 28, a video decoder 30, and a display device 32. According to this disclosure, video encoder 20 of source device 12 may be configured to apply techniques for, among other things, storing one or more decoding units of video data in a picture buffer; obtaining respective buffer removal times for one or more decoding units; removing decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units; and decoding video data corresponding to the removed decoding units. In other examples, the source device and destination device may include other components or arrangements. For example, source device 12 may receive video data from an external video source (such as an external camera) rather than integrated video source 18. Similarly, destination device 14 may interface with an external display device rather than including integrated display device 32.

The illustrated system 10 of FIG. 1 is merely an example. Techniques for storing one or more decoding units of video data in a picture buffer; obtaining corresponding buffer removal times for one or more decoding units; removing decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units; and decoding video data corresponding to the removed decoding units may be performed by any digital video encoding and/or decoding device. Although the techniques of this disclosure are typically performed by a video encoding device, they may also be performed by a video encoder/decoder, often referred to as a "CODEC." Furthermore, the techniques of this disclosure may also be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of such coding devices, with source device 12 generating coded video data for transmission to destination device 14. In some examples, devices 12 and 14 may operate in a substantially symmetrical manner, such that each device 12 and 14 includes video encoding and decoding components. Thus, system 10 may support one-way or two-way video transmission between video devices 12 and 14, for example, for video streaming, video playback, video broadcasting, or video telephony.

Video source 18 of source device 12 may include a video capture device such as a video camera, a video archive containing previously captured video, and/or a video feed from a video content provider. As a further alternative, video source 18 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, as mentioned above, the techniques described in this disclosure are generally applicable to video coding and can be applied to wireless and/or wired applications. In each case, captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be modulated by modem 22 according to a communication standard and transmitted to destination device 14 via transmitter 24. Modem 22 may include various mixers, filters, amplifiers, or other components designed for signal modulation. Transmitter 24 may include circuitry designed for transmitting data, including amplifiers, filters, and one or more antennas.

Receiver 26 of destination device 14 receives information via channel 16, and modem 28 demodulates the information. Likewise, the video encoding process may implement one or more of the techniques described herein, particularly to store one or more decoding units of video data in a picture buffer, obtain corresponding buffer removal times for the one or more decoding units, remove the decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode the video data corresponding to the removed decoding units. The information communicated via channel 16 may include syntax information defined by video encoder 20, which may also be used by video decoder 30, including syntax elements that describe characteristics and/or processing of macroblocks, coding tree units, slices, and other coded units (e.g., groups of pictures (GOPs)). Display device 32 displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

In the example of FIG1 , communication channel 16 may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines, or any combination of wireless and wired media. Communication channel 16 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. Communication channel 16 generally represents any suitable communication medium or collection of different communication media, including any suitable combination of wired or wireless media, for transmitting video data from source device 12 to destination device 14. Communication channel 16 may include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 12 to destination device 14. In other examples, source device 12 may store encoded data on a storage medium, such as storage device 34, rather than transmitting the data. Similarly, destination device 14 may be configured to retrieve the encoded data from storage device 34 or another storage medium or device.

Video encoder 20 and video decoder 30 may operate according to a video compression standard such as the standards described herein. However, the techniques of this disclosure are not limited to any particular coding standard. Although not shown in FIG. 1 , in some aspects, video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams. Where applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP).

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), wireless communication devices including video coding devices such as encoders or decoders, discrete logic, software, hardware, firmware, or any combination thereof. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in a respective camera, computer, mobile device, subscriber device, broadcast device, set-top box, server, or other device.

A video sequence typically includes a series of video frames. A group of pictures (GOP) typically includes a series of one or more video frames. A GOP may include syntax data in a header of the GOP, in a header of one or more frames of the GOP, or elsewhere, that describes the number of frames included in the GOP. Each frame may include frame syntax data that describes the encoding mode used for the corresponding frame. Video encoder 20 typically operates on video blocks (also referred to as decoding units (CUs)) within individual video frames to encode video data. A video block may correspond to a largest decoding unit (LCU) or a partition of an LCU. A video block may have a fixed or variable size, and its size may vary according to a specified decoding standard. Each video frame may include multiple slices. Each slice may include multiple LCUs, which may be arranged into several partitions, also referred to as sub-CUs. An LCU may also be referred to as a decoding tree unit.

As an example, the ITU-T H.264 standard supports intra-prediction at various block sizes, such as 16x16, 8x8, or 4x4 for luma components, and 8x8 for chroma components, and inter-prediction at various block sizes, such as 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, and 4x4 for luma components, and corresponding scaled sizes for chroma components. In this disclosure, "NxN" and "NxN" may be used interchangeably to refer to the pixel dimensions of a block in terms of vertical and horizontal dimensions, e.g., 16x16 pixels or 16x16 pixels. Generally, a 16x16 block will have 16 pixels in the vertical direction (y=16) and 16 pixels in the horizontal direction (x=16). Similarly, an NxN block typically has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. The pixels in a block can be arranged in rows and columns. Furthermore, a block does not necessarily need to have the same number of pixels in the horizontal direction as in the vertical direction. For example, a block may include N×M pixels, where M is not necessarily equal to N.

A video block may comprise a block of pixel data in the pixel domain, or a block of transform coefficients in the transform domain, e.g., after applying a transform such as a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to residual video block data representing pixel differences between the coded video block and the predictive video block. In some cases, a video block may comprise a block of quantized transform coefficients in the transform domain.

Smaller video blocks can provide better resolution and can be used for locations of a video frame that include a high level of detail. In general, blocks and various partitions (sometimes referred to as sub-blocks) can be considered video blocks. Additionally, a slice can be considered a plurality of video blocks, such as blocks and/or sub-blocks. Each slice can be an independently decodable unit of a video frame. Alternatively, a frame itself can be a decodable unit, or other portions of a frame can be defined as decodable units. The term "coded unit" can refer to any independently decodable unit of a video frame, such as an entire frame or a slice of a frame, a group of pictures (GOP) of a coded video sequence, or another independently decodable unit defined according to an applicable coding technique.

After intra-predictive or inter-predictive coding is performed to generate predictive data and residual data, and after any transform (such as the 4x4 or 8x8 integer transforms used in H.264/AVC, or discrete cosine transforms (DCTs)) is performed to generate transform coefficients, quantization of the transform coefficients may be performed. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the coefficients. The quantization process may reduce the bit depth associated with some or all of the coefficients. For example, an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

HEVC refers to a block of video data as a coding unit (CU), which may include one or more prediction units (PUs) and/or one or more transform units (TUs). This disclosure may also use the term "block" to refer to any of a CU, PU, or TU. Syntax data within the bitstream may define a largest coding unit (LCU), which is the largest coding unit in terms of the number of pixels. In general, a CU has a purpose similar to that of a macroblock of H.264, except that a CU does not have a size distinction. Therefore, a CU may be split into several sub-CUs. In general, references to a CU in this disclosure may refer to the largest coding unit of a picture, or a sub-CU of an LCU. An LCU may be split into several sub-CUs, and each sub-CU may be further split into several sub-CUs. Syntax data for the bitstream may define the maximum number of times an LCU may be split, which is called the CU depth. Therefore, the bitstream may also define a smallest coding unit (SCU).

An LCU may be associated with a quadtree data structure. In general, a quadtree data structure includes one node per CU, where the root node corresponds to the LCU. If a CU is split into four sub-CUs, the node corresponding to the CU includes four leaf nodes, each of which corresponds to one of the sub-CUs. Each node of the quadtree data structure may provide syntax data for the corresponding CU. For example, a node in the quadtree may include a split flag that indicates whether the CU corresponding to the node is split into sub-CUs. Syntax elements for a CU may be defined recursively and may depend on whether the CU is split into sub-CUs. If a CU is not split further, it is referred to as a leaf-CU. In the present invention, the four sub-CUs of a leaf-CU will also be referred to as leaf-CUs, but there is no explicit splitting of the original leaf-CU. For example, if a CU of size 16×16 is not split further, the four 8×8 sub-CUs may also be referred to as leaf-CUs, but the 16×16 CU has not yet been split.

Furthermore, the TUs of a leaf-CU may also be associated with corresponding quadtree data structures. That is, a leaf-CU may include a quadtree that indicates how the leaf-CU is split into TUs. This disclosure refers to the quadtree that indicates how an LCU is partitioned as a CU quadtree, and the quadtree that indicates how a leaf-CU is partitioned into TUs as a TU quadtree. The root node of a TU quadtree typically corresponds to a leaf-CU, while the root node of a CU quadtree typically corresponds to an LCU. Unsplit TUs of a TU quadtree may be referred to as leaf-TUs.

A leaf-CU may include one or more prediction units (PUs). In general, a PU represents all or part of the corresponding CU and may include data for retrieving reference samples for the PU. For example, when the PU is inter-mode encoded, the PU may include data defining a motion vector for the PU. The data defining the motion vector may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution used for the motion vector (e.g., quarter-pel precision or eighth-pel precision), the reference frame to which the motion vector points, and/or the reference list used for the motion vector (e.g., list 0 or list 1). The data for the leaf-CU defining the PU may also describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may be different depending on whether the CU is uncoded, intra-prediction mode encoded, or inter-prediction mode encoded. For intra-coding, a PU may be considered the same as a leaf transform unit, described below.

A leaf-CU may include one or more transform units (TUs). The transform units may be specified using a TU quadtree structure, as discussed above. That is, a split flag may indicate whether the leaf-CU is split into four transform units. Each transform unit may then be further split into four sub-TUs. When a TU is not further split, it may be referred to as a leaf-TU. In general, a split flag may indicate the splitting of a leaf-TU into square TUs. To indicate the splitting of a TU into non-square TUs, other syntax data may be included, such as syntax data indicating that the TU is to be partitioned according to a non-square quadtree transform (NSQT).

Typically, for intra coding, all leaf-TUs belonging to a leaf-CU share the same intra prediction mode. That is, the same intra prediction mode is typically applied to calculate prediction values for all TUs of a leaf-CU. For intra coding, the video encoder may use the intra prediction mode to calculate a residual value for each leaf-TU as the difference between the portion of the predictive value corresponding to the TU and the original block. The residual value may be transformed, quantized, and scanned. For inter coding, the video encoder may perform prediction at the PU level and may calculate a residual for each PU. The residual value corresponding to a leaf-CU may be transformed, quantized, and scanned. For inter coding, a leaf-TU may be larger or smaller than a PU. For intra coding, a PU may be collocated with the corresponding leaf-TU. In some examples, the maximum size of a leaf-TU may be the size of the corresponding leaf-CU.

In general, this disclosure uses the terms CU and TU to refer to leaf-CU and leaf-TU, respectively, unless otherwise noted. In general, the techniques of this disclosure are about transforming, quantizing, scanning, and entropy encoding data for a CU. As an example, the techniques of this disclosure include selecting a transform to be used to transform the residual values of an intra-predicted block based on the intra-prediction mode used to predict the block. This disclosure also uses the terms "directional transform" or "designed transform" to refer to such transforms that depend on the direction of the intra-prediction mode. That is, a video encoder may select a directional transform to apply to a transform unit (TU). As mentioned above, intra-prediction includes predicting a TU of a current CU of a picture from previously coded CUs and TUs of the same picture. More specifically, a video encoder may use a specific intra-prediction mode to intra-predict a current TU of a picture.

After quantization, entropy coding of the quantized data may be performed, for example, according to content adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), probability interval partitioning entropy coding (PIPE), or another entropy coding method. A processing unit configured for entropy coding or another processing unit may perform other processing functions, such as zero run-length coding of quantized coefficients, and/or generation of syntax information, such as coded block pattern (CBP) values, macroblock types, coding modes, maximum macroblock sizes for coded units (such as frames, slices, macroblocks, or sequences), or other syntax information.

Video encoder 20 may be configured to perform inverse quantization and inverse transform to store a decoded block to be used as a reference for predicting a subsequent block in, for example, the same or identical frame to be temporally predicted. Video encoder 20 may further send syntax data, such as block-based syntax data, frame-based syntax data, and GOP-based syntax data, to video decoder 30, for example, in a frame header, block header, slice header, or GOP header. The GOP syntax data may describe the number of frames in the respective GOP, and the frame syntax data may indicate the encoding/prediction mode used to encode the corresponding frame.

Where applicable, video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined video encoder/decoder (CODEC). An apparatus including video encoder 20 and/or video decoder 30 may include an integrated circuit, a microprocessor, and/or a wireless communication device, such as a cellular telephone.

According to the techniques of this disclosure, video encoder 20 and/or video decoder 30 may be configured to, among other things, store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for the one or more decoding units, remove the decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding units.

The following definitions are provided with respect to example video encoder 20 and/or video decoder 30 configured to, among other things, store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for the one or more decoding units, remove the decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding units.

For the purpose of describing one set of instances, the term "decoding unit" may be defined as follows:

Decoding unit: An access unit or a subset of an access unit. If sub_pic_cpb_flag is equal to 0, the decoding unit is an access unit; otherwise, the decoding unit is a subset of an access unit. When sub_pic_cpb_flag is equal to 1, the first decoding unit in an access unit includes all non-VCL NAL units in the access unit and the first VCL NAL unit in the same access unit, and every other decoding unit in the access unit is a coded slice NAL unit that is not the first coded slice NAL unit in the access unit.

For the purposes of describing the second set of examples, the term "decoding unit" may be defined as follows, where the additional definition of the term "sub-picture" is used in the corresponding example definition of "decoding unit":

Decoding unit: An access unit or a subset of an access unit. If sub_pic_cpb_flag is equal to 0, the decoding unit is an access unit; otherwise, the decoding unit is a subset of an access unit. When sub_pic_cpb_flag is equal to 1, the first decoding unit in an access unit includes all non-VCL NAL units in the access unit and the first sub-picture of a picture in the same access unit, and every other decoding unit in the access unit is a sub-picture that is not the first sub-picture in the access unit.

Sub-picture: A number of coded blocks of a coded picture that are consecutive in decoding order.

In the definition according to the second example set provided above, when more than one sub-picture is included in a slice, byte alignment may be applied at the end of each sub-picture, in contrast to the byte alignment used for image blocks in, for example, HEVC WD6. Furthermore, the entry point of each sub-picture (except the first sub-picture in a coded picture) may be signaled.

In some alternatives, when the bitstream contains multiple scalable layers or views, a decoding unit can be defined as a layer representation or view component. All non-VCL units before the first VCL NAL unit of a layer representation or view component also belong to the decoding unit containing the layer representation or view component.

The following description of example hypothetical reference decoder (HRD) operations, example operations of a coded picture buffer, example timing of bitstream arrival, example timing of decoding unit removal, example decoding of a decoding unit, example operations of a decoded picture buffer, example removal of a picture from the decoded picture buffer, example picture output, and example current decoded picture marking and storage is provided to illustrate examples of video encoder 20 and/or video decoder 30. Video encoder 20 and/or video decoder 30 may be configured to, among other functions, store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for one or more decoding units, remove decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding units. In other examples, the operations may be defined or performed differently. In this manner, video encoder 20 and/or video decoder 30 may be configured to operate according to various examples of HRD operations described below.

As an example, HRD operation can be generally described as follows: the CPB size (number of bits) is CpbSize[SchedSelIdx]. The DPB size (number of picture storage buffers) for temporal layer X is max_dec_pic_buffering[X]+1 for each X in the range of 0 to max_temporal_layers_minus1 (inclusive). In this example, the HRD can operate as follows: data associated with access units that flow into the CPB according to a specified arrival schedule can be delivered by a hypothetical stream scheduler (HSS) (i.e., a delivery scheduler). Data associated with each decoding unit can be instantaneously removed and decoded at the CPB removal time by a transient decoding process. Each decoded picture can be placed in the DPB. The decoded picture can be removed from the DPB at the later of the DPB output time or the time when the decoded picture is no longer required for inter-frame prediction reference.

The arithmetic in these examples can be done with real values so that rounding errors are not propagated.For example, the number of bits in the CPB just before or after removal of a decoding unit is not necessarily an integer.

The variable _tc can be derived as follows and can be referred to as the clock tick:

t _c =num_units_in_tick÷time_scale (C-1)

The following may be specified for expressing constraints in example annex modifications to HEVC:

Let access unit n be the nth access unit in decoding order, where the first access unit is access unit 0;

Let picture n be a coded picture or a decoded picture of access unit n;

Let decoding unit m be the mth decoding unit in decoding order, where the first decoding unit is decoding unit 0.

Some example techniques for operating a coded picture buffer (CPB) are described below. According to some video coding techniques, various methods of CPB operation may be implemented. The specifications in the HEVC WD6 section on CPB operation may be modified by this disclosure and may be applied independently to each CPB parameter set present and to both Type I and Type II conformance points.

Some examples involving the timing of bitstream arrival are described below. The HRD may be initialized by any of the Buffering Period Supplemental Enhancement Information (SEI) messages. Prior to initialization, the CPB may be empty. After initialization, the HRD may not be initialized again by subsequent Buffering Period SEI messages.

The access unit associated with the buffering period SEI message that initializes the CPB may be referred to as access unit 0. Each decoding unit may be referred to as decoding unit m, where the number m identifies the specific decoding unit. The first decoding unit in decoding order in access unit 0 may be referred to as decoding unit 0. The value of m may be incremented by 1 for each subsequent decoding unit in decoding order.

The time when the first bit of decoding unit m begins to enter the CPB can be called the initial arrival time t _ai (m). The initial arrival time of a decoding unit can be derived as follows:

If the decoding unit is decoding unit 0, then t _ai (0) = 0,

Otherwise (the decoding unit is decoding unit m, where m>0), the following applies:

If cbr_flag[SchedSelIdx] is equal to 1, then the initial arrival time for decoding unit m is equal to the final arrival time of decoding unit m-1 (which is derived below), i.e.,

t _ai (m) = t _af (m-1) (C-2)

Otherwise (cbr_flag[SchedSelIdx] is equal to 0), the initial arrival time for decoding unit m is derived from the following formula:

t _ai (m)=Max (t _af (m-1), t _{ai, earliest} (m)) (C-3)

where t _ai,earliest (m) is derived as follows.

If decoding unit m is not the first decoding unit of the subsequent buffering period, then tai _,earliest (m) can be derived as:

t _ai,earliest (m)＝t _r,n (m)-(initial_cpb_removal_delay[SchedSelIdx]+initial_cpb_removal_delay_offset[SchedSelIdx])÷90000 (C-4)

where t _r,n (m) is the nominal removal time of decoding unit m from the CPB as specified, and initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx] are specified in the previous buffering period SEI message;

Otherwise (decoding unit m is the first decoding unit of the subsequent buffering period), tai _,earliest (m) can be derived as:

t _ai,earliest (m)＝t _r,n (m)-(initial_cpb_removal_delay[SchedSelIdx]÷90000)(C-5)

where initial_cpb_removal_delay[SchedSelIdx] is specified in the buffering period SEI message associated with the access unit containing decoding unit m.

The final arrival time for decoding unit m can be derived as follows:

t _af (m)＝t _ai (m)+b(m)÷BitRate[SchedSelIdx] (C-6)

where b(m) may be the size of decoding unit m in bits, counting the bits of VCL NAL units and filler data NAL units for Type I conformance points or counting all bits of the Type II bitstream for Type II conformance points.

In some examples, the values of SchedSelIdx, BitRate[SchedSelIdx], and CpbSize[SchedSelIdx] may be constrained as follows:

If the content of the active sequence parameter set for the access unit containing decoding unit m is different from the content of the active sequence parameter set for the previous access unit, then the HSS selects a value of SchedSelIdx SchedSelIdx1 from among the values of SchedSelIdx provided in the active sequence parameter set for the access unit containing decoding unit m, said value SchedSelIdx1 resulting in BitRate[SchedSelIdx1] or CpbSize[SchedSelIdx1] for the access unit containing decoding unit m. The value of BitRate[SchedSelIdx1] or CpbSize[SchedSelIdx1] may be different from the value of BitRate[SchedSelIdx0] or CpbSize[SchedSelIdx0] for the value of SchedSelIdx SchedSelIdx0 in use for the previous access unit;

Otherwise, the HSS continues to operate with the previous values of SchedSelIdx, BitRate[SchedSelIdx], and CpbSize[SchedSelIdx].

When the HSS selects a value for BitRate[SchedSelIdx] or CpbSize[SchedSelIdx] that is different from the value of the previous access unit, the following may apply in some instances:

The variable BitRate[SchedSelIdx] takes effect at time t _ai (m)

The variable CpbSize[SchedSelIdx] takes effect as follows:

If the new value of CpbSize[SchedSelIdx] exceeds the old CPB size, it takes effect at time t _ai (m).

Otherwise, the new value of CpbSize[SchedSelIdx] takes effect at the CPB removal time of the last decoding unit of the access unit containing decoding unit m.

When sub_pic_cpb_flag is equal to 1, the initial CPB arrival time _tai (n) of access unit n may be set to the initial CPB arrival time of the first decoding unit in access unit n, and the final CPB arrival time _taf (n) of access unit n may be set to the final CPB arrival time of the last decoding unit in access unit n.

Some examples of timing and decoding of decoding units related to decoding unit removal are described below. When decoding unit m is a decoding unit with m equal to 0 (the first decoding unit of an access unit for which the HRD is initialized), the nominal removal time of the decoding unit from the CPB can be specified by the following equation:

t _{r, n} (0)＝initial_cpb_removal_delay[SchedSelIdx]÷90000 (C-7)

When decoding unit m is the first decoding unit of the first access unit that does not initialize the HRD buffering period, the nominal removal time of the decoding unit from the CPB can be specified by the following formula:

t _{r, n} (m) = t _{r, n} (m _b ) + t _c *cpb_removal_delay (m) (C-8)

where t _r,n (m _b ) is the nominal removal time of the first decoding unit of the last buffering period, and cpb_removal_delay(m) is the value of cpb_removal_delay[i] for decoding unit m as specified in the picture timing SEI message associated with the access unit containing decoding unit m.

When decoding unit n is the first decoding unit of the buffering period, m _b can be set so that the removal time t _r,n (m) of decoding unit n is equal to m. The nominal removal time t _r,n (m) of decoding unit m that is not the first decoding unit of the buffering period can be given by:

t _{r, n} (m) = t _{r, n} (m _b ) + t _c *cpb_removal_delay (m) (C-9)

where t _r,n (m _b ) is the nominal removal time of the first decoding unit of the current buffering period, and cpb_removal_delay(m) is the value of cpb_removal_delay[i] for decoding unit m as specified in the picture timing SEI message associated with the access unit containing decoding unit m.

The removal time of decoding unit m can be specified as follows:

If low_delay_hrd_flag is equal to 0 or t _r,n (m) >= t _af (m), then the removal time of decoding unit n can be specified by:

t _r (m) = t _{r, n} (m) (C-10)

Otherwise (low_delay_hrd_flag is equal to 1 and t _r,n (m) < t _af (m)), the removal time of decoding unit m is specified by:

t _r (m)＝t _r，n (m)+t _c *Ceil((t _af (m)-t _r，n (m))÷t _c ) (C-11)

The latter condition indicates that the size b(m) of decoding unit m is so large that it prevents removal at the nominal removal time.

When sub_pic_cpb_flag is equal to 1, the nominal CPB removal time t _r,n (n) of access unit n can be set to the normal CPB removal time of the last decoding unit in access unit n, and the CPB removal time t _r (n) of access unit n can be set to the CPB removal time of the last decoding unit in access unit n.

In some examples, at the CPB removal time of decoding unit m, the decoding unit may be decoded instantaneously.

Some examples of the operation of a decoded picture buffer (DPB) are described below. The decoded picture buffer may contain picture storage buffers. Each of the picture storage buffers may contain a decoded picture that is marked as "used for reference" or retained for future output. Prior to initialization, the DPB may be empty (DPB fullness set to zero). The following steps of these examples of the techniques of this disclosure may occur in the listed sequence.

Some examples of picture removal from a decoded picture buffer (DPB) are described below. In some examples, picture removal from the DPB may occur instantaneously at the CPB removal time of the first decoding unit of access unit n (containing the current picture) before decoding of the current picture (but after parsing the slice header of the first slice of the current picture), and may proceed as follows.

The decoding process for reference picture sets as specified in subclause 8.3.2 of HEVC WD6 may be invoked. If the current picture is an instantaneous decoder refresh (IDR) picture, the following may apply:

1. When an IDR picture is not the first IDR picture decoded (e.g., when the no-previous-picture-output flag has a value not equal to 1) and the value of pic_width_in_luma_samples (e.g., picture width in luma samples) or pic_height_in_luma_samples, or max_dec_pic_buffering derived from the active sequence parameter set is different from the value of pic_width_in_luma_samples or pic_height_in_luma_samples, or max_dec_pic_buffering, respectively, derived from the sequence parameter set that is active for the previous picture, no_output_of_prior_pics_flag may be inferred to be equal to 1 or set to 1 by the HRD, regardless of the actual value of no_output_of_prior_pics_flag. Decoder implementations may handle picture or DPB size changes more gracefully than HRDs regarding changes in pic_width_in_luma_samples or pic_height_in_luma_samples.

2. When no_output_of_prior_pics_flag is equal to 1 or is set or inferred to be equal to 1, all picture storage buffers in the DPB may be left empty without output of the pictures they contain, and the DPB fullness may be set to 0.

All pictures k in the DPB for which all of the following conditions hold may be removed from the DPB: picture k is marked as "unused for reference"; picture k has PicOutputFlag equal to 0, or its DPB output time is less than or equal to the CPB removal time of the first decoding unit of the current picture n (denoted as decoding unit m); that is,

t _o，dpb (k)＜＝t _r (m)

When a picture is removed from the DPB, the DPB fullness may be decremented by 1.

Some examples of picture output are described as follows. The following scenario may occur instantaneously at the CPB removal time _tr (m) of the last decoding unit (denoted as decoding unit m) of access unit n (containing the current picture). Picture n may be considered decoded after the last decoding unit of the picture is decoded.

The variable maxPicOrderCnt (for maximum picture order count (POC)) may be set equal to the maximum of the values of PicOrderCntVal (for picture order count (POC) values) for the current picture and all pictures in the DPB that are currently marked as "used for short-term reference" or have a DPB output time greater than t _r (m). The variable minPicOrderCnt (for minimum picture order count (POC)) may be set equal to the minimum of the values of PicOrderCntVal for the current picture and all pictures in the DPB that are currently marked as "used for short-term reference" or have a DPB output time greater than t _r (m). The following may be a requirement for bitstream conformance: the value of maxPicOrderCnt-minPicOrderCnt shall be less than MaxPicOrderCntLsb/2.

When picture n has PicOutputFlag equal to 1, its DPB output time t _o,dpb (n) can be derived as follows:

t _{o, dpb} (n) = t _r (m) + t _c *dpb_output_delay (n) (C-12)

where dpb_output_delay(n) is the value of dpb_output_delay specified in the picture timing SEI message associated with access unit n. The output of the current picture may be specified as follows:

If PicOutputFlag is equal to 1 and t _o,dpb (n) = t _r (m), then the current picture is output;

Otherwise, if PicOutputFlag is equal to 0, then the current picture is not output but may be stored in the DPB as further specified below;

Otherwise (PicOutputFlag is equal to 1 and t _o,dpb (n)＞t _r (m)), the current picture is output later and will be stored in the DPB (as further specified below), and is output at time t _o,dpb (n) unless indicated not to be output at a time before t _o,dpb (n) by decoding or inference of no_output_of_prior_pics_flag equal to 1.

When output, the current or selected picture may be cropped using the cropping rectangle specified in the active sequence parameter set, thereby generating a cropped picture based on the selected picture (i.e., the current picture). When picture n is the picture being output and is not the last picture of the output bitstream, the value of Δt _o,dpb (n) is defined as:

Δt _o,dpb (n)=t _o,dpb (n _n )-t _o,dpb (n) (C-13)

where n _n indicates the picture that follows picture n in output order and has PicOutputFlag equal to 1. Additional details of the boosting process and the pruning process are provided further below.

The following describes some examples involving the marking and storage of the current decoded picture. The following scenario may occur instantaneously at the CPB removal time t _r (m) of the last decoding unit of access unit n (containing the current picture). The current decoded picture may be stored in the DPB in an empty picture storage buffer, and the DPB fullness may be incremented by 1. If the current picture is a reference picture, it may be marked as "used for reference", otherwise it may be marked as "unused for reference".

The following example syntax and semantics for signaling of the CPB behavior mode are provided with respect to an example video encoder 20 and/or video decoder 30 configured to, among other things, store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for the one or more decoding units, remove the decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding units.

Some examples of the syntax and semantics for signaling the CPB behavior mode are described below. The syntax of the video usability information (VUI) parameters can be changed by adding the sub-picture CPB flag sub_pic_cpb_flag, as shown in Table 1 below:

Table 1

1 . In this example, Table 1 includes an additional flag "sub_pic_cpb_flag" relative to conventional HEVC. This sub-picture CPB flag "sub_pic_cpb_flag" may be used to signal whether the set of video data provided to the coded picture buffer (CPB) includes sub-picture parameters for sub-picture decoding. The presence of these sub-picture parameters that may be signaled by the flag "sub_pic_cpb_flag" may include buffer removal times, which include a respective buffer removal time (i.e., CPB removal time) for each of one or more decoding units. One example of the semantics of sub_pic_cpb_flag is as follows. The syntax element sub_pic_cpb_flag equal to 0 may specify that the CPB operates at the access unit level. The syntax element sub_pic_cpb_flag equal to 1 may specify that the CPB operates at the decoding unit level, which may be at the level of the access unit or a subset of the access unit, the subset corresponding to the sub-picture. When sub_pic_cpb_flag is not present, its value may be set to be inferred to be equal to 0, which may indicate that the video data does not include default states for sub-picture parameters for sub-picture decoding.

Some examples of the syntax and semantics for signaling CPB removal time for a decoding unit are described below. The syntax of the buffering period SEI message may remain unchanged as in HEVC WD6, while the semantics of the syntax elements initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx] may change as follows. In this example, the syntax element initial_cpb_removal_delay[SchedSelIdx] may specify the delay for the SchedSelIdx-th CPB between an arrival time for a first bit of coded data associated with the first decoding unit in the access unit associated with the buffering period SEI message entering the CPB and a removal time for coded data associated with the same decoding unit leaving the CPB for the first buffering period after HRD initialization. This syntax element may have a length in bits given by initial_cpb_removal_delay_length_minus1+1. In this case, this may refer to units of a 90 kHz clock. In this example, this syntax element initial_cpb_removal_delay[SchedSelIdx] may not be equal to 0 and may not exceed 90000*(CpbSize[SchedSelIdx]÷BitRate[SchedSelIdx]), the time equivalent of the CPB size in units of a 90 kHz clock.

In this example, the syntax element initial_cpb_removal_delay_offset[SchedSelIdx] may be used in conjunction with the syntax element cpb_removal_delay for the SchedSelIdx-th CPB to specify the initial transfer time of the decoding unit to the CPB. Furthermore, the syntax element initial_cpb_removal_delay_offset[SchedSelIdx] may be in units of a 90 kHz clock. The initial_cpb_removal_delay_offset[SchedSelIdx] syntax element may be a fixed-length code whose length in bits is given by initial_cpb_removal_delay_length_minus1+1. This syntax element may not be used by the decoder and may only be required for the transfer scheduler (HSS) specified in Annex C of HEVC WD6.

In some examples, the syntax and semantics of the picture timing SEI message may be changed as shown in Table 2 below:

Table 2

pic_timing(payloadSize){ Descriptor if(CpbDpbDelaysPresentFlag){ if(sub_pic_cpb_flag) num_decoding_units_minus1 ue(v) for(i=0;i<=num_decoding_units_minus1;i++) cpb_removal_delay[i] u(v) dpb_output_delay u(v) } }

In the example of Table 2, the pic_timing SEI message includes an additional num_decoding_units_minus1 signal and a for loop over several decoding units that signals the corresponding removal delay of the decoding unit from the coded picture buffer when the sub_pic_cpb_flag of the VUI parameter (e.g., according to Table 1 above) is true. Thus, the pic_timing SEI message may include information indicating the number of clock ticks to wait until each of the multiple decoding units is removed from the coded picture buffer when the VUI parameter indicates that the CPB operates at the decoding unit level. The removal delay of the decoding unit may be the same delay in units of payload or other data units for each decoding unit. In other examples, different removal delays may apply to different decoding units. In the case of an implicit time conversion of the number of bits relative to the bit processing rate for the applicable clock, the removal delay may be expressed in terms of the number of bits.

The syntax of a picture timing SEI message may depend on the contents of the sequence parameter set that is active for the coded picture associated with the picture timing SEI message. However, unless a picture timing SEI message for an instantaneous decoding refresh (IDR) access unit is preceded by a buffering period SEI message within the same access unit, activation of the associated sequence parameter set (and, for an IDR picture that is not the first picture in the bitstream, determination that the coded picture is an IDR picture) may not occur until the decoding of the first coded slice network abstraction layer (NAL) unit of the coded picture. Because the coded slice NAL units of a coded picture follow the picture timing SEI message in NAL unit order, situations may arise where it is necessary for the decoder to store the raw byte sequence payload (RBSP) containing the picture timing SEI message until it is determined that the parameters of the sequence parameter set will be active for the coded picture and then perform parsing of the picture timing SEI message. The decoder may store one or more decoding units of video data in a picture buffer in consecutive decoding order.

In one example, the presence of picture timing SEI messages in the bitstream may be specified as follows: if CpbDpbDelaysPresentFlag is equal to 1, one picture timing SEI message may be present in each access unit of the coded video sequence. Otherwise, CpbDpbDelaysPresentFlag is equal to 0, and no picture timing SEI message may be present in any access unit of the coded video sequence.

In this example, the syntax element num_decoding_units_minus1 plus 1 may specify the number of decoding units associated with the picture timing SEI message in the access unit.When sub_pic_cpb_flag is equal to 0, the syntax element num_decoding_units_minus1 may not be present and the value may be set or inferred to be 0.

In this example, the syntax element cpb_removal_delay[i] may specify the number of clock ticks to wait after the first decoding unit in an access unit associated with the most recent buffering period SEI message in the previous access unit is removed from the CPB before the i-th decoding unit in the access unit associated with the picture timing SEI message is removed from the CPB. This value may also be used to calculate the earliest possible time that decoding unit data will arrive at the CPB for the HSS. The syntax element may be a fixed-length code whose length in bits is given by cpb_removal_delay_length_minus1+1. cpb_removal_delay[i] may be the remainder of a modulo-2 ^{(cpb_removal_delay_length_minus1+1)} counter.

The value of cpb_removal_delay_length_minus1, which determines the length (in bits) of the syntax element cpb_removal_delay[i], may be the value of cpb_removal_delay_length_minus1 coded in the sequence parameter set that is active for the coded picture associated with the picture timing SEI message. However, cpb_removal_delay[i] may specify the number of clock ticks relative to the removal time of the first decoding unit in the previous access unit containing the buffering period SEI message, which may be an access unit of a different coded video sequence.

In this example, the syntax element dpb_output_delay may be used to calculate the DPB output time of a picture.The syntax element dpb_output_delay may specify the number of clock ticks to wait after removal of the last decoding unit in an access unit from the CPB before outputting a decoded picture from the DPB.

When a picture is still marked as "used for short-term reference" or "used for long-term reference", the picture may not be removed from the DPB at the output time of the picture. Only one dpb_output_delay may be specified for a decoded picture. The length of the syntax element dpb_output_delay may be given in bits by dpb_output_delay_length_minus1+1. When max_dec_pic_buffering[max_temporal_layers_minus1] is equal to 0, dpb_output_delay may be equal to 0.

The output time derived from dpb_output_delay for any picture output from an output timing consistent decoder may precede the output time derived from dpb_output_delay for all pictures in any subsequent coded video sequence in decoding order. The picture output order established by the value of this syntax element may be the same order as the order established by the value of PicOrderCnt(). For pictures that are not output by the "step-up" process because they precede an IDR picture with no_output_of_prior_pics_flag equal to 1 or inferred to be 1 in decoding order, the output time derived from dpb_output_delay may increase as the value of PicOrderCnt() relative to all pictures in the same coded video sequence increases. In an alternative example, new SEI messages, each associated with a decoding unit, may be specified, which may be referred to as decoding unit timing SEI messages, to convey the CPB removal delay for the associated decoding unit.

Thus, by implementing any combination of example definitions, example HRD operations, example operations of a coded picture buffer, example timing of bitstream arrival, example timing of decoding unit removal, example decoding of a decoding unit, example operations of a decoded picture buffer, example removal of pictures from a decoded picture buffer, example picture output, and example current decoded picture marking and storage, along with example syntax and semantics for signaling of a CPB behavior mode, video encoder 20 and/or video decoder 30 may be configured to, among other things, store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for the one or more decoding units, remove decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding units.

As an alternative to the techniques described above, a decoding unit may be defined as follows: “An access unit or a subset of an access unit. If SubPicCpbFlag is equal to 0, the decoding unit is an access unit. Otherwise, the decoding unit includes one or more VCL NAL units and associated non-VCL NAL units in the access unit. For the first VCL NAL unit in an access unit, the associated non-VCL NAL units are all non-VCL NAL units in the access unit that precede the first VCL NAL unit and the filler data NAL unit (if any) that immediately follows the first non-VCL NAL unit. For a VCL NAL unit that is not the first VCL NAL unit in an access unit, the associated non-VCL NAL unit is the filler data NAL unit (if any) that immediately follows the non-VCL NAL unit.”

In this example, the hypothetical reference decoder (HRD) operation can be summarized as follows. The CPB size (number of bits) is CpbSize[SchedSelIdx]. The DPB size (number of picture storage buffers) for temporal layer X may be max_dec_pic_buffering[X]+1 for each X in the range of 0 to max_temporal_layers_minus1 (inclusive). The variable SubPicCpbPreferredFlag may serve as a sub-picture coded picture buffer preference flag and may be specified externally or set to 0 if not specified externally. A separate sub-picture coded picture parameter present flag, sub_pic_cpb_params_present_flag, may be used to signal whether the parameters required to code a subset of one or more access units are available. A single sub-picture coded picture buffer flag, SubPicCpbFlag, may indicate whether both the sub-picture coded picture buffer preference flag and the sub-picture coded picture parameter present flag are positive or set to 1. The video coder may use this sub-picture coded picture buffer flag, SubPicCpbFlag, to determine whether to code an access unit of video data or a subset of one or more access units of video data, such as a sub-picture, as the video data is removed from the CPB.

The variable SubPicCpbFlag can be derived as follows:

SubPicCpbFlag=SubPicCpbPreferredFlag&&sub_pic_cpb_params_present_flag

(C-1)

If SubPicCpbFlag is equal to 0, the CPB may operate at the access unit level, and each decoding unit may be an access unit. Otherwise, the CPB may operate at the sub-picture level, and each decoding unit may be a subset of an access unit.

Video decoder 30 / 108 may determine that one or more decoding units include an access unit by determining that a sub-picture coded picture buffer preferred flag (eg, SubPicCpbPreferredFlag) has a value of 0 or determining that a sub-picture coded picture buffer parameter present flag (eg, sub_pic_cpb_params_present_flag) has a value of 0.

The HRD (e.g., video encoder 20 and/or video decoder 30) may operate as follows. Data associated with decoding units that flow into the CPB according to a specified arrival schedule may be delivered by the HSS. In one example, data associated with each decoding unit may be instantaneously removed and decoded at the CPB removal time by a transient decoding process. Each decoded picture may be placed in the DPB. At a later time than the DPB output time or when the decoded picture is no longer required for inter-frame prediction reference, the decoded picture may be removed from the DPB.

The arithmetic operations described in this disclosure may be performed with real values so that rounding errors are not propagated.For example, the number of bits in the CPB just before or after removal of a decoding unit may not necessarily be an integer.

The variable _tc can be derived as follows and is called the clock tick:

t _c =num_units_in_tick÷time_scale (C-1)

The following may be specified for expressing the constraints in this example of the present technology:

Let access unit n be the nth access unit in decoding order, where the first access unit is access unit 0;

Let picture n be a coded picture or a decoded picture of access unit n;

Let decoding unit m be the mth decoding unit in decoding order, where the first decoding unit is decoding unit 0.

The operation of a coded picture buffer (CPB) may be defined as follows: The specification in this example may be applied independently to each CPB parameter set present, and to both Type I and Type II conformance points.

Regarding the timing of bitstream arrival, the HRD can be initialized by any of the buffering period SEI messages. Before initialization, the CPB can be empty. After initialization, the HRD may not be initialized again by subsequent buffering period SEI messages.

Each access unit may be referred to as a corresponding access unit n, where the number n identifies a particular access unit. The access unit associated with the buffering period SEI message that initializes the CPB may be referred to as access unit 0. The value of n may be incremented by one for each subsequent access unit in decoding order.

Each decoding unit may be referred to as decoding unit m, where the number m identifies the particular decoding unit. The first decoding unit in decoding order in access unit 0 may be referred to as decoding unit 0. The value of m may be incremented by 1 for each subsequent decoding unit in decoding order.

In this example, if the variable SubPicCpbFlag is equal to 0, then the variable InitCpbRemovalDelay[SchedSelIdx] may be set to initial_cpb_removal_delay[SchedSelIdx] of the associated buffering period SEI message, and InitCpbRemovalDelayOffset[SchedSelIdx] may be set to initial_cpb_removal_delay_offset[SchedSelIdx] of the associated buffering period SEI message. Otherwise, the variable InitCpbRemovalDelay[SchedSelIdx] may be set to initial_du_cpb_removal_delay[SchedSelIdx] of the associated buffering period SEI message, and InitCpbRemovalDelayOffset[SchedSelIdx] may be set to initial_du_cpb_removal_delay_offset[SchedSelIdx] of the associated buffering period SEI message.

The time when the first bit of decoding unit n begins to enter the CPB can be called the initial arrival time t _ai (m). The initial arrival time of a decoding unit can be derived as follows:

If the decoding unit is decoding unit 0, then t _ai (0) = 0;

Otherwise (the decoding unit is decoding unit m, where m>0), the following applies:

If cbr_flag[SchedSelIdx] is equal to 1, then the initial arrival time for decoding unit m may be equal to the final arrival time of access unit m-1 (which is derived below), i.e.,

t _ai (m) = t _af (m-1) (C-2)

Otherwise (cbr_flag[SchedSelIdx] is equal to 0), the initial arrival time for decoding unit m can be derived as follows:

t _ai (m)=Max (t _af (m-1), t _{ai, earliest} (m)) (C-3)

where t _ai,earliest (m) can be derived as follows:

If decoding unit n is not the first decoding unit of the subsequent buffering period, then tai _,earliest (m) can be derived as:

t _{ai，earliest} (m)＝t _r，n (m)-(InitCpbRemovalDelay[SchedSelIdx]+InitCpbRemovalDelayOffset[SchedSelIdx])÷90000 (C-4)

where t _r,n (m) is the nominal removal time of decoding unit m from the CPB;

Otherwise (decoding unit m is the first decoding unit of the subsequent buffering period), tai _,earliest (m) can be derived as:

t _ai,earliest (m)＝t _r,n (m)-(InitCpbRemovalDelay[SchedSelIdx]÷90000) (C-5)

The final arrival time t _af for decoding unit m can be derived as follows:

t _af (m)＝t _ai (m)+b(m)÷BitRate[SchedSelIdx] (C-6)

where b(m) is the size of decoding unit m in bits, counting the bits of VCL NAL units and filler data NAL units for Type I conformance points or counting all bits of the Type II bitstream for Type II conformance points.

In some examples, the values of SchedSelIdx, BitRate[SchedSelIdx], and CpbSize[SchedSelIdx] may be constrained as follows:

If the contents of the active sequence parameter set for the access unit containing decoding unit m are different from the contents of the active sequence parameter set for the previous access unit, then the HSS may select a value of SchedSelIdx, SchedSelIdx1, from among the values of SchedSelIdx provided in the active sequence parameter set for the access unit containing decoding unit m, which value SchedSelIdx1 results in BitRate[SchedSelIdx1] or CpbSize[SchedSelIdx1] for the access unit containing decoding unit m. The value of BitRate[SchedSelIdx1] or CpbSize[SchedSelIdx1] may be different from the value of BitRate[SchedSelIdx0] or CpbSize[SchedSelIdx0] for the value of SchedSelIdx, SchedSelIdx0, in use for the previous access unit.

Otherwise, the HSS may continue to operate with the previous values of SchedSelIdx, BitRate[SchedSelIdx], and CpbSize[SchedSelIdx].

When the HSS selects a value for BitRate[SchedSelIdx] or CpbSize[SchedSelIdx] that is different from the value of the previous access unit, the following applies:

The variable BitRate[SchedSelIdx] may take effect at time t _ai (m);

The variable CpbSize[SchedSelIdx] can be put into effect as follows:

If the new value of CpbSize[SchedSelIdx] exceeds the old CPB size, it may take effect starting at time t _ai (m);

Otherwise, the new value of CpbSize[SchedSelIdx] may take effect at the CPB removal time of the last decoding unit of the access unit containing decoding unit m.

When the variable SubPicCpbFlag is equal to 1, the initial CPB arrival time _tai (n) of access unit n may be set to the initial CPB arrival time of the first decoding unit in access unit n, and the final CPB arrival time _taf (n) of access unit n may be set to the final CPB arrival time of the last decoding unit in access unit n. When SubPicCpbFlag is equal to 0, each decoding unit may be an access unit, so that the initial and final CPB arrival times of access unit n may be the initial and final CPB arrival times of decoding unit m.

The following discussion provides an example of the timing of decoding unit removal and decoding for a decoding unit. If SubPicCpbFlag is equal to 0, then the variable CpbRemovalDelay(m) may be set to the value of cpb_removal_delay specified in the picture timing SEI message associated with the access unit for decoding unit m. Otherwise, the variable CpbRemovalDelay(m) may be set to the value of du_cpb_removal_delay[i] for decoding unit m specified in the picture timing SEI message associated with the access unit containing decoding unit m.

When decoding unit m is a decoding unit with m equal to 0 (the first decoding unit of an access unit for which the HRD is initialized), the nominal removal time of the decoding unit from the CPB can be specified by the following formula:

t _{r, n} (0)＝InitCpbRemovalDelay[SchedSelIdx]÷90000 (C-7)

When decoding unit m is the first decoding unit of the first access unit that does not initialize the HRD buffering period, the nominal removal time of the decoding unit from the CPB can be specified by the following formula:

t _{r, n} (m) = t _{r, n} (m _b ) + t _c *CpbRemovalDelay (m) (C-8)

where t _r,n ( _mb ) is the nominal removal time of the first decoding unit in the previous buffering period.

When decoding unit m is the first decoding unit of the buffering period, _mb may be set such that the removal time tr _,n (m) of decoding unit m is equal to m.

The nominal removal time tr _,n (m) of a decoding unit m that is not the first decoding unit of a buffering period can be given by:

t _{r, n} (m) = t _{r, n} (m _b ) + t _c *CpbRemovalDelay (m) (C-9)

where t _r,n ( _mb ) is the nominal removal time of the first decoding unit in the current buffering period.

The removal time of decoding unit m can be specified as follows:

If low_delay_hrd_flag is equal to 0 or t _r,n (m) >= t _af (m), then the removal time of decoding unit m can be specified by:

t _r (m) = t _{r, n} (m) (C-10)

Otherwise (low_delay_hrd_flag is equal to 1 and t _r,n (m) < t _af (m)), and the removal time of decoding unit m can be specified by the following formula:

t _r (m)＝t _r，n (m)+t _c *Ceil((t _af (m)-t _r，n (m))÷t _c ) (C-11)

In this example, the latter condition indicates that the size b(m) of decoding unit m is so large that it prevents removal at the nominal removal time.

When SubPicCpbFlag is equal to 1, the nominal CPB removal time t _r,n (n) of access unit n can be set to the normal CPB removal time of the last decoding unit in access unit n; the CPB removal time t _r (n) of access unit n can be set to the CPB removal time of the last decoding unit in access unit n. When SubPicCpbFlag is equal to 0, in this example, each decoding unit m is access unit n, and therefore, the nominal CPB removal time and CPB removal time of access unit n are the nominal CPB removal time and CPB removal time of decoding unit m. In some examples, at the CPB removal time of decoding unit m, the decoding unit can be decoded instantaneously.

In this example, the decoded picture buffer (DPB) may operate as follows. The decoded picture buffer may contain one or more picture storage buffers. Each of the picture storage buffers may contain a decoded picture that is marked as "used for reference" or retained for future output. Prior to initialization, the DPB may be empty (the DPB fullness variable may be set to zero). The following steps of this example may occur in the sequence listed.

First, the picture can be removed from the DPB. Removal of the picture from the DPB can occur instantaneously at the CPB removal time of the first decoding unit of access unit n (containing the current picture), before decoding of the current picture (but after parsing the slice header of the first slice of the current picture), and can proceed as follows. The decoding process for reference picture sets as specified in subclause 8.3.2 of HEVC WD6 can be invoked. If the current picture is an IDR picture, the following may apply: when the IDR picture is not the first IDR picture decoded and the value of pic_width_in_luma_samples or pic_height_in_luma_samples or max_dec_pic_buffering derived from the active sequence parameter set is different from the value of pic_width_in_luma_samples or pic_height_in_luma_samples or max_dec_pic_buffering, respectively, derived from the sequence parameter set that is active for the previous picture, no_output_of_prior_pics_flag (i.e., no previous picture output flag) may be inferred by the HRD to be equal to 1, or set to 1 by the HRD for its own processing, regardless of the actual value of no_output_of_prior_pics_flag. The decoder implementation may attempt to handle picture or DPB size changes more gracefully than the HRD regarding changes in pic_width_in_luma_samples or pic_height_in_luma_samples.

When no_output_of_prior_pics_flag is equal to 1 or is inferred to be equal to 1, all picture storage buffers in the DPB may be emptied of the output of the pictures they contain, and the DPB fullness may be set to 0. (Further processing that may be performed when no_output_of_prior_pics_flag has a value not equal to 1 is further described below.) All pictures k in the DPB for which the following conditions hold may be removed from the DPB:

Image k is marked as “not used for reference”;

Picture k has PicOutputFlag equal to 0, or its DPB output time is less than or equal to the CPB removal time of the first decoding unit (denoted as decoding unit m) of current picture n; ie, to _,dpb (k)<= _tr (m).

When a picture is removed from the DPB, the DPB fullness may be decremented by 1. For picture output, the following may occur instantaneously at the CPB removal time _tr (n) of access unit n:

Picture n may be considered decoded after the last decoding unit of the picture is decoded.

When picture n has PicOutputFlag equal to 1, its DPB output time t _o,dpb (n) can be derived as follows:

t _{o, dpb} (n) = t _r (n) + t _c *dpb_output_delay (n) (C-12)

where dpb_output_delay(n) is the value of dpb_output_delay specified in the picture timing SEI message associated with access unit n.

The output of the current image can be specified as follows:

If PicOutputFlag is equal to 1 and t _o,dpb (n)=t _r (n), then the current picture can be output;

Otherwise, if PicOutputFlag is equal to 0, the current picture may not be output but may be stored in the DPB;

Otherwise (PicOutputFlag is equal to 1 and t _o,dpb (n)>t _r (n)), the current picture may be output later and may be stored in the DPB (as specified in subclause C.3.3 of HEVC WD6 as modified by this disclosure), and may be output at time t _o,dpb (n), unless indicated not to be output at a time before t _o,dpb (n) by decoding or inference of no_output_of_prior_pics_flag being equal to 1. In other words, if the no prior picture output flag is not equal to 1, the current picture may be stored in the DPB and may be output later (e.g., at time t _o,dpb (n)).

When output, the picture may be cropped using the crop rectangle specified in the active sequence parameter set.

When picture n is a picture that is output and is not the last picture of the output bitstream, the value of the DPB output time interval Δt _o,dpb (n) may be defined as follows:

Δt _o,dpb (n)=t _o,dpb (n _n )-t _o,dpb (n) (C-13)

Where n _n may indicate a picture that follows picture n in output order and has PicOutputFlag equal to 1, so that the DPB output time interval Δt _o,dpb (n) may be defined as the difference between the DPB output time of the subsequent picture following picture n in output order and the DPB output time of picture n.

For the current decoded picture marking and storage, the following scenarios may be instantaneously implemented at the CPB removal time t _r (n) of access unit n: the current decoded picture may be stored in the DPB in an empty picture storage buffer, and the DPB fullness may be incremented by 1; if the current picture is a reference picture, it may be marked as "used for reference", otherwise, it may be marked as "unused for reference".

For output-order operation of the DPB, the decoded picture buffer may contain one or more picture storage buffers. Each of these picture storage buffers may contain a decoded picture that is marked as "used for reference" or retained for future output. When the HRD is initialized, the DPB may be empty. The following steps may occur in the order listed.

A picture may be removed from the DPB as follows: Removal of a picture from the DPB before decoding of the current picture (but after parsing the slice header of the first slice of the current picture) may be performed instantaneously when the first decoding unit of the access unit containing the current picture is removed from the CPB and may proceed as follows.

The decoding process for reference picture sets as specified in subclause 8.3.4.3 of HEVC WD6 as modified in accordance with this disclosure may be invoked (as partially described above and further described below).

If the current picture is an IDR picture, the following applies:

When an IDR picture is not the first IDR picture decoded and the value of pic_width_in_luma_samples or pic_height_in_luma_samples or max_dec_pic_buffering derived from the active sequence parameter set is different from the value of pic_width_in_luma_samples or pic_height_in_luma_samples or max_dec_pic_buffering, respectively, derived from the sequence parameter set that was active for the previous picture, no_output_of_prior_pics_flag may be set or inferred to be equal to 1 by the HRD, regardless of the actual value of no_output_of_prior_pics_flag. Decoder implementations may attempt to handle changes in the value of pic_width_in_luma_samples or pic_height_in_luma_samples or max_dec_pic_buffering more gracefully than the HRD;

When no_output_of_prior_pics_flag is equal to 1 or is inferred to be equal to 1, all picture storage buffers in the DPB may be left empty without output of the pictures they contain;

Otherwise, the picture storage buffer containing pictures marked as "not needed for output" and "unused for reference" may be left empty (no output).

The “boost” process as specified in subclause C.5.2.1 of HEVC WD6 as modified by this disclosure may be repeatedly called until there is an empty picture storage buffer to store the current decoded picture when any of the following conditions holds:

The number of pictures marked as "needed for output" in the DPB is greater than the number of reorder pictures at the current temporal layer, i.e., num_reorder_pics[temporal_id]; or,

The number of pictures in the DPB whose temporal layer identifier value temporal_id is less than or equal to the temporal layer identifier value temporal_id of the current picture is equal to the maximum picture buffering value of the current temporal layer plus 1, that is, max_dec_pic_buffering[temporal_id]+1; or

When the current picture is an IDR picture, the no previous picture output flag no_output_of_prior_pics_flag has a value not equal to 1 and is not inferred to be equal to 1 for the picture.

The following steps may be performed: the picture storage buffers containing pictures marked as "not needed for output" and "not used for reference" may be made empty (no output); and all non-empty picture storage buffers in the DPB may be made empty by repeatedly calling the "boost" process specified below.

Therefore, the "boost" process may be invoked under any of the following conditions:

the current picture is an IDR picture and no_output_of_prior_pics_flag is not equal to 1 and is not set or inferred to be equal to 1, as specified in subclause C.5.2 of HEVC WD6 as modified by this disclosure; or,

the number of pictures marked as "needed for output" in the DPB is greater than the number of reorder pictures at the current temporal layer, i.e., num_reorder_pics[temporal_id], as specified in subclause C.5.2 of HEVC WD6 as modified by this invention; or,

The number of pictures in the DPB with temporal_id lower than or equal to the temporal layer identifier value temporal_id of the current picture is equal to the maximum picture buffering value of the current temporal layer plus 1, i.e., max_dec_pic_buffering[temporal_id]+1, as specified in subclause C.5.2 of HEVC WD6 as modified by this invention.

The "improvement" process may include the following sequential steps:

1. The picture to be outputted first may be selected as the picture having the minimum value of PicOrderCntVal of all pictures marked as “needed for output” in the DPB.

2. The picture is cropped using the cropping rectangle specified in the active sequence parameter set for the picture, the cropped picture may be output, and the picture may be marked as "not needed for output."

3. If the picture storage buffer including the cropped and output picture contains pictures marked as "unused for reference", the picture storage buffer may be emptied.

When the last decoding unit of access unit n containing the current picture is removed from the CPB, the following may happen instantaneously for picture decoding, marking, and storage.

The current picture may be considered decoded after the last decoding unit of the picture is decoded. The current decoded picture may be stored in an empty picture storage buffer in the DPB, and the following may apply:

If the current decoded picture has PicOutputFlag equal to 1, it may be marked as "needed for output";

Otherwise (the current decoded picture has PicOutputFlag equal to 0), it may be marked as “not needed for output”.

If the current decoded picture is a reference picture, it may be marked as “used for reference”, otherwise (the current decoded picture is a non-reference picture), it may be marked as “unused for reference”.

Therefore, the improvement process may include: selecting a picture in the DPB that has a minimum picture order count (POC) value of the pictures and is marked as required for output as a selected picture; cropping the selected picture as specified in the active sequence parameter set for the selected picture, thereby generating a cropped picture based on the selected picture; outputting the cropped picture; and marking the selected picture as not required for output.

Using the semantics defined below, syntax elements may be used to signal CPB behavior modes. The syntax and semantics of the VUI parameters may be changed as shown in Table 3 below (in this example, the semantics of the existing syntax elements are unchanged relative to HEVC WD6):

Table 3

In the example of Table 3, the VUI parameters include an additional flag, sub_pic_cpb_params_present_flag, relative to conventional HEVC. The semantics for this flag may be defined as follows: sub_pic_cpb_params_present_flag, when equal to 1, may specify that the sub-picture level CPB removal delay parameters are present and that the CPB may operate at either the access unit level or the sub-picture level. The variable sub_pic_cpb_flag, when equal to 0, may specify that the sub-picture level CPB removal delay parameters are not present and that the CPB must operate at the access unit level. When sub_pic_cpb_params_present_flag is not present, its value may be set or inferred to be equal to 0.

In the case of using the semantics described below, syntax elements may also be used to signal the CPB removal time of the decoding unit. In this example, the syntax elements may be signaled in a buffering period SEI message, for example, according to the example of Table 4:

Table 4

buffering_period(payloadSize){ Descriptor seq_parameter_set_id ue(v) if(NalHrdBpPresentFlag){ for(SchedSelIdx=0; SchedSelIdx<=cpb_cnt_minus1; SchedSelIdx++){ initial_cpb_removal_delay[SchedSelIdx] u(v) initial_cpb_removal_delay_offset[SchedSelIdx] u(v) if(sub_pic_cpb_flag){ initial_du_cpb_removal_delay[SchedSelIdx] u(v) initial_du_cpb_removal_delay_offset[SchedSelIdx] u(v) } } } if(VclHrdBpPresentFlag){ for(SchedSelIdx=0; SchedSelIdx<=cpb_cnt_minus1; SchedSelIdx++){ initial_cpb_removal_delay[SchedSelIdx] u(v) initial_cpb_removal_delay_offset[SchedSelIdx] u(v) if(sub_pic_cpb_flag){ initial_du_cpb_removal_delay[SchedSelIdx] u(v) initial_du_cpb_removal_delay_offset[SchedSelIdx] u(v) } } } }

In the example of Table 4, the buffering period SEI message includes additional conditions relative to conventional HEVC, which further include: when sub_pic_cpb_flag is true, adding two syntax elements initial_du_cpb_removal_delay[SchedSelIdx] and initial_du_cpb_removal_delay_offset[SchedSelIdx]. This condition and additional syntax elements can be added within either or both of the conditions for the following situations: when NalHrdBpPresentFlag is true and/or when VclHardBpPresentFlag is true.

Table 5 provides an alternative example, in which different SEI messages are defined to signal the initial CPB removal delay and initial CPB removal delay offset for sub-picture level CPB operation:

Table 5

du_buffering_period(payloadSize){ Descriptor seq_parameter_set_id ue(v) if(NalHrdBpPresentFlag){ for(SchedSelIdx=0; SchedSelIdx<=cpb_cnt_minus1; SchedSelIdx++){ initial_du_cpb_removal_delay[SchedSelIdx] u(v) initial_du_cpb_removal_delay_offset[SchedSelIdx] u(v) } } if(VclHrdBpPresentFlag){ for(SchedSelIdx=0; SchedSelIdx<=cpb_cnt_minus1; SchedSelIdx++){ initial_du_cpb_removal_delay[SchedSelIdx] u(v) initial_du_cpb_removal_delay_offset[SchedSelIdx] u(v) } } }

In the example of Table 4 above, when NalHrdBpPresentFlag or VclHrdBpPresentFlag is equal to 1, the buffering period SEI message can be associated with any access unit in the bitstream, and the buffering period SEI message can be associated with each IDR access unit, each CRA access unit, and each access unit associated with a recovery point SEI message. For some applications, the frequent presence of the buffering period SEI message may be desirable. In some examples, between two instances of the buffering period SEI message in decoding order, the buffering period may be specified as a set of access units.

In the examples of Tables 4 and 5 above, the variable seq_parameter_set_id may specify a sequence parameter set containing sequence HRD attributes. The value of seq_parameter_set_id may be equal to the value of seq_parameter_set_id in the picture parameter set referenced by the primary coded picture associated with the buffering period SEI message. In some examples, the value of seq_parameter_set_id may be in the range of 0 to 31 (inclusive).

In the example of Table 4 above, initial_cpb removal_delay[SchedSelIdx] may specify the delay for the SchedSelIdx-th CPB between the arrival time of the first bit of coded data associated with the access unit associated with the buffering period SEI message into the CPB, and the removal time of the coded data associated with the same access unit for the first buffering period after HRD initialization that leaves the CPB. This syntax element may have a length in bits given by initial_cpb_removal_delay_length_minus1+1. In this example, it may be in units of a 90 kHz clock. In this example, the syntax element initial_cpb_removal_delay[SchedSelIdx] may not be equal to 0, and in this example, may not exceed 90000*(CpbSize[SchedSelIdx]÷BitRate[SchedSelIdx]), the time equivalent of the CPB size in units of a 90 kHz clock.

In the example of Table 4 above, the syntax element initial_cpb_removal_delay_offset[SchedSelIdx] may be used in conjunction with cpb_removal_delay for the SchedSelIdx-th CPB to specify the initial delivery time of the coded access unit to the CPB. In this example, the syntax element initial_cpb_removal_delay_offset[SchedSelIdx] may be in units of a 90 kHz clock. The initial_cpb_removal_delay_offset[SchedSelIdx] syntax element may be a fixed-length code whose length in bits is given by initial_cpb_removal_delay_length_minus1+1. This syntax element may not be used by the decoder and may only be required for the delivery scheduler (HSS) specified in Annex C of HEVC WD6. Throughout the entire coded video sequence, the sum of initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx] may be constant for each value of SchedSelIdx.

In the examples of Tables 4 and 5 above, the syntax element initial_du_cpb_removal_delay[SchedSelIdx] may specify the delay between the arrival time of the first bit of coded data associated with the first decoding unit in the access unit associated with the buffering period SEI message into the CPB and the removal time of the coded data associated with the same decoding unit for the first buffering period after HRD initialization when it leaves the CPB for the SchedSelIdx-th CPB. This syntax element may have a length in bits given by initial_cpb_removal_delay_length_minus1+1. In this example, this syntax element may be in units of a 90 kHz clock. In this example, the syntax element initial_du_cpb_removal_delay[SchedSelIdx] may not be equal to 0, and may not exceed 90000*(CpbSize[SchedSelIdx]÷BitRate[SchedSelIdx]), the time equivalent of the CPB size in units of a 90 kHz clock.

In the examples of Tables 4 and 5 above, the syntax element initial_du_cpb_removal_delay_offset[SchedSelIdx] may be used in conjunction with cpb_removal_delay for the SchedSelIdx-th CPB to specify the initial delivery time of the decoding unit to the CPB. In this example, the syntax element initial_cpb_removal_delay_offset[SchedSelIdx] may be in units of a 90 kHz clock. The initial_du_cpb_removal_delay_offset[SchedSelIdx] syntax element may be a fixed-length code whose length in bits is given by initial_cpb_removal_delay_length_minus1+1. In this example, this syntax element may not be used by the decoder and may only be required for the delivery scheduler (HSS) specified in Annex C of HEVC WD6.

Throughout the entire coded video sequence, the sum of initial_du_cpb_removal_delay[SchedSelIdx] and initial_du_cpb_removal_delay_offset[SchedSelIdx] may be constant for each value of SchedSelIdx.

Table 6 below provides an example picture timing SEI message syntax:

Table 6

pic_timing(payloadSize){ Descriptor if(CpbDpbDelaysPresentFlag){ cpb_removal_delay u(v) dpb_output_delay u(v) if(sub_pic_cpb_flag){ num_decoding_units_minus1 ue(v) for(i=0;i<=num_decoding_units_minus1;i++){ num_nalus_in_du_minus1[i] ue(v) du_cpb_removal_delay[i] u(v) } } } }

In this example, the picture timing SEI message includes an additional conditional statement for sub_pic_cpb_flag that, when true, signals the num_decoding_units_minus1 syntax element and a for loop that signals the corresponding numb_nalus_in_du_minus1 and du_cpb_removal_delay for each of the decoding units. Alternatively, the mapping of NAL units to each decoding unit can be signaled using other means, for example, by including a decoding unit ID for each VCL NAL unit, for example, in the NAL unit header, slice header, or a new SEI message. The decoding ID for each non-VCL NAL unit can be the same as the associated VCL NAL unit.

The syntax of the picture timing SEI message in the example of Table 6 may depend on the contents of the sequence parameter set that is active for the coded picture associated with the picture timing SEI message. However, unless the picture timing SEI message for an IDR access unit is preceded by a buffering period SEI message within the same access unit, activation of the associated sequence parameter set (and, for an IDR picture that is not the first picture in the bitstream, determination that the coded picture is an IDR picture) may not occur until the decoding of the first coded slice NAL unit of the coded picture. Since the coded slice NAL units of a coded picture may follow the picture timing SEI message in NAL unit order, there may be situations where the decoder stores the RBSP containing the picture timing SEI message until it determines which sequence parameter will be active for the coded picture and then performs analysis of the picture timing SEI message.

Following the example of Table 6, the presence of a picture timing SEI message in a bitstream may be specified as follows.

If CpbDpbDelaysPresentFlag is equal to 1, one picture timing SEI message may be present in each access unit of the coded video sequence;

Otherwise (CpbDpbDelaysPresentFlag is equal to 0), no picture timing SEI message needs to be present in any access unit of the coded video sequence.

The variable cpb_removal_delay may specify the number of clock ticks to wait after the access unit associated with the most recent buffering period SEI message in the previous access unit is removed from the CPB before the access unit data associated with the picture timing SEI message is removed from the buffer (see subclause E.2.1 of HEVC WD6). This value may also be used to calculate the earliest possible time that access unit data enters the CPB for the HSS, as specified in Annex C of HEVC WD6. The syntax element may be a fixed-length code whose length in bits is given by cpb_removal_delay_length_minus1+1. cpb_removal_delay may be the remainder of a modulo-2 (cpb_removal_delay_length_minus1+1) counter. The value of cpb_removal_delay_length_minus1, which determines the length (in bits) of the syntax element cpb_removal_delay, may be the value of cpb_removal_delay_length_minus1 coded in the sequence parameter set that is active for the primary coded picture associated with the picture timing SEI message. However, cpb_removal_delay may specify the number of clock ticks relative to the removal time of the previous access unit containing the buffering period SEI message, which may be an access unit of a different coded video sequence.

The variable dpb_output_delay can be used to calculate the DPB output time of a picture. This variable can specify the number of clock ticks to wait after the last decoding unit in an access unit is removed from the CPB before outputting the decoded picture from the DPB (see subclause C.2 of HEVC WD6). In this example, when a picture is still marked as "used for short-term reference" or "used for long-term reference", the picture may not be removed from the DPB at the output time of the picture. In this example, only one dpb_output_delay variable can be specified for a decoded picture.

The length of the syntax element dpb_output_delay may be given in bits by dpb_output_delay_length_minus1 + 1. When max_dec_pic_buffering[max_temporal_layers_minus1] is equal to 0, dpb_output_delay may also be equal to 0.

The output time derived from dpb_output_delay of any picture output from an output timing consistent decoder (as specified in subclause C.2 of HEVC WD6 as modified by this disclosure) may be before the output time derived from dpb_output_delay of all pictures in any subsequent coded video sequence in decoding order.

The picture output order established by the value of this syntax element may be the same order as the order established by the value of PicOrderCnt() (as specified by subclause C.5 of HEVC WD6).

For pictures that are not output by the "boost" process of subclause C.5 of HEVC WD6 as modified by this disclosure because they precede, in decoding order, an IDR picture with no_output_of_prior_pics_flag equal to 1 or set or inferred to be equal to 1, the output time derived from dpb_output_delay may increase as the value of PicOrderCnt() relative to all pictures within the same coded video sequence increases.

The variable num_decoding_units_minus1 plus 1 may specify the number of decoding units associated with the picture timing SEI message in the access unit. For example, the value of num_decoding_units_minus1 may be in the range of 0 to X, inclusive.

The variable num_nalus_in_du_minus1[i] plus 1 may specify the number of NAL units associated with the picture timing SEI message in the i-th decoding unit of the access unit. For example, the value of num_nalus_in_du_minus1[i] may be in the range of 0 to X, inclusive.

The first decoding unit of an access unit may include the first num_nalus_in_du_minus1[0]+1 consecutive NAL units in decoding order in the access unit. The i-th (where i is greater than 0) decoding unit of an access unit may include the num_nalus_in_du_minus1[i]+1 consecutive NAL units that immediately follow the last NAL unit in the previous decoding unit of the access unit in decoding order. For example, there may be at least one VCL NAL unit in each decoding unit.

The variable du_cpb_removal_delay[i] may specify the number of clock ticks to wait after the first decoding unit in an access unit associated with the most recent buffering period SEI message in the previous access unit is removed from the CPB before the i-th decoding unit in the access unit associated with the picture timing SEI message is removed from the CPB (see subclause E.2.1 of HEVC WD6). This value may also be used to calculate the earliest possible time that decoding unit data enters the CPB for the HSS, as specified in Annex C of HEVC WD6. The syntax element may be a fixed-length code, whose length in bits may be given by cpb_removal_delay_length_minus1+1. du_cpb_removal_delay[i] may be the remainder of a modulo-2 (cpb_removal_delay_length_minus1+1) counter. The value of cpb_removal_delay_length_minus1, which determines the length (in bits) of the syntax element du_cpb_removal_delay[i], may be the value of cpb_removal_delay_length_minus1 coded in the sequence parameter set that is active for the coded picture associated with the picture timing SEI message. However, du_cpb_removal_delay[i] specifies the number of clock ticks relative to the removal time of the first decoding unit in the previous access unit containing the buffering period SEI message, which may be an access unit of a different coded video sequence.

FIG2 is a block diagram illustrating an example of a video encoder 20 that may implement techniques as described in this disclosure, particularly those related to: storing one or more decoding units of video data in a picture buffer; obtaining respective buffer removal times for the one or more decoding units; removing decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units; and decoding video data corresponding to the removed decoding units. Video encoder 20 may perform intra- and inter-coding of blocks within a video frame, the blocks comprising coding units (CUs), or sub-CUs of a CU. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames of a video sequence. Intra-mode (I-mode) may refer to any of several spatially-based compression modes, and inter-mode, such as unidirectional prediction (P-mode) or bidirectional prediction (B-mode), may refer to any of several temporally-based compression modes. 2 , it should be understood that video encoder 20 may further include components for intra-mode encoding, such as intra-prediction unit 46. Additional components that may also be included are not illustrated in FIG. 2 for brevity and clarity.

2 , video encoder 20 receives a video block that is included in a current video block within a video frame to be encoded. In the example of FIG2 , video encoder 20 includes motion compensation unit 44, motion estimation unit 42, reference picture memory 64, summer 50, transform unit 52, quantization unit 54, entropy encoding unit 56, buffer 90, and coded picture buffer 92. For video block reconstruction, video encoder 20 also includes inverse quantization unit 58, inverse transform unit 60, and summer 62.

During the encoding process, video encoder 20 receives a video frame or slice to be coded. The frame or slice may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-frame predictive coding of the received video block relative to one or more blocks in one or more reference frames to provide temporal compression. Intra-frame prediction unit 46 may also perform intra-frame predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial compression.

Mode select unit 40 may select one of the coding modes (intra or inter), e.g., based on the error result, and may provide the resulting intra- or inter-coded block to summer 50 to generate residual block data and to summer 62 to reconstruct the encoded block for use as a reference frame.

Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation is the process of generating motion vectors, which estimate the motion for video blocks. For example, a motion vector may indicate the displacement of a predictive block within a predictive reference frame (or other coded unit) relative to the current block being coded within the current frame (or other coded unit). A predictive block is a block that is found to closely match the block to be coded in terms of pixel difference, which may be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. A motion vector may also indicate the displacement of partitions of a macroblock. Motion compensation may involve obtaining or generating a predictive block based on the motion vector determined by motion estimation. As mentioned, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated.

In the case of inter-coding, motion estimation unit 42 calculates motion vectors for video blocks of inter-coded frames by comparing them with video blocks of reference frames in reference picture memory 64. Motion compensation unit 44 may also interpolate sub-integer pixels of reference frames (e.g., I-frames or P-frames). As an example, motion vectors may be predicted from two lists of reference frames: list 0, which includes reference frames that have a display order earlier than the display order of the current frame being encoded, and list 1, which includes reference frames that have a display order later than the display order of the current frame being encoded. Thus, the data stored in reference picture memory 64 may be organized according to these two lists of reference frames.

Motion estimation unit 42 compares blocks of one or more reference frames from reference picture memory 64 to blocks to be encoded of a current frame (e.g., a P-frame or a B-frame). When a reference frame in reference picture memory 64 includes values for sub-integer pixels, the motion vector calculated by motion estimation unit 42 may refer to a sub-integer pixel position of the reference frame. Motion estimation unit 42 and/or motion compensation unit 44 may also be configured to calculate values for sub-integer pixel positions of reference frames stored in reference picture memory 64 when values for sub-integer pixel positions are not stored in reference picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44. A reference frame block identified by a motion vector may be referred to as a predictive block.

Motion compensation unit 44 may calculate prediction data based on the predictive block. Video encoder 20 forms a residual video block by subtracting the prediction data provided by motion compensation unit 44 from the original video block being coded. Summer 50 represents the component that performs this subtraction operation. Transform unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, producing a video block comprising residual transform coefficient values. Transform unit 52 may perform other transforms, such as those defined by the H.264 standard, which are conceptually similar to DCT. As other examples, transform unit 52 may perform a wavelet transform, an integer transform, a subband transform, or another type of transform. Transform unit 52 applies the transform to the residual block, producing a block of residual transform coefficients. The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. Quantization unit 54 quantizes the residual transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter.

After quantization, entropy encoding unit 56 entropy codes the quantized transform coefficients. For example, entropy encoding unit 56 may perform content adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), probability interval partitioning entropy coding (PIPE), or another entropy coding technique. After entropy coding by entropy encoding unit 56, the encoded video data may be buffered or stored in coded picture buffer 92, transmitted to another device, and/or archived for later transmission or retrieval. In the case of context adaptive binary arithmetic coding, the context may be based on neighboring macroblocks.

In some cases, entropy coding unit 56 or another unit of video encoder 20 may be configured to perform other coding functions in addition to entropy coding. For example, entropy coding unit 56 may be configured to determine coded block pattern (CBP) values for macroblocks and partitions. Similarly, in some cases, entropy coding unit 56 may perform run-length coding of coefficients in a largest coding unit (LCU) or a sub-CU of an LCU. In particular, entropy coding unit 56 may apply a zigzag scan or other scan pattern to scan the transform coefficients in an LCU or partition and encode runs of zeros for further compression. Entropy coding unit 56 and/or other elements of video encoder 20 may also form decoding units from the encoded video data. For example, the decoding units may be sub-pictures, such as a sequence of treeblocks, one or more slices, one or more waves, and/or one or more image blocks. Entropy coding unit 56 and/or other elements of video encoder 20 may also add padding data for sub-pictures of different sizes to achieve byte alignment. Entropy encoding unit 56 may also construct header information with appropriate syntax elements for transmission in the encoded video bitstream. For example, the header information may include signaling data indicating whether the decoding unit is an access unit or a sub-access unit. This signaling data may include signaling the value of a sub-picture coded picture buffer preference flag signaled in the HRD parameters. For example, entropy encoding unit 56 and/or other elements of video encoder 20 may also add syntax elements such as a buffering period SEI message, signaling VUI parameters, signaling data indicating entry points for various sub-pictures, and/or a buffer removal time for the decoding unit.

Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the frames of reference picture memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reconstructed video block for storage in reference picture memory 64. The reconstructed video block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block to inter-code a block in a subsequent video frame.

Reference picture memory 64 may include buffer 90. Buffer 90 may be or include or be included in a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). Buffer 90 may include a picture buffer and/or a decoded picture buffer and may operate in accordance with any combination of the example coded picture buffer and/or decoded picture buffer behaviors described herein. For example, video encoder 20 may use buffer 90 to perform decoded block pattern (DPB) management and/or perform coded block pattern (CPB) management of coded picture buffer 92 in accordance with the techniques of this disclosure.

Coded picture buffer 92 may be or include or be included in a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). Although coded picture buffer 92 is shown as forming part of video encoder 20, in some examples, coded picture buffer 92 may form part of a device, unit, or module external to video encoder 20. For example, coded picture buffer 92 may form part of a stream scheduler unit (or delivery scheduler or hypothetical stream scheduler (HSS)) external to video encoder 20. Video encoder 20 may form decoding units from the encoded video data and provide the decoding units to the stream scheduler unit. In some examples, video encoder 20 may form decoding units having a varying number of bits or a varying number of blocks. The stream scheduler unit may implement the techniques of this disclosure to send decoding units, including sub-pictures such as a sequence of treeblocks, one or more slices, one or more waves, and/or one or more image blocks, to the video decoder for decoding at a time that can be indicated by an obtained (e.g., signaled) buffer removal time. In some examples, video encoder 20 may form decoding units that each include a number of coding blocks arranged consecutively in decoding order. The stream scheduler unit may further decapsulate the access unit to extract the one or more network abstraction layer (NAL) units that include the decoding unit. Similarly, the stream scheduler unit may decapsulate the NAL unit to extract the decoding unit.

According to the hypothetical reference decoder (HRD) behavior as modified by the techniques of this disclosure, video encoder 20 may store access units to and remove access units from coded picture buffer 92. For example, video encoder 20 may apply HRD parameters including an initial CPB removal delay, CPB size, bit rate, initial DPB output delay, and DPB size, as well as a buffer removal time for a decoding unit, and a value of a sub-picture coded picture buffer preference flag to signal whether the decoding unit of video data is an access unit or a subset of an access unit. Video encoder 20 may form an SEI message in the access unit that signals the buffering period and buffer removal time for the decoding unit. For example, video encoder 20 may provide video usability information (VUI) parameters having a syntax including a sub-picture CPB flag, such as in the example of Table 1 above.

The decoding unit may include sub-pictures of a common picture, and video encoder 20 may include a buffer removal time for each of the sub-pictures of the common picture in the SEI message for the access unit. Different sub-pictures may be encoded with different amounts of data, some of which are encoded with different numbers of bits or blocks, and video encoder 20 may form corresponding respective buffer removal times for each of the sub-pictures of the common picture. Video encoder 20 may also encode some pictures that have sub-pictures of the same data size. Other components may also perform one or more of the functions attributed above to video encoder 20. For example, an encapsulation unit of a source device (such as source device 12 of FIG. 1 ) may also form an SEI message that includes any of the above parameters.

Thus, video encoder 20 may specify that each sub-picture may include a number of coded blocks of a coded picture that are consecutive in decoding order, and that a coded block may be identical to a treeblock, or a subset of a treeblock. Video encoder 20 may specify that the coding of sub-pictures and the allocation of bits to different sub-pictures in a picture may be performed without requiring that each sub-picture (i.e., a set of treeblocks) in a picture be coded with the same amount of bits. Video encoder 20 may signal a CPB removal time for each sub-picture in the bitstream, rather than deriving the CPB removal time from a signaled picture-level CPB removal time. Video encoder 20 may also include more than one sub-picture in a slice and apply byte alignment at the end of each sub-picture. For example, video encoder 20 may also signal the entry point of each sub-picture with a value indicating the byte alignment of at least one of the sub-pictures within a larger set of video data, such as a slice, tile, or frame. Video encoder 20 may apply any one or more of these features in different examples according to this disclosure.

2 for reference picture memory 64, buffer 90, and coded picture buffer 92 are for illustrative purposes. Reference picture memory 64, buffer 90, and coded picture buffer 92 may be located in a single storage device or in any number of distinct storage devices. The storage devices may include any combination of volatile and/or non-volatile computer-readable media.

As such, video encoder 20 represents an example of a video coder that is configured to, among other things, store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for the one or more decoding units, remove the decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding units.

FIG3 is a block diagram illustrating an example of a video decoder 30 that decodes an encoded video sequence. In the example of FIG3, video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intra-prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference picture memory 82, a summer 80, a coded picture buffer 94, and a buffer 96. In some examples, video decoder 30 may perform a decoding pass that is generally reciprocal to the encoding pass described with respect to video encoder 20 (FIG. 2). Motion compensation unit 72 may generate prediction data based on motion vectors received from entropy decoding unit 70.

Motion compensation unit 72 may use motion vectors received in the bitstream to identify a prediction block in a reference frame in reference picture memory 82. Intra-prediction unit 74 may use an intra-prediction mode received in the bitstream to form a prediction block from spatially adjacent blocks. Inverse quantization unit 76 inverse quantizes (i.e., dequantizes) the quantized block coefficients provided in the bitstream and decoded by entropy decoding unit 70. The inverse quantization process may include a conventional process, such as defined by the H.264 decoding standard. The inverse quantization process may also include using a quantization parameter QP _Y calculated by encoder 20 for each macroblock to determine a degree of quantization and, likewise, a degree of inverse quantization that should be applied.

Inverse transform unit 78 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to produce a residual block in the pixel domain. Motion compensation unit 72 produces motion compensated blocks, possibly performing interpolation based on interpolation filters. Identifiers of interpolation filters to be used for motion estimation with sub-pixel precision may be included in syntax elements. Motion compensation unit 72 may use interpolation filters, as used by video encoder 20, during encoding of a video block to calculate interpolated values for sub-integer pixels of a reference block. Motion compensation unit 72 may determine the interpolation filters used by video encoder 20 based on received syntax information and use the interpolation filters to produce a predictive block.

Motion compensation unit 72 uses some of the following: syntax information to determine the size of the macroblocks used to encode the frames of the encoded video sequence; partition information describing how each macroblock of the frames of the encoded video sequence is partitioned; a mode indicating how each partition is encoded; one or more reference frames (and reference frame lists) for each inter-coded macroblock or partition; and other information used to decode the encoded video sequence. Summer 80 sums the residual block with the corresponding prediction block produced by motion compensation unit 72 or the intra-prediction unit to form a decoded block.

Reference picture memory 82 may include buffer 96. Buffer 96 may be or include a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). Buffer 96 may include one or more picture buffers and/or one or more decoded picture buffers, and may operate according to any combination of the example coded picture buffer and/or decoded picture buffer behaviors described herein. For example, video decoder 30 may use buffer 96 to perform DPB management and/or perform CPB management of coded picture buffer 94 according to the techniques of this disclosure.

Coded picture buffer 94 may be implemented as a data storage device, such as any permanent or volatile memory capable of storing data, such as synchronous dynamic random access memory (SDRAM), embedded dynamic random access memory (eDRAM), or static random access memory (SRAM). Coded picture buffer 94 may operate according to any combination of the example coded picture buffer behaviors disclosed herein.

Although coded picture buffer 94 is shown as forming part of video decoder 30, in some examples, coded picture buffer 94 may form part of a device, unit, or module external to video decoder 30. For example, coded picture buffer 94 may form part of a stream scheduler unit external to video decoder 30. The stream scheduler unit may implement the techniques of this disclosure to send decoding units, including sub-pictures such as a treeblock sequence, one or more slices, one or more waves, and/or one or more image blocks, to video decoder 30 for decoding at a time indicated by an obtained (e.g., signaled) buffer removal time. The stream scheduler unit may further decapsulate the access unit to extract the one or more network abstraction layer (NAL) units that comprise the decoding unit. Similarly, the stream scheduler unit may decapsulate the NAL unit to extract the decoding unit.

According to the hypothetical reference decoder (HRD) behavior as modified by the techniques of this disclosure, video decoder 30 may receive access units and store and remove access units to and from coded picture buffer 94. For example, video decoder 30 may decode and obtain HRD parameters, including an initial CPB removal delay, CPB size, bit rate, initial DPB output delay, and DPB size, as well as a buffer removal time for a decoding unit, and a value of a sub-picture coded picture buffer preference flag that signals whether the decoding unit of video data is an access unit or a subset of an access unit. Video decoder 30 may decode and obtain an SEI message in the access unit that signals the buffering period and buffer removal time for the decoding unit. For example, video decoder 30 may decode and obtain video usability information (VUI) parameters having a syntax that includes a sub-picture CPB flag, such as in the example of Table 1 above.

The decoding unit may include sub-pictures of a common picture, and video decoder 30 may decode and obtain the buffer removal time for each of the sub-pictures of the common picture in the SEI message for the access unit. Different sub-pictures may be encoded with different amounts of data, some of which are encoded with different numbers of bits or blocks, and video decoder 30 may decode and obtain the corresponding respective buffer removal time for each of the sub-pictures of the common picture. Video decoder 30 may also decode and obtain some pictures that have sub-pictures of the same data size.

Thus, video decoder 30 can decode and obtain a sub-picture that may include several coded blocks of a coded picture that are consecutive in decoding order, such that a coded block may be identical to a treeblock, or a subset of a treeblock. In some examples, video decoder 30 can decode and obtain the CPB removal time for each sub-picture in the bitstream, rather than deriving the CPB removal time from a signaled picture-level CPB removal time. Video decoder 30 may also decode and obtain more than one sub-picture in a slice, and may receive byte offset information indicating the starting point for each decoding unit to determine where each decoding unit begins, and decode and obtain information about additional non-data signals or padding signals that provide byte alignment at the end of each sub-picture. For example, video decoder 30 may also obtain the entry point of each sub-picture using a value indicating the byte alignment of at least one of the sub-pictures within a larger set of video data, such as a slice, tile, or frame. Video decoder 30 may apply any one or more of these features in different examples according to this disclosure.

3 are for illustrative purposes. Reference picture memory 82, buffer 96, and coded picture buffer 94 may be located in a single storage device or in any number of distinct storage devices. The storage devices may include any combination of volatile and/or non-volatile computer-readable media.

As such, video decoder 30 represents an example of a video coder that is configured to store one or more decoding units of video data in a picture buffer, obtain respective buffer removal times for the one or more decoding units, remove the decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units, and decode video data corresponding to the removed decoding units.

FIG4 is a block diagram illustrating an example destination device 100 that may implement any or all of the techniques of this disclosure. In this example, destination device 100 includes an input interface 102, a stream scheduler 104, a coded picture buffer 106, a video decoder 108, a decoded picture buffer 110, a rendering unit 112, and an output interface 114. Destination device 100 may generally correspond to destination device 14 ( FIG1 ). Input interface 102 may include any input interface capable of receiving a coded bitstream of video data. For example, input interface 102 may include receiver 26 and/or modem 28 as in FIG1 , a network interface such as a wired or wireless interface, a memory or storage interface, a disk drive for reading data from an optical disk (such as an optical drive interface or a magnetic media interface), or other interface components.

Input interface 102 may receive a coded bitstream comprising video data and provide the bitstream to stream scheduler 104. In accordance with the techniques of this disclosure, stream scheduler 104 extracts units of video data, such as access units and/or decoding units, from the bitstream and stores the extracted units in coded picture buffer 106. As such, stream scheduler 104 represents an example implementation of an HSS as discussed in the examples above. Coded picture buffer 106 may be generally identical to coded picture buffer 94 (FIG. 3), except that, as shown in FIG. 4, coded picture buffer 106 is separate from video decoder 108. In different examples, coded picture buffer 106 may be separate from video decoder 108 or integrated as part of video decoder 108.

Video decoder 108 includes a decoded picture buffer 110. Video decoder 108 may be generally consistent with video decoder 30 of Figures 1 and 3. Decoded picture buffer 110 may be generally consistent with buffer 96. Thus, video decoder 108 may decode decoding units of coded picture buffer 106 according to the techniques of this disclosure.

Furthermore, video decoder 108 may output decoded pictures from decoded picture buffer 110 in accordance with the techniques of this disclosure, as discussed above. Video decoder 108 may pass the output pictures to rendering unit 112. Rendering unit 112 may crop the pictures as discussed above in accordance with the techniques of this disclosure, and then pass the cropped pictures to output interface 114. Output interface 114 may, in turn, provide the cropped pictures to a display device, which may be substantially identical to display device 32. The display device may form part of destination device 100 or may be communicatively coupled to destination device 100. For example, the display device may include a screen, touch screen, projector, or other display unit integrated with destination device 100, or may include a separate display such as a television, monitor, projector, touch screen, or other device communicatively coupled to destination device 100. The communicative coupling may include a wired or wireless coupling, such as via a coaxial cable, a composite video cable, a component video cable, a High-Definition Multimedia Interface (HDMI) cable, radio frequency broadcast, or other wired or wireless coupling.

FIG5 is a flowchart illustrating an example method according to techniques of this disclosure, the method including, for example, by the video decoder 30 of FIG1 or 3 or the video decoder 108 of FIG4 (collectively, “video decoder 30/108”), removing decoding units of video data from a picture buffer according to obtained buffer removal times. The example method of FIG5 may be described as being performed by the video decoder 30/108 as an example, with it being understood that any one or more aspects of the method of FIG5 may also be performed or implemented by other devices or components. In the example of FIG5, the video decoder 30/108 may store one or more decoding units of video data in a picture buffer (202). The video decoder 30/108 may obtain respective buffer removal times for one or more decoding units, wherein obtaining the respective buffer removal times includes receiving respective signaled values indicating respective buffer removal times for at least one of the decoding units (204). The video decoder 30/108 may remove the decoding units from the picture buffer according to the obtained buffer removal times for each of the decoding units (206). Video decoder 30/108 may also decode video data corresponding to the removed decoding units, where decoding the video data includes decoding at least one of the decoding units (208).In other examples, video decoder 30/108 and/or other devices or elements may also perform different or additional functions.

FIG6 is a flowchart illustrating another example method according to the techniques of this disclosure, the method being similar in some respects to the method of FIG5 , including, for example, by the video decoder 30 of FIG1 or 3 or the video decoder 108 of FIG4 (collectively, “video decoder 30/108”), removing decoding units of video data from a picture buffer according to obtained buffer removal times. The example method of FIG6 may also be described as being performed by the video decoder 30/108 as an example, with it being understood that any one or more aspects of the method of FIG6 may also be performed or implemented by other devices or components. In the example of FIG6 , the video decoder 30/108 may store one or more decoding units of video data in a coded picture buffer (CPB) (402), obtain corresponding buffer removal times for the one or more decoding units (404), remove decoding units from the CPB according to the obtained buffer removal times for each of the decoding units (406), determine whether the CPB operates at the access unit level or the sub-picture level (408), and decode video data corresponding to the removed decoding units (410). If the CPB operates at the access unit level, decoding the video data includes decoding the access unit contained in the decoding unit (412). If the CPB operates at the sub-picture level, decoding the video data includes decoding a subset of the access units contained in the decoding unit (414).

For example, if the video decoder 30/108 determines that the CPB operates at the access unit level, the video decoder 30/108 may decode the access unit corresponding to the video data of the removed decoding unit (412). If the video decoder 30/108 determines that the CPB operates at the sub-picture level, the video decoder 30/108 may decode the access unit subset corresponding to the video data of the removed decoding unit (414). For example, the video decoder 30/108 may determine that the one or more decoding units include an access unit by determining that a sub-picture coded picture buffer preferred flag (e.g., SubPicCpbPreferredFlag) is negative or has a value of 0 or determining that a sub-picture coded picture buffer parameter present flag (e.g., sub_pic_cpb_params_present_flag) is negative or has a value of 0. The video decoder 30/108 may determine that one or more decoding units include an access unit subset by determining that both a sub-picture coded picture buffer preference flag (e.g., SubPicCpbPreferredFlag) is positive or has a value of 1 and a sub-picture coded picture buffer parameter present flag (e.g., sub_pic_cpb_params_present_flag) is positive or has a value of 1. The video decoder 30/108 may also use a single sub-picture coded picture buffer flag, SubPicCpbFlag, which may be set to SubPicCpbPreferredFlag && sub_pic_cpb_params_present_flag, to determine whether both base flags are positive and that the video decoder 30/108 may code for the access unit subset.

FIG7 is a flowchart illustrating another example method of processing video data according to techniques of this disclosure, the method including, for example, outputting a cropped picture in a boosting process by the video decoder 30 of FIG1 or 3 or the video decoder 108 of FIG4 (collectively, “video decoder 30/108”). In the example of FIG7, the video decoder 30/108 may perform the boosting process, as described above with reference to the boosting process example, if any of certain conditions are met. In particular, if the current picture is an instantaneous decoding refresh (IDR) picture (302) and the no previous picture output flag has a value not equal to 1 (304), which may include if the no previous picture output flag has a value that is not inferred to be equal to 1 or set to 1 by the HRD, for example, then the video decoder 30/108 may perform the boosting process. The video decoder 30/108 may also perform the boosting process if the number of pictures marked as needed for output in the decoded picture buffer (DPB) is greater than the number of reordered pictures at the current temporal layer (306). Video decoder 30/108 may also perform a boosting process if the number of pictures in the DPB with temporal layer identifier values lower than or equal to the temporal layer identifier value of the current picture is equal to the maximum picture buffer value of the current temporal layer plus 1 (308).

If any of the specified conditions (302 and 304, or 306, or 308) are met, the video decoder 30/108 may perform a boosting process as follows. The video decoder 30/108 may select a picture in the DPB that has the smallest picture order count (POC) value of the pictures and is marked as needed for output as the selected picture (312). The video decoder 30/108 may crop the selected picture as specified in the active sequence parameter set for the selected picture, thereby generating a cropped picture based on the selected picture (314). The video decoder 30/108 may output the cropped picture (316). The video decoder 30/108 may mark the selected picture as not needed for output (318).

In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. Thus, computer-readable media may generally correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) communication media such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and/or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

By way of example, and not limitation, these computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Likewise, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transitory, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, as used herein, the term "processor" may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Likewise, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or provided by a set of interoperable hardware units including one or more processors as described above, in conjunction with appropriate software and/or firmware.

Various embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims (26)

1. A method for decoding video data, the method comprising: Based on the sub-image decoded image buffer preferred flag and the sub-image decoded image buffer parameter existence flag, the value of at least one flag is determined. The value of the sub-image decoded image buffer preferred flag is specified by an external means or set to 0 when not specified by an external means. The sub-image decoded image buffer parameter existence flag indicates whether the parameters required for a subset of the decoding access unit exist. The value of at least one flag determines whether the decoded image buffer (CPB) operates at the access unit level or the sub-image level; and Depending on the level at which the CPB operates, one of the following two sets of operations is performed: When the CPB operates at the access unit level: The decoding unit stored in the CPB is determined to include an access unit; Remove the decoding unit, including the corresponding access unit, from the CPB; and Decoding the removed decoding unit, which includes the corresponding access unit; or When the CPB operates at the sub-image level: It is determined that the decoding units stored in the CPB include a subset of the access units; Remove the decoding unit comprising a subset of the corresponding access unit from the CPB; and Decoding the subset of decoding units that have been removed, including the corresponding access units. 2. The method of claim 1, further comprising determining the value of the preferred flag of the decoded image buffer for the sub-image. 3. The method of claim 1, further comprising determining the value of the presence flag of the decoded image buffer parameter of the sub-image. 4. The method according to claim 1, Based on the fact that the values of the preferred flag of the decoded image buffer for the sub-image and the present flag of the parameter of the decoded image buffer for the sub-image are each set to 1, the value of at least one flag is set to 1. The value of at least one of the flags being 1 indicates that the CPB is operating at the sub-picture level. 5. The method according to claim 1, Based on the fact that at least one of the values of the sub-image decoded image buffer preferred flag and the sub-image decoded image buffer parameter existence flag is set to 0, the value of the at least one flag is set to 0, wherein the value of the at least one flag being 0 indicates that the CPB is operating at the access unit level. 6. An apparatus for decoding video data, the apparatus comprising: A memory configured to store at least a portion of the video data; and A video decoder for processing said portion of the video data, the video decoder being configured to: The value of at least one flag is determined based on the sub-image decoded image buffer preference flag and the sub-image decoded image buffer parameter presence flag. The value of the sub-image decoded image buffer preference flag is specified externally or set to 0 when not specified externally. The sub-image decoded image buffer parameter presence flag indicates whether the parameters required for a subset of the decoding access unit exist. The value of at least one flag determines whether the decoded image buffer (CPB) operates at the access unit level or the sub-image level; and Depending on the level at which the CPB operates, the video decoder is configured to perform one of the following two sets of operations: When the operation of the CPB at the access unit level is determined: The decoding unit stored in the CPB is determined to include an access unit; Remove the decoding unit, including the corresponding access unit, from the CPB; and Decoding the removed decoding unit, which includes the corresponding access unit; or When the CPB operation at the sub-image level is determined: It is determined that the decoding units stored in the CPB include a subset of the access units; Remove the decoding unit comprising a subset of the corresponding access unit from the CPB; and Decoding the subset of decoding units that have been removed, including the corresponding access units. 7. The apparatus of claim 6, wherein the video decoder is further configured to determine the value of the preferred flag of the sub-picture via the decoded picture buffer. 8. The apparatus of claim 6, wherein the video decoder is further configured to determine the value of the presence flag of the sub-picture via the decoded picture buffer parameter. 9. The apparatus of claim 6, wherein the video decoder is further configured to: Based on the fact that the value of the preferred flag of the decoded image buffer for the sub-image and the value of the parameter existence flag of the decoded image buffer for the sub-image are each set to 1, the value of at least one flag is set to 1. The value of at least one of the flags being 1 indicates that the CPB is operating at the sub-picture level. 10. The apparatus of claim 6, wherein the video decoder is further configured to... Based on the fact that at least one of the values of the preferred flag of the decoded image buffer for the sub-image and the value of the flag containing the parameters of the decoded image buffer for the sub-image is set to 0, the value of the at least one flag is set to 0. The value of at least one of the flags being 0 indicates that the CPB is operating at the access unit level. 11. The apparatus of claim 6, wherein the apparatus comprises at least one of the following: One or more integrated circuits; One or more microprocessors; One or more digital signal processors (DSPs); One or more field-programmable gate arrays (FPGAs); Desktop computer; Portable computers; Tablet PC; cell phone; TV set; camera; Display devices; Digital media player; Video game console; Video game devices; Video streaming devices; or Wireless communication device. 12. The apparatus of claim 6, further comprising a camera configured to capture at least a portion of the video data. 13. The apparatus of claim 6, further comprising a display configured to output at least a portion of the video data. 14. An apparatus for decoding video data, the apparatus comprising: A means for determining the value of at least one flag based on a sub-image decoded image buffer preference flag and a sub-image decoded image buffer parameter presence flag, wherein the value of the sub-image decoded image buffer preference flag is specified externally or set to 0 when not specified externally, and the sub-image decoded image buffer parameter presence flag indicates whether a parameter required for a subset of the decoding access unit exists. A means for determining whether the decoded image buffer CPB operates at the access unit level or the sub-image level based on the value of the at least one flag; Means for determining, when the CPB operates at the access unit level, that a decoding unit stored in the CPB includes an access unit; Means for removing a decoding unit comprising a corresponding access unit from the CPB when the CPB operates at the access unit level; A means for decoding a decoding unit including the corresponding access unit when the CPB operates at the access unit level; Means for determining, when the CPB operates at the sub-picture level, that the decoding units stored in the CPB include a subset of the access units; Means for removing a subset of decoding units comprising the corresponding access units from the CPB when the CPB operates at the sub-image level; and A means for decoding a subset of the corresponding access unit that has been removed when the CPB operates at the sub-picture level. 15. The apparatus of claim 14, further comprising means for determining the value of the preferred flag of the decoded image buffer for the sub-image. 16. The apparatus of claim 14, further comprising means for determining the value of the presence flag of the sub-image via the decoded image buffer parameter. 17. The device of claim 14, further comprising: A means for setting the value of at least one flag to 1 based on the value of a preferred flag of the sub-image decoded image buffer and the value of a flag indicating the presence of parameters of the sub-image decoded image buffer, wherein the value of at least one flag being 1 indicates that the CPB is operating at the sub-image level. 18. The apparatus of claim 14, further comprising means for setting the value of the at least one flag to 0 based on at least one of the value of the sub-image decoded image buffer preferred flag and the value of the sub-image decoded image buffer parameter existence flag, wherein the value of the at least one flag being 0 indicates that the CPB operates at the access unit level. 19. The device of claim 14, wherein the device comprises at least one of the following: One or more integrated circuits; One or more microprocessors; One or more digital signal processors (DSPs); One or more field-programmable gate arrays (FPGAs); Desktop computer; Portable computers; Tablet PC; cell phone; TV set; camera; Display devices; Digital media player; Video game console; Video game devices; Video streaming devices; or Wireless communication device, including video decoding device. 20. The apparatus of claim 14, further comprising means for capturing at least a portion of the video data. 21. The apparatus of claim 14, further comprising means for outputting at least a portion of the video data for display. 22. A non-transitory computer-readable storage medium comprising instructions stored thereon, which, when executed, cause one or more processors of a video decoding apparatus to: The value of at least one flag is determined based on the sub-image decoded image buffer preference flag and the sub-image decoded image buffer parameter presence flag. The value of the sub-image decoded image buffer preference flag is specified externally or set to 0 when not specified externally. The sub-image decoded image buffer parameter presence flag indicates whether the parameters required for a subset of the decoding access unit exist. The value of at least one flag determines whether the decoded image buffer (CPB) operates at the access unit level or the sub-image level; and Depending on the level at which the CPB operates, the one or more processors are further caused to perform one of the following two sets of operations: When the CPB operates at the access unit level: The decoding unit stored in the CPB is determined to include an access unit; Remove the decoding unit, including the corresponding access unit, from the CPB; and Decoding the removed decoding unit, which includes the corresponding access unit; or When the CPB operates at the sub-image level: It is determined that the decoding units stored in the CPB include a subset of the access units; Remove the decoding unit comprising a subset of the corresponding access unit from the CPB; and Decoding the subset of decoding units that have been removed, including the corresponding access units. 23. The non-transitory computer-readable storage medium of claim 22, wherein the instructions further cause the one or more processors to determine the value of the preferred flag of the sub-image via the decoded image buffer. 24. The non-transitory computer-readable storage medium of claim 22, wherein the instructions further cause the one or more processors to determine the value of the sub-image decoded image buffer parameter presence flag. 25. The non-transitory computer-readable storage medium of claim 22, wherein the instructions further cause the one or more processors to: Based on the fact that the values of the preferred flag of the decoded image buffer for the sub-image and the present flag of the parameter of the decoded image buffer for the sub-image are each set to 1, the value of at least one flag is set to 1. The value of at least one of the flags being 1 indicates that the CPB operates at the sub-picture level. 26. The non-transitory computer-readable storage medium of claim 22, wherein the instructions further cause the one or more processors to set the value of at least one flag to 0 based on at least one of the value of the sub-image decoded image buffer preferred flag and the value of the sub-image decoded image buffer parameter existence flag, wherein the value of at least one flag being 0 indicates that the CPB operates at the access unit level.