HK1204181B - Coding sei nal units for video coding - Google Patents

Description

Decode SEI NAL units for video decoding

Related applications

This application claims the rights of:

U.S. Provisional Application No. 61/670,066, filed Jul. 10, 2012, which is hereby incorporated by reference herein in its entirety.

Technical Field

This disclosure relates generally to processing video data, and more particularly to random access pictures for use in video data.

Background Art

Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones", video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in the standards defined by ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262, ISO/IEC MPEG-2 Visual, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10 Advanced Video Coding (AVC), and the currently developing High Efficiency Video Coding (HEVC) standard, and extensions of such standards. By implementing such video coding techniques, video devices can more efficiently transmit, receive, encode, decode, and/or store digital video information.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based predictive video coding, a video slice (e.g., a video frame or a portion of a video frame) may be partitioned into video blocks, which may also be referred to as macroblocks, treeblocks, coding tree units (CTUs), coding tree blocks (CTBs), coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames.

Spatial prediction or temporal prediction results in coding a predictive block for the block. Residual data represents the pixel differences between the original block to be coded and the predictive block. Pixels may also be referred to as picture elements, pels, or samples. Inter-coded blocks are encoded based on motion vectors pointing to a block of reference samples forming the predictive block, and residual data indicating the difference between the coded block and the predictive block. Intra-coded blocks are encoded based on an intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to the transform domain, generating residual transform coefficients that may then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, may be scanned to generate a one-dimensional vector of transform coefficients, and entropy coding may be applied to achieve even greater compression.

Summary of the Invention

In general, this disclosure describes techniques for processing video data. Specifically, this disclosure describes techniques that can be used to reduce latency in video applications, such as interactive applications, provide improvements in random access to coded video sequences, and provide information for video content with a fixed picture rate and supporting temporal scalability.

In one example, a method of decoding video data includes: decapsulating a slice of a random access point (RAP) picture of a bitstream from a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures, and whether the RAP picture is an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; determining whether the RAP picture can have associated leading pictures based on the NAL unit type value; and decoding video data of the bitstream following the RAP picture based on the determining whether the RAP picture can have associated leading pictures.

In another example, a device for decoding video data includes a processor configured to: decapsulate a slice of a random access point (RAP) picture of a bitstream from a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures, and whether the RAP picture is an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; determine whether the RAP picture can have associated leading pictures based on the NAL unit type value; and decode video data of the bitstream following the RAP picture based on the determination of whether the RAP picture can have associated leading pictures.

In another example, a device for decoding video data includes: means for decapsulating a slice of a random access point (RAP) picture of a bitstream from a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures, and whether the RAP picture is an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; means for determining whether the RAP picture can have associated leading pictures based on the NAL unit type value; and means for decoding video data of the bitstream following the RAP picture based on the determining whether the RAP picture can have associated leading pictures.

In another example, a computer-readable storage medium storing instructions that, when executed, cause a processor to: decapsulate a slice of a random access point (RAP) picture of a bitstream from a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures, and whether the RAP picture is an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; determine whether the RAP picture can have associated leading pictures based on the NAL unit type value; and decode video data of the bitstream following the RAP picture based on the determination of whether the RAP picture can have associated leading pictures.

In another example, a method of generating a bitstream including video data includes determining whether a random access point (RAP) picture is of a type that can have associated leading pictures, and whether the RAP picture comprises an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; encapsulating a slice of the RAP picture in a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures; and generating a bitstream including the NAL unit.

In another example, a device for generating a bitstream including video data includes a processor configured to: determine whether a random access point (RAP) picture is of a type that can have associated leading pictures, and whether the RAP picture comprises an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; encapsulate slices of the RAP picture in a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures; and generate a bitstream including the NAL unit.

In another example, a device for generating a bitstream including video data includes: means for determining whether a random access point (RAP) picture is of a type that can have associated leading pictures, and whether the RAP picture comprises an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; means for encapsulating a slice of the RAP picture in a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures; and means for generating a bitstream including the NAL unit.

In another example, a computer-readable storage medium storing instructions that, when executed, cause a processor to: determine whether a random access point (RAP) picture is of a type that can have associated leading pictures, and whether the RAP picture comprises an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; encapsulate a slice of the RAP picture in a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures; and generate a bitstream including the NAL unit.

In another example, a method of decoding video data includes: determining, for a supplemental enhancement information (SEI) network abstraction layer (NAL) unit of a bitstream, a NAL unit type value of the SEI NAL unit indicating whether the NAL unit comprises a prefix SEI NAL unit including a prefix SEI message or a suffix SEI NAL unit including a suffix SEI message; and decoding video data of the bitstream following the SEI NAL unit based on whether the SEI NAL unit is the prefix SEI NAL unit or the suffix SEI NAL unit and data of the SEI NAL unit.

In another example, a device for decoding video data includes a processor configured to: determine, for a supplemental enhancement information (SEI) network abstraction layer (NAL) unit of a bitstream, a NAL unit type value of the SEI NAL unit indicating whether the NAL unit comprises a prefix SEI NAL unit including a prefix SEI message or a suffix SEI NAL unit including a suffix SEI message; and decode video data of the bitstream following the SEI NAL unit based on whether the SEI NAL unit is the prefix SEI NAL unit or the suffix SEI NAL unit and data of the SEI NAL unit.

In another example, a device for decoding video data includes: means for determining, for a supplemental enhancement information (SEI) network abstraction layer (NAL) unit of a bitstream, whether a NAL unit type value of the SEI NAL unit indicates whether the NAL unit comprises a prefix SEI NAL unit including a prefix SEI message or a suffix SEI NAL unit including a suffix SEI message; and means for decoding video data of the bitstream following the SEI NAL unit based on whether the SEI NAL unit is the prefix SEI NAL unit or the suffix SEI NAL unit and data of the SEI NAL unit.

In another example, a computer-readable storage medium storing instructions that, when executed, cause a processor to: determine, for a supplemental enhancement information (SEI) network abstraction layer (NAL) unit of a bitstream, a NAL unit type value of the SEI NAL unit indicating whether the NAL unit comprises a prefix SEI NAL unit including a prefix SEI message or a suffix SEI NAL unit including a suffix SEI message; and decode video data of the bitstream following the SEI NAL unit based on whether the SEI NAL unit is the prefix SEI NAL unit or the suffix SEI NAL unit and data of the SEI NAL unit.

In another example, a method of generating a bitstream including video data includes determining whether a supplemental enhancement information (SEI) message is a prefix SEI message or a suffix SEI message, wherein the SEI message includes data related to the encoded video data; encapsulating the SEI message in a SEI NAL unit, wherein the SEI NAL unit includes a NAL unit type value indicating whether the SEI NAL unit is a prefix SEI NAL unit or a suffix SEI NAL unit, and whether the SEI message is a prefix SEI message or a suffix SEI message; and generating a bitstream including at least the SEI NAL unit.

In another example, a device for generating a bitstream including video includes a processor configured to: determine whether a supplemental enhancement information (SEI) message is a prefix SEI message or a suffix SEI message, wherein the SEI message includes data related to encoded video data; encapsulate the SEI message in a SEI NAL unit, wherein the SEI NAL unit includes a NAL unit type value indicating whether the SEI NAL unit is a prefix SEI NAL unit or a suffix SEI NAL unit, and whether the SEI message is a prefix SEI message or a suffix SEI message; and generate a bitstream including at least the SEI NAL unit.

In another example, a device for generating a bitstream including video data includes: means for determining whether a supplemental enhancement information (SEI) message is a prefix SEI message or a suffix SEI message, wherein the SEI message includes data related to the encoded video data; means for encapsulating the SEI message in a SEI NAL unit, wherein the SEI NAL unit includes a NAL unit type value indicating whether the SEI NAL unit is a prefix SEI NAL unit or a suffix SEI NAL unit, and whether the SEI message is a prefix SEI message or a suffix SEI message; and means for generating a bitstream including at least the SEI NAL unit.

In another example, a computer-readable storage medium having instructions stored thereon that, when executed, cause a processor to: determine whether a supplemental enhancement information (SEI) message is a prefix SEI message or a suffix SEI message, wherein the SEI message includes data related to encoded video data; encapsulate the SEI message in a SEI NAL unit, wherein the SEI NAL unit includes a NAL unit type value indicating whether the SEI NAL unit is a prefix SEI NAL unit or a suffix SEI NAL unit, and whether the SEI message is a prefix SEI message or a suffix SEI message; and generate a bitstream including at least the SEI NAL unit.

In another example, a method of presenting video data includes determining an integer value for the video data; determining a difference between a presentation time of a first picture and a presentation time of a second picture, wherein the difference is equal to the integer value multiplied by a clock tick value; and presenting the first picture and the second picture based on the determined difference.

In another example, a device for presenting video data includes a processor configured to: determine an integer value for the video data; determine a difference between a presentation time of a first picture and a presentation time of a second picture, wherein the difference is equal to the integer value multiplied by a clock tick value; and present the first picture and the second picture based on the determined difference.

In another example, a device for presenting video data includes: a device for determining an integer value for the video data; a device for determining a difference between a presentation time of a first picture and a presentation time of a second picture, wherein the difference is equal to the integer value multiplied by a clock tick value; and a device for presenting the first picture and the second picture based on the determined difference.

In another example, a computer-readable storage medium stores instructions that, when executed, cause a processor to: determine an integer value for the video data; determine a difference between a presentation time of a first picture and a presentation time of a second picture, wherein the difference is equal to the integer value multiplied by a clock tick value; and present the first picture and the second picture based on the determined difference.

In another example, a method of generating a bitstream including video data includes generating data indicating whether a difference between a presentation time of a first picture and a presentation time of a second picture is an integer multiple of a clock tick value; and when the data indicates that the difference is the integer multiple of the clock tick value, generating data representing the integer multiple.

In another example, a device for generating a bitstream including video data includes a processor configured to: generate data indicating whether a difference between a presentation time of a first picture and a presentation time of a second picture is an integer multiple of a clock tick value; and when the data indicates that the difference is the integer multiple of the clock tick value, generate data representing the integer multiple.

In another example, a device for generating a bitstream including video data includes: means for generating data indicating whether a difference between a presentation time of a first picture and a presentation time of a second picture is an integer multiple of a clock tick value; and means for generating data representing the integer multiple when the data indicates that the difference is the integer multiple of the clock tick value.

In another example, a computer-readable storage medium stores instructions that, when executed, cause a processor to: generate data indicating whether the difference between the presentation time of a first image and the presentation time of a second image is an integer multiple of a clock tick value; and when the data indicates that the difference is the integer multiple of the clock tick value, generate data representing the integer multiple.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a conceptual diagram illustrating a video sequence coded according to predictive video coding techniques.

2 is a conceptual diagram illustrating an example of a coded video sequence.

3 is a block diagram illustrating an example video encoding and decoding system that may utilize the techniques described in this disclosure.

4 is a block diagram illustrating an example encapsulation unit that may implement the techniques described in this disclosure.

5 is a flow diagram illustrating an example of generating VCL NAL units according to the techniques of this disclosure.

6 is a flow diagram illustrating an example of generating non-VCL NAL units according to the techniques of this disclosure.

7 is a flow diagram illustrating an example of signaling a presentation time delta value.

8 is a block diagram illustrating an example video encoder that may implement the techniques described in this disclosure.

9 is a flow diagram illustrating an example of determining a presentation time delta value.

10 is a block diagram illustrating an example video decoder that may implement the techniques described in this disclosure.

DETAILED DESCRIPTION

This disclosure describes various improved video coding designs. Specifically, this disclosure describes techniques that can be used to reduce delay in video applications, such as conversational applications, and to provide improvements in random access to coded video sequences.

Digital video devices implement video compression techniques to more efficiently encode and decode digital video information. Video compression techniques may be defined according to video coding standards, such as AVC or HEVC. The ITU-T H.264/MPEG-4 (AVC) standard was formulated by the ITU-T Video Coding Experts Group (VCEG) in conjunction with the ISO/IEC Moving Picture Experts Group (MPEG) as a product of a collective effort known as the Joint Video Team (JVT). The H.264 standard was developed by an ITU-T study group and is described in ITU-T Recommendation H.264 (Advanced Video Coding for General Audiovisual Services) dated March 2005, which may be referred to herein as the H.264 standard or H.264 specification or the H.264/AVC standard or specification. The Joint Video Team (JVT) continues to work on extensions to H.264/MPEG-4 AVC.

The latest working draft of HEVC, referred to as “HEVC Working Draft 7” or “WD7”, is described in document JCTVC-I1003_d5 (Bross et al., “WD7: Working Draft 7 of High-Efficiency Video Coding (HEVC)”, Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, 9th Meeting: Geneva, Switzerland, April 27, 2012 - May 7, 2012). Additionally, another recent working draft of HEVC (Working Draft 9) is described in document HCTVC-K1003_d7 (Bross et al., "High Efficiency Video Coding (HEVC) Text Specification Draft 9," Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, 11th Meeting: Shanghai, China, October 2012). The upcoming HEVC standard may also be referred to as ISO/IEC 23008-HEVC, which is intended to be the standard number for the delivery version of HEVC. In some aspects, the techniques described in this disclosure may be applied to devices that generally conform to H.264 and/or the upcoming HEVC standard. Although the techniques of this disclosure are described with respect to the H.264 standard and the upcoming HEVC standard, the techniques of this disclosure are generally applicable to any video coding standard.

A video sequence typically comprises a series of video frames, also referred to as pictures. Examples of video applications that encode and/or decode video sequences include local playback, streaming, broadcast, multicast, and conversational applications. Conversational applications include video telephony and video conferencing, and are also referred to as low-latency applications. Conversational applications require a relatively low end-to-end latency for the entire system, i.e., the delay between the time a video frame is captured at a first digital video device and the time it is displayed at a second digital video device. For conversational applications, an acceptable end-to-end latency is typically less than 400 ms, with an end-to-end latency of approximately 150 ms considered excellent.

Each step associated with processing a video sequence may contribute to the overall end-to-end delay. Examples of delays associated with processing a video sequence include capture delay, preprocessing delay, encoding delay, transmission delay, receive buffering delay (for dejittering), decoding delay, decoded picture output delay, post-processing delay, and display delay. The delay associated with decoding a video sequence according to a particular video coding standard may be referred to as codec delay, and may include encoding delay, decoding delay, and decoded picture output delay. Codec delay should be minimized in interactive applications. Specifically, the coding structure of a video sequence should ensure that the output order of pictures in the video sequence is the same as the decoding order of the pictures in the video sequence, so that the decoded picture output delay is equal to zero. The coding structure of a video sequence refers, in part, to the allocation of picture types used to encode the video sequence.

A group of pictures (GOP) generally comprises a sequence of one or more pictures arranged according to display order. According to HEVC, a video encoder may divide a video frame or picture into a series of equally sized video blocks. A video block may have a luma component (denoted as Y) and two chroma components (denoted as U and V or Cb and Cr). Such video blocks may also be referred to as largest coding units (LCUs), treeblocks, or coding treeblock units (CTUs). An LCU of HEVC may be broadly similar to macroblocks of previous standards such as H.264/AVC. However, an LCU is not necessarily limited to a specific size. According to HEVC, syntax data within the bitstream may define an LCU based on the number of horizontal luma samples and/or vertical luma samples. For example, an LCU may be defined as comprising 64x64 or 32x32 luma samples. Additionally, an LCU may be partitioned into multiple coding units (CUs) according to a quadtree partitioning scheme. Generally, quadtree partitioning refers to recursively splitting a CU into four sub-CUs. Syntax data associated with the coded bitstream may define the maximum number of times an LCU can be split, referred to as the maximum CU depth, and may also define the minimum size of a CU. Thus, the bitstream may also define a smallest coding unit (SCU). For example, an SCU may be defined as comprising 8x8 luma samples.

Furthermore, according to HEVC, a video encoder may partition a picture into multiple slices, where each of the slices includes an integer number of LCUs. A slice may be an I slice, a P slice, or a B slice, where I, P, and B define how other video blocks are used to predict a CU. An I slice is predicted using an intra-prediction mode (e.g., predicted from a video block within the same frame). Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture. A P slice is predicted using a unidirectional inter-prediction mode (e.g., predicted from a video block in a previous frame). A B slice is predicted using a bidirectional inter-prediction mode (e.g., predicted from video blocks in a previous frame and a subsequent frame). Inter-coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or pictures of a video sequence.

FIG1 is a conceptual diagram illustrating a video sequence coded according to predictive video coding techniques. As illustrated in FIG1 , video sequence 100 includes pictures Pic ₁ through Pic _10. In the conceptual diagram of FIG1 , pictures Pic ₁ through Pic ₁₀ are arranged and sequentially numbered according to the order in which they will be displayed. As described in more detail below, the display order does not necessarily correspond to the decoding order. As illustrated in FIG1 , video sequence 100 includes GOPs ₁ and ₂ , where pictures Pic ₁ through Pic ₅ are included in GOP ₁ , and pictures Pic ₆ through Pic ₁₀ are included in GOP _2. FIG1 illustrates the partitioning of Pic ₅ into slices ₁ and ₂ , where each of slices ₁ and ₂ includes consecutive LCUs according to a raster scan from left to right and top to bottom. Although not shown, the other pictures illustrated in FIG1 can be partitioned into one or more slices in a similar manner. 1 also illustrates the concept of an I slice, a P slice, or a B slice for GOP _2. The arrows associated with each of Pic ₆ through Pic ₁₀ in GOP ₂ indicate whether the picture includes an I slice, a P slice, or a B slice based on the reference picture indicated by the arrow. In FIG1 , pictures Pic ₆ and Pic ₉ represent pictures that include an I slice (i.e., reference the pictures themselves), pictures Pic ₇ and Pic ₁₀ represent pictures that include a P slice (i.e., each references a previous picture), and Pic ₈ represents a picture that includes a B slice (i.e., references both a previous picture and a subsequent picture).

In HEVC, each of the video sequences: GOP, picture, slice, and CU can be associated with syntax data that describes the properties of the video coding. For example, a slice includes a header that includes syntax elements indicating whether the slice is an I slice, a P slice, or a B slice. In addition, HEVC includes the concept of parameter sets. A parameter set is a syntax structure that includes syntax elements that allow a video decoder to reconstruct a video sequence. HEVC utilizes a hierarchical parameter set mechanism, in which syntax elements are included in a type of parameter set based on the frequency with which the syntax elements are expected to change. The parameter set mechanism in HEVC decouples the transmission of infrequently changing information from the transmission of coded block data. In addition, in some applications, parameter sets can be transported "out-of-band," that is, they are not transported together with the unit containing the coded video data. Out-of-band transmission is generally reliable.

In HEVC WD7, a parameter set ID is used to identify a specific parameter set. In HEVC WD7, the parameter set ID is an unsigned integer exponential Golomb-coded (Exp-Golomb-coded) syntax element, starting from the left bit. HEVC WD7 defines the following parameter sets:

Video Parameter Set (VPS): A VPS is a syntax structure that contains syntax elements that apply to zero or more entire coded video sequences. That is, a VPS includes syntax elements that are expected to remain unchanged for a sequence of frames (e.g., picture order, number of reference frames, and picture size). A VPS is identified using a VPS ID. A sequence parameter set includes the VPS ID.

Sequence Parameter Set (SPS) - An SPS is a syntax structure that includes syntax elements that apply to zero or more entire coded video sequences. That is, an SPS includes syntax elements that are expected to remain unchanged for a sequence of frames (e.g., picture order, number of reference frames, and picture size). An SPS is identified using an SPS ID. A picture parameter set includes the SPS ID.

Picture Parameter Set (PPS) - A PPS is a syntax structure that includes syntax elements applicable to one or more pictures. That is, a PPS includes syntax elements that can change from picture to picture within a sequence (e.g., entropy coding mode, quantization parameter, and bit depth). A PPS parameter set is identified using a PPS ID. The slice header includes the PPS ID.

Adaptation Parameter Set (APS) - An APS is a syntax structure that contains syntax elements applicable to one or more pictures. An APS includes syntax elements that are expected to change within pictures of a sequence (e.g., block size and deblocking filtering). An APS set is identified using an APS ID. A slice header may include an APS ID.

According to the parameter set type defined in HEVC WD7, each SPS references the VPS ID, each PPS references the SPS ID, and each slice header references the PPS ID and possibly the APS ID. It should be noted that in some cases, a linear reference relationship including the VPS ID in the SPS and the SPS ID in the PPS may be inefficient. For example, although VPS is supported in HEVC WD7, most of the sequence level information parameters are still only present in the SPS. In addition to the parameter set concept, HEVC includes the concept of decoded video sequence and access unit. According to HEVC WD7, decoded video sequence and access unit are defined as follows:

Coded video sequence: A sequence of access units consisting of, in decoding order: a CRA access unit, an IDR access unit, or a BLA access unit that is the first access unit in the bitstream, followed by zero or more non-IDR and non-BLA access units that include up to all subsequent access units but do not include any subsequent IDR or BLA access units [CRA access units, IDR access units, and BLA access units are described in detail below].

Access unit: A set of NAL units that are consecutive in decoding order and contain one coded picture. In addition to coded slice NAL units of coded pictures, an access unit may also contain other NAL units that do not contain slices of coded pictures. Decoding of an access unit always produces a decoded picture.

NAL unit refers to a network abstraction layer unit. Thus, according to HEVC, the bitstream of coded video data includes a sequence of NAL units. An access unit is a set of NAL units that are arranged contiguously in decoding order and contain exactly one coded picture, and a coded video sequence includes a sequence of access units arranged in decoding order. FIG2 is a conceptual diagram illustrating an example of a coded video sequence. FIG2 represents an example of a coded video sequence 200 that may correspond to GOP ₂ illustrated in FIG1 . As illustrated in FIG2 , coded video sequence 200 includes access units corresponding to each of Pics ₆ to _10. The access units of coded video sequence 200 are arranged sequentially according to decoding order. It should be noted that the access unit corresponding to Pic ₉ precedes the access unit corresponding to Pic _8. Therefore, the decoding order does not correspond to the display order illustrated in FIG1 . In this example, this is due to the fact that Pic ₈ references Pic ₉ . Therefore, Pic ₉ must be decoded before Pic ₈ can be decoded. Figure 2 illustrates that the access unit corresponding to Pic ₉ includes NAL units: AU delimiter NAL unit 202, PPS NAL unit 204, slice ₁ NAL unit 206, and slice ₂ NAL unit 208. Each NAL unit may include a header that identifies the NAL unit type.

HEVC defines two categories of NAL unit types: coded slice NAL units (VCLs) and non-VCL NAL units. A coded slice NAL unit contains a slice of video data. In the example illustrated in FIG2 , slice ₁ NAL unit 206 and slice ₂ NAL unit 208 each contain a slice of video data and are examples of VCL NAL units. In the example of FIG2 , each of slice ₁ NAL unit 206 and slice ₂ NAL unit 208 may be an I slice. Non-VCLs include information other than a slice of video data. For example, non-VCLs may contain delimiter data or parameter sets. In the example illustrated in FIG2 , AU delimiter NAL unit 202 includes information to define the boundaries of the access unit corresponding to Pic ₉ from the access unit corresponding to Pic _7. Additionally, PPS NAL unit 204 includes a picture parameter set. Thus, the AU delimiter NAL unit 202 and the PPS NAL unit 204 are examples of non-VCL NAL units.

Another example of a non-VCL NAL unit in HEVC is the Supplemental Enhancement Information (SEI) NAL unit. The SEI mechanism, supported in both AVC and HEVC, enables encoders to include metadata in the bitstream that is not required for correctly decoding the sample values of output pictures, but which can be used for various other purposes, such as picture output timing, display, and loss detection and concealment. For example, an SEI NAL unit may include a picture timing message used by a video decoder when decoding the bitstream. The picture timing message may include information indicating when the video decoder should begin decoding VCL NAL units. An encoder may include any number of SEI NAL units in an access unit, and each SEI NAL unit may contain one or more SEI messages. The draft HEVC standard includes syntax and semantics for several SEI messages, but does not specify their handling because they do not affect the baseline decoding process. One reason for including SEI messages in the draft HEVC standard is to enable the supplementary data to be interpreted identically in different systems using HEVC. Specifications and systems using HEVC may require the encoder to generate certain SEI messages or may define specific handling of received SEI messages of a particular type. Table 1 lists the SEI messages specified in HEVC and briefly describes their purposes.

Table 1: Overview of SEI messages

Random access refers to decoding a video bitstream starting from a coded picture that is not the first coded picture in the bitstream. Random access to the bitstream is required in many video applications, such as broadcasting and streaming, for example, for users to switch between different channels, jump to a specific part of the video, or switch to a different bitstream to achieve stream adaptation (e.g., to achieve bit rate, frame rate, or spatial resolution scalability). For a video sequence, random access is achieved by having a coding structure that includes random access point (RAP) pictures or access units multiple times at regular intervals. Instantaneous decoder refresh (IDR) pictures, clean random access (CRA) pictures, and broken link access (BLA) pictures are RAP picture types defined in HEVC WD7. Each of the IDR pictures, CRA pictures, and BLA pictures contains only I slices. However, based on the defined reference constraints, each of the IDR pictures, CRA pictures, and BLA pictures is different.

IDR pictures are specified in AVC and defined according to HEVC WD7. Although IDR pictures can be used for random access, they are constrained because pictures following the IDR picture in decoding order cannot use pictures decoded before the IDR picture as references. In the example illustrated in Figures 1 and 2, as described above, pic ₆ in video sequence 100 may be an IDR picture. Due to the constraints associated with IDR pictures, bitstreams that rely on IDR pictures for random access may have significantly lower coding efficiency.

To improve coding efficiency, HEVC introduces the concept of CRA pictures. According to HEVC WD7, similar to IDR pictures, CRA pictures only include I slices. However, pictures that follow a CRA picture in decoding order but precede it in output order are allowed to use pictures decoded before the CRA picture as references. Pictures that follow a CRA picture in decoding order but precede it in output order are referred to as leading pictures associated with the CRA picture (or leading pictures of a CRA picture). If decoding starts from an IDR picture or CRA picture preceding the current CRA picture, the leading pictures of the CRA picture can be correctly decoded. However, when random access occurs from a CRA picture, the leading pictures of the CRA picture may not be correctly decoded. Referring to the examples illustrated in Figures 1 and 2, Pic ₉ may be a CRA picture and Pic ₈ may be a leading picture of Pic ₉ . If GOP ₂ is accessed at Pic ₆ , Pic ₈ may be correctly decodable, but if GOP ₂ is accessed as Pic ₉ , it may not be correctly decodable. This is due to the fact that if GOP ₂ is accessed as Pic ₉ , Pic ₇ may not be available. To prevent error propagation from reference pictures that may not be available depending on where decoding starts, according to HEVC WD7, all pictures following a CRA picture in both decoding order and output order are constrained to not use any pictures preceding the CRA picture in decoding order or output order (including leading pictures) as reference. In addition, leading pictures are typically discarded during random access decoding.

Bitstream splicing refers to the concatenation of two or more bitstreams or portions thereof. For example, a second bitstream may be appended to a first bitstream, possibly with some modifications made to one or both of the bitstreams to produce the spliced bitstream. The first coded picture in the second bitstream is also referred to as a splicing point. Thus, pictures after the splicing point in the spliced bitstream originate from the second bitstream, while pictures before the splicing point in the spliced bitstream originate from the first bitstream. Bitstream splicing is typically performed by a bitstream splicer. Bitstream splicers are often lightweight and less intelligent than video encoders. For example, a bitstream splicer may not be equipped with entropy decoding and encoding capabilities. Temporal scalability is an application that can use bitstream splicing. Temporal scalability may refer to decoding a video sequence at one or more frame rates. For example, it may be possible to decode a video sequence at 30 frames per second (fps) or 60 fps, depending on system capabilities. To achieve temporal scalability, a video sequence may include multiple temporal layers. Each temporal layer is a coded video sequence associated with a frame rate. The temporal layer with the highest frame rate may be referred to as the highest temporal layer. Multiple temporal layers may be spliced together to produce a video sequence with the highest frame rate, for example, splicing a coded video sequence with 30 fps with a coded video sequence that achieves 60 fps.

Bitstream switching can be used in an adaptive streaming environment. The bitstream switching operation at certain pictures in the switched bitstream is actually a bitstream splicing operation, where the splicing point is the bitstream switching point, that is, the first picture from the switched bitstream. It should be noted that bitstream switching is typically performed on two streams with the same coding structure. That is, the two streams have the same prediction structure and the same allocation of IDR pictures, CRA pictures, P pictures, B pictures, etc.

Following the introduction of CRA pictures, the concept of broken link access (BLA) pictures was further introduced in HEVC WD7, which is based on the concept of CRA pictures. BLA pictures typically originate from bitstream splicing at the location of a CRA picture, and in the spliced bitstream, the splice-point CRA picture is changed to a BLA picture. The most essential difference between BLA pictures and CRA pictures is as follows: for CRA pictures, if decoding starts from a RAP picture that precedes the CRA picture in decoding order, the associated leading pictures are correctly decodable, but when random access starts from the CRA picture, the associated leading pictures may not be correctly decodable. For BLA pictures, the associated leading pictures may not be correctly decodable in all cases, even when decoding starts from a RAP picture that precedes the BLA picture in decoding order. It should be noted that for a particular CRA picture or BLA picture, some of the associated leading pictures are correctly decodable, even when the CRA picture or BLA picture is the first picture in the bitstream. Such leading pictures are referred to as decodable leading pictures (DLPs), and other leading pictures are referred to as non-decodable leading pictures (NLPs). In HEVC WD9, NLPs are also referred to as flagged-for-discard (TFD) pictures. It should be noted that all leading pictures associated with IDR pictures are DLP pictures. Table 2 is a table included in HEVC WD7 that specifies the NAL units defined according to HEVC WD7. As illustrated in Table 2, the NAL unit types in HEVC WD7 include CRA pictures, BLA pictures, IDR pictures, VPS, SPS, PPS, and APS NAL unit types, which correspond to the pictures and parameter sets described above.

Table 2: HEVC WD7 NAL unit type code and NAL unit type category

To simplify NAL unit allocation, S. Kanumuri, G. Sullivan, “Refinement of Random Access Point Support,” 10th Conf, Stockholm, SE, July 2012, document JCTVC-J0344 (hereinafter “Kanumuri”), which is incorporated herein by reference in its entirety, proposed: (1) a constraint on IDR pictures such that there are no leading pictures associated with any IDR picture (i.e., no pictures that follow the IDR picture in decoding order and precede the IDR picture in output order), and (2) a modified allocation of NAL unit types 4 to 7 for RAP pictures, as defined in Table 2 above, as follows:

Table 3: Proposed NAL unit types according to Kanumuri

In Table 3, the SAP type refers to the stream access point type defined in ISO/IEC 14496-12, Version 4, "Information technology - Coding of audio-visual objects - Part 12: ISO base media file format" (w12640, 100th MPEG meeting, Geneva, April 2012), which is incorporated herein by reference in its entirety. As described above, IDR pictures and BLA/CRA pictures are functionally different for bitstream switching, but they are functionally the same for random access (e.g., seek applications). For bitstream switching at IDR pictures, the video coding system can know or assume that presentation can be continuous without failure (e.g., loss of non-presented pictures). This is because pictures following the IDR picture in decoding order cannot use pictures decoded before the IDR picture as references (i.e., the leading pictures associated with the IDR picture are DLPs). However, for bitstream switching at BLA pictures, some overlap decoding of one or more pictures from both streams may be required to ensure continuous presentation. Without additional capabilities, this overlap decoding may not currently be possible for HEVC WD7-compliant decoders. Without additional capabilities, there may not be any pictures to be presented at the associated TFD picture location because they may have been discarded. This can result in presentation that is not necessarily continuous. Furthermore, the problem is the same even if the BLA picture is a BLA picture without an associated TFD picture because the TFD pictures present in the original bitstream may have been discarded. Furthermore, if TFD pictures are not present in the original bitstream, then CRA pictures (later changed to BLA pictures due to bitstream splicing/switching, etc.) may have been coded as IDR pictures. Therefore, not marking IDR pictures with leading pictures as IDR pictures (i.e., not allowing IDR pictures to have leading pictures), as proposed by Kanumuri, makes IDR pictures less friendly to system bitstream switching.

From the perspective of streaming systems, such as Dynamic Streaming over HTTP (DASH), it is beneficial to be able to easily identify which picture is a RAP picture and, when decoding starts from a RAP picture, to identify the earliest presentation time (e.g., the earliest picture order count (POC) value). Therefore, the existing design of assigning NAL unit types to different RAP pictures, DLP pictures, and TFD pictures can be further improved to be more friendly to streaming systems. According to the existing design, for each RAP picture, when decoding starts from the RAP picture, the system must check whether there is an associated DLP picture to know whether the RAP picture's own presentation time is the earliest presentation time. In addition, the system must check and compare the presentation times of all DLP pictures to calculate the earliest presentation time value.

Video coding standards include video buffering model specifications. In AVC and HEVC, the buffering model is called the Hypothetical Reference Decoder (HRD), which includes buffering models for both the coded picture buffer (CPB) and the decoded picture buffer (DPB). According to HEVC WD7, the HRD is defined as a hypothetical decoder model that specifies constraints on the variability of conforming NAL unit streams or conforming bitstreams that can be produced by the encoding process. Therefore, in AVC and HEVC, bitstream conformance and decoder conformance are specified as part of the HRD specification. According to HEVC WD7, the CPB is a first-in, first-out buffer containing access units in decoding order, and the DPB is a buffer that holds decoded pictures for reference. According to the HRD, the CPB and DPB behaviors are mathematically specified. The HRD imposes constraints directly on timing, buffer size, and bitrate, and indirectly on bitstream characteristics and statistics. The complete set of HRD parameters includes five basic parameters: initial CPB removal delay, CPB size, bitrate, initial DPB output delay, and DPB size. According to HEVC WD7, HRD parameters can be included in the Video Usability Information (VUI) parameters, and the VUI parameters can be included in the SPS. It should be noted that although HRD is referred to as a decoder, HRD is generally required on the encoder side to ensure bitstream conformance, and is generally not required on the decoder side. HEVC WD7 specifies two types of bitstreams for HRD conformance, namely Type I and Type II. HEVC WD7 also specifies two types of decoder conformance, namely output timing decoder conformance and output order decoder conformance.

In the AVC and HEVC HRD models, decoding or CPB removal is based on access units and assumes that picture decoding is instantaneous. In real-world applications, the time required to decode a picture is unlikely to be zero. Therefore, in practical applications, if a conforming decoder strictly follows the decoding time signaled, for example, in a picture timing SEI message to start decoding an access unit, the earliest possible time that a particular decoded picture can be output is equal to the decoding time of that particular picture plus the time required to decode that particular picture.

Sub-picture based CPB behavior similar to that described in Ye-Kui Wang et al., "Sub-picture based CPB operation" (9th meeting: Geneva, CH, May 2012, JCTVC-I0588 (hereinafter referred to as "Wang")) has been included in HEVC WD7. Wang sub-picture based CPB allows CPB removal at the access unit (AU) level or the sub-picture level. Allowing AU-level or sub-picture level CPB removal helps achieve reduced codec latency in an interoperable manner. When access unit level CPB removal occurs, the access unit is removed from the CPB each time the removal operation occurs. When sub-picture level CPB removal occurs, the decoding unit (DU) containing one or more slices is removed from the CPB each time the removal operation occurs.

In addition to AU-level CPB removal timing information, sub-picture-level CPB removal timing information can also be signaled. When CPB removal timing information is presented for both AU-level and sub-picture-level removal, the decoder can choose to operate the CPB at the AU level or the sub-picture level. It should be noted that in order to achieve the current picture timing SEI message and the mechanism for simultaneously implementing both AU-level and DU-level HRD CPB removal to achieve sub-picture delay, the DU must be sent before all AUs are encoded, and the AU-level SEI message must not be sent before all AUs are encoded.

According to HEVC WD7, timing information may include information defining the temporal distance between the HRD output times of two consecutive pictures. HEVC WD7 defines the following timing information syntax elements:

time_scale is the number of time units that pass in one second. For example, a time coordinate system that uses a 27 MHz clock to measure time has a time_scale of 27,000,000. time_scale should be greater than 0.

num_units_in_tick is the number of time units that the clock operates at the frequency time_scale Hz corresponding to one increment of the clock tick counter (called a clock tick). num_units_in_tick should be greater than 0.

Therefore, based on the values of time_scale and num_units_in_tick, the so-called clock tick variable _tc can be derived as follows:

t _c =num_units_in_tick÷time_scale (1)

According to HEVC WD7, a clock tick variable can be used to constrain HRD output time. That is, in some cases, it may be required that the difference between the presentation times of two consecutive pictures in output order (i.e., the first and second pictures) is equal to the clock tick. HEVC WD7 includes a fixed_pic_rate_flag syntax element, which indicates whether the difference between the presentation times of two consecutive pictures in output order is equal to the clock tick. The fixed_pic_rate_flag syntax element can be included in a set of VUI parameters, which can be included in an SPS. In HEVC WD7, when the fixed_pic_rate_flag syntax element is equal to one, the temporal distance between the HRD output times of any two consecutive pictures in output order is constrained to be equal to the determined clock tick if either of the following conditions holds: (1) the second picture is in the same coded video sequence as the first picture; or (2) the second picture is in a different coded video sequence from the first picture, and in the coded video sequence containing the second picture, fixed_pic_rate_flag is equal to one, and the value of num_units_in_tick÷time_scale is the same for both coded video sequences. When the fixed_pic_rate_flag syntax element is equal to zero, these constraints do not apply to the temporal distance between the HRD output times of any two consecutive pictures in output order (i.e., the first picture and the second picture). It should be noted that when fixed_pic_rate_flag is not present, it is inferred to be equal to 0. It should be noted that according to HEVC WD7, when fixed_pic_rate_flag is equal to 1, stream adaptation based on temporal scalability will need to change the value of time_scale or num_units_in_tick in the case of dropping some of the highest temporal layers. It should be noted that HEVC WD7 provides the following semantics for fixed_pic_rate_flag:

When fixed_pic_rate_flag is equal to 1 for a coded video sequence containing picture n, the value calculated for At o,dpb(n) as specified in Equation C-13 shall be equal to t _c as specified in Equation C-1 (using the value of t _c for the coded video sequence containing picture n) when, for a subsequent picture n _n designated for use in Equation _C -13, one or more of the following conditions hold:

Picture _n is in the same coded video sequence as picture n.

Picture n _n is in different coded video sequences, and fixed_pic_rate_flag is equal to 1 in the coded video sequence containing picture n _n , and the value of num_units_in_tick÷time_scale is the same for both coded video sequences.

In HEVC WD7, Equation C-1 corresponds to Equation (1), and Equation C-13 is defined as follows:

Δt _o,dpb (n)=t _o,dpb (n _n )-t _o,dpb (n) (2)

In view of the timing and random access characteristics associated with HEVC WD7 mentioned above, this disclosure describes techniques that can be used to reduce latency in video applications, such as conversational applications, and provide improvements in random access to coded video sequences. In one example, this disclosure describes techniques for assigning NAL unit types. In another example, this disclosure describes sub-picture level or decoding unit level HRD behavior. In another example, this disclosure describes techniques for reference parameter set IDs. In yet another example, this disclosure describes techniques for providing improved semantics for the fixed_pic_rate_flag syntax element. It should be noted that any and all combinations of these and other techniques described herein can be incorporated into video encoding and decoding systems.

FIG3 is a block diagram illustrating an example video encoding and decoding system 10 that may utilize the techniques described herein. In detail, the video encoding and decoding system may utilize the techniques described herein related to: (1) assigning NAL unit types, (2) sub-picture level or decoding unit level HRD behavior, (3) reference parameter set IDs, (4) improved semantics of fixed_pic_rate_flag, or any and all combinations of such techniques. Video encoding and decoding system 10 is an example of a video system that may be used for any of the following video applications: local playback, streaming, broadcast, multicast, and/or interactive applications. Source device 12 and destination device 14 are examples of decoding devices, where source device 12 generates encoded video data for transmission to destination device 14. In some examples, source device 12 and destination device 14 may operate in a substantially symmetrical manner such that each of source device 12 and destination device 14 includes video encoding and decoding components. Thus, system 10 may be configured to support one-way or two-way video transmission between source device 12 and destination device 14 .

Although the techniques described herein are described in conjunction with source device 12 and destination device 14, the techniques may be performed by any digital video encoding and/or decoding device. The techniques of this disclosure may also be performed by a video preprocessor. Additionally, although the techniques of this disclosure are generally described as being performed by a video encoding device and a video decoding device, they may also be performed by a video encoder/decoder (often referred to as a "CODEC"). Thus, each of the video encoder 20 and video decoder 30 in FIG. 3 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in the respective device. Additionally, a device including the video encoder 20 and/or video decoder 30 may comprise an integrated circuit, a microprocessor, and/or a wireless communication device, such as a cellular telephone. Although not shown in FIG. 3 , in some aspects, the video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams. Where applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP).

3 , system 10 includes a source device 12 that provides encoded video data to be decoded at a later time by a destination device 14. In detail, source device 12 provides the encoded video data to destination device 14 via a computer-readable medium 16. Destination device 14 may receive the encoded video data to be decoded via computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets (e.g., so-called “smart” phones, so-called “smart” pads), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

Computer-readable medium 16 may comprise any type of medium or device capable of moving encoded video data from source device 12 to destination device 14. Computer-readable medium 16 may include transitory media, such as wireless broadcasts or wired network transmissions, or storage media (i.e., non-transitory storage media) such as a hard drive, flash drive, compact disc, digital video disc, Blu-ray disc, or other computer-readable media. In some examples, a network server (not shown) may receive the encoded video data from source device 12 and provide the encoded video data to destination device 14, for example, via a network transmission. Similarly, a computing device at a media production facility (e.g., a disc pressing facility) may receive the encoded video data from source device 12 and produce a disc containing the encoded video data.

In one example, computer-readable medium 16 may comprise a communication medium to enable source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be used to facilitate communication from source device 12 to destination device 14.

The storage device may include any of a variety of distributed or locally accessible data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by source device 12. Destination device 14 can access the stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing and transmitting the encoded video data to destination device 14. Example file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. Destination device 14 can access the encoded video data via any standard data connection, including an Internet connection. This data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.) suitable for accessing the encoded video data stored on the file server, or a combination of both. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

The techniques of this disclosure are not necessarily limited to wireless applications or settings. They may be applied to video coding supporting any of a variety of multimedia applications, such as wireless television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), encoding of digital video onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

In the example of FIG3 , source device 12 includes a video source 18, a coding structure unit 19, a video encoder 20, an encapsulation unit 21, and an output interface 22. Destination device 14 includes an input interface 28, a decapsulation unit 29, a video decoder 30, and a display device 32. In other examples, source device 12 and destination device 14 may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18 (e.g., an external camera). Similarly, destination device 14 may interface with an external display device rather than including an integrated display device. The components of source device 12 and destination device 14 may each be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques described herein are implemented partially in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques.

Video source 18 of source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, and/or a video feed interface for receiving video from a video content provider. As another alternative, video source 18 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, as mentioned above, the techniques described in this disclosure may be generally applicable to video coding and may be applied to wireless and/or wired applications. In each case, captured video, pre-captured video, or computer-generated video may be received by video encoder 20. Output interface 22 may be configured to output encoded video data (e.g., a coded video sequence) onto computer-readable medium 16. In some examples, the coded video sequence may be output from output interface 22 to a storage device. Input interface 28 of destination device 14 receives the encoded video data from computer-readable medium 16. Display device 32 displays the decoded video data to a user and may comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

Coding structure unit 19, video encoder 20, encapsulation unit 21, decapsulation unit 29, and video decoder 30 may operate according to a video coding standard such as the upcoming HEVC described above and may generally conform to the HEVC Test Model (HM). Alternatively, video encoder 20 and video decoder 30 may operate according to other proprietary or industry standards, such as the ITU-T H.264 standard (alternatively known as MPEG-4 Part 10, Advanced Video Coding (AVC)), or extensions of such standards. Coding structure unit 19, video encoder 20, encapsulation unit 21, decapsulation unit 29, and video decoder 30 may also operate according to modified versions of the video coding standards, where the modified versions of the video coding standards are modified to include any and all combinations of the techniques described herein.

Video encoder 20 may divide a video frame or picture into a series of equally sized video blocks, such as CUs described in HEVC WD7. A CU includes a coding node and prediction units (PUs) and transform units (TUs) associated with the coding node. The size of a CU corresponds to the size of the coding node and must be square in shape. The size of a CU may range from 8x8 pixels up to a treeblock size of 64x64 pixels or larger. Each CU may contain one or more PUs and one or more TUs. Syntax data associated with a CU may describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may differ between cases where the CU is skip or direct mode encoded, intra-prediction mode encoded, or inter-prediction mode encoded. The shape of a PU may be non-square. Syntax data associated with a CU may also describe, for example, the partitioning of the CU into one or more TUs according to a quadtree. The shape of a TU may be square or non-square (e.g., rectangular).

The HEVC standard allows for transforms based on TUs, which can be different for different CUs. The size of a TU is typically set based on the size of the PU within a given CU defined for the partitioned LCU, but this may not always be the case. A TU is typically the same size as a PU, or smaller than a PU. In some examples, a quadtree structure called a "residual quadtree" (RQT) can be used to subdivide the residual samples corresponding to a CU into smaller units. The leaf nodes of the RQT can be referred to as transform units (TUs). Pixel difference values associated with a TU can be transformed to produce quantizable transform coefficients.

A leaf-CU may include one or more prediction units (PUs). Generally, a PU represents a spatial region corresponding to all or a portion of the corresponding CU and may include data used to retrieve reference samples for the PU. In addition, a PU includes data related to prediction. For example, when the PU is intra-mode encoded, the data for the PU may be included in a residual quadtree (RQT), which may include data describing the intra-prediction mode of the TU corresponding to the PU. As another example, when the PU is inter-mode encoded, the PU may include data defining one or more motion vectors for the PU. The data defining the motion vector of the PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., quarter-pixel precision or eighth-pixel precision), the reference picture to which the motion vector points, and/or the reference picture list (e.g., List 0, List 1, or List C) for the motion vector.

A leaf-CU with one or more PUs may also include one or more transform units (TUs). Transform units may be specified using an RQT (also known as a TU quadtree structure), as discussed above. For example, a split flag may indicate whether a leaf-CU is split into four transform units. Each transform unit may then be further split into additional sub-TUs. When a TU is not further split, it may be referred to as a leaf-TU. Generally, for intra coding, all leaf-TUs belonging to a leaf-CU share the same intra prediction mode. That is, generally the same intra prediction mode is applied to calculate the predicted values for all TUs of a leaf-CU. For intra coding, the video encoder may use the intra prediction mode to calculate the residual value of each leaf-TU as the difference between the portion of the CU corresponding to the TU and the original block. A TU is not necessarily limited to the size of a PU. Thus, a TU may be larger or smaller than a PU. For intra coding, a PU may be co-located with a corresponding leaf-TU of the same CU. In some examples, the maximum size of a leaf-TU may correspond to the size of the corresponding leaf-CU.

In addition, the TUs of a leaf-CU may also be associated with corresponding quadtree data structures referred to as residual quadtrees (RQTs). That is, a leaf-CU may include a quadtree that indicates how the leaf-CU is partitioned into TUs. The root node of a TU quadtree generally corresponds to a leaf-CU, while the root node of a CU quadtree generally corresponds to a treeblock (or LCU). Unsplit TUs of an RQT are referred to as leaf-TUs. In general, unless otherwise noted, this disclosure uses the terms CU and TU to refer to leaf-CUs and leaf-TUs, respectively. This disclosure uses the term "block" to refer to any of a CU, PU, or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., macroblocks and subblocks thereof in H.264/AVC).

As an example, the HM supports prediction with various PU sizes. Assuming a particular CU is 2Nx2N, the HM supports intra prediction with PU sizes of 2Nx2N or NxN, and inter prediction with symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, or NxN. The HM also supports asymmetric partitioning for inter prediction with PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N. In asymmetric partitioning, one direction of the CU is not partitioned, while the other direction is partitioned into 25% and 75%. The portion of the CU corresponding to the 25% partition is indicated by an "n" followed by an indication of "up," "down," "left," or "right." Thus, for example, "2NxnU" refers to a 2Nx2N CU partitioned horizontally with a 2Nx0.5N PU at the top and a 2Nx1.5N PU at the bottom.

In this disclosure, "NxN" and "N by N" may be used interchangeably to refer to the pixel dimensions of a video block in terms of vertical and horizontal dimensions, e.g., 16x16 pixels or 16 by 16 pixels. Generally, a 16x16 block will have 16 pixels in the vertical direction (y=16) and 16 pixels in the horizontal direction (x=16). Similarly, an NxN block generally has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. The pixels in a block may be arranged in rows and columns. Furthermore, a block need not necessarily have the same number of pixels in the horizontal direction as in the vertical direction. For example, a block may comprise NxM pixels, where M is not necessarily equal to N.

After intra-predictive coding or inter-predictive coding using the PUs of a CU, video encoder 20 may calculate residual data for the TUs of the CU. A PU may include syntax data that describes a method or mode of generating predictive pixel data in the spatial domain (also referred to as the pixel domain), and a TU may include coefficients in the transform domain after applying a transform, such as a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform, to the residual video data. The residual data may correspond to pixel differences between pixels of an unencoded picture and prediction values corresponding to the PU. Video encoder 20 may form TUs including the residual data for the CU, and then transform the TUs to generate transform coefficients for the CU.

After applying any transforms to generate transform coefficients, video encoder 20 may perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the coefficients, thereby providing further compression. The quantization process may reduce the bit depth associated with some or all of the coefficients. For example, an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

After quantization, the video encoder may scan the transform coefficients, generating a one-dimensional vector from the two-dimensional matrix comprising the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) coefficients at the front of the array and lower energy (and therefore higher frequency) coefficients at the back of the array. In some examples, video encoder 20 may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector that can be entropy encoded. In other examples, video encoder 20 may perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 20 may entropy encode the one-dimensional vector, for example, according to context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method. Video encoder 20 may also entropy encode syntax elements associated with the encoded video data for use by video decoder 30 in decoding the video data.

To perform CABAC, video encoder 20 may assign context within a context model to a symbol to be transmitted. The context may relate to, for example, whether neighboring values of the symbol are non-zero. To perform CAVLC, video encoder 20 may select a variable length code for the symbol to be transmitted. Codewords in VLC may be constructed so that relatively shorter codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, using VLC may achieve bit savings (compared to, for example, using codewords of equal length for each symbol to be transmitted). Probability determinations may be made based on the context assigned to the symbol.

As described above, a video sequence may be coded according to a determined video coding structure, wherein the coding structure defines the allocation of picture types (e.g., RAP pictures and non-RAP pictures) used to encode the video sequence. For example, a video sequence may be coded with RAP pictures included at predetermined intervals to facilitate random access to the video sequence. This coding structure may be used in broadcast applications. Additionally, a video sequence may be coded according to a coding structure that minimizes latency for low-latency applications. Coding structure unit 19 may be configured to determine a coding structure to be used by video encoder 20 to encode a video sequence received from video source 18. In one example, coding structure unit 19 may store a predefined coding structure corresponding to a respective video application. Coding structure unit 19 may be configured to output information indicating a specific coding structure to each of video encoder 20 and encapsulation unit 21. Video encoder 20 receives the video sequence from video source 18 and the coding structure information from coding structure unit 19 and generates encoded video data. Encapsulation unit 21 receives encoded video data and information indicating a specific coding structure from video encoder 20 and generates a coded video sequence including storage units. Decapsulation unit 29 may be configured to receive the coded video sequence and parse access units and NAL units. Video decoder 30 may be configured to receive NAL units and reconstruct video data based on information contained in the received NAL units.

It should be noted that coding structure unit 19 and/or video encoder 20 may be configured to generate syntax elements for inclusion in parameter sets. In some examples, coding structure unit 19 may be configured to generate syntax elements for inclusion in a higher-level parameter set, such as an SPS, and video encoder 20 may be configured to perform video encoding based on the syntax elements received from the coding structure and output the entropy-encoded syntax elements as part of the encoded video data.

According to the techniques of this disclosure, the assignment of NAL unit types may be performed in a manner that allows a device, such as destination device 14, to easily identify RAP pictures and associated timing information. In one example, an IDR picture that does not have an associated leading picture has a different NAL unit type than an IDR picture that may have an associated leading picture. For example, an IDR picture that does not have an associated leading picture has a NAL unit type of M, while an IDR picture that may have an associated leading picture has a NAL unit type of N, where M is not equal to N, as illustrated in Table 4. Note that in the example illustrated in Table 4, the leading picture associated with the IDR picture may be a DLP picture. In one example, the NAL unit types illustrated in Table 4 may be incorporated into the HEVC WD7 NAL unit type codes and NAL unit type categories illustrated in Table 2. For example, the inverted NAL unit type values in Table 2 may be used for NAL unit types M and N in Table 4.

Table 4: Different IDR NAL unit types

In another example, a CRA picture that does not have an associated leading picture has a different NAL unit type than a CRA picture that may have an associated leading picture. Furthermore, a CRA picture that does not have an associated TFD picture has a different NAL unit type than a CRA picture that may have an associated TFD picture. Thus, three different NAL unit types may be used for different types of CRA pictures, as illustrated in Table 5. In one example, the NAL unit types illustrated in Table 5 may be incorporated into the HEVC WD7 NAL unit type codes and NAL unit type categories illustrated in Table 2. For example, the inverted NAL unit type values in Table 1 may be used for NAL unit types X, Y, and Z in Table 5.

Table 5: Different CRA NAL unit types

In another example, a BLA picture that does not have an associated leading picture may have a different NAL unit type than a BLA picture that may have an associated leading picture. Furthermore, a BLA picture that does not have an associated TFD picture may have a different NAL unit type than a BLA picture that may have an associated TFD picture. Thus, three different NAL unit types may be used for different types of BLAs, as illustrated in Table 6. In one example, the NAL unit types illustrated in Table 6 may be incorporated into the HEVC WD7 NAL unit type codes and NAL unit type categories illustrated in Table 2. For example, the inverted NAL unit type values in Table 2 may be used for NAL unit types A, B, and C in Table 6.

Table 6: Different BLA NAL unit types

Any and all combinations of NAL unit types described with respect to Tables 4 through 6 may be used for the allocation of NAL unit types. In one example, all NAL unit types described with respect to Tables 4 through 6 may be used for the allocation of NAL unit types. Table 7 illustrates an example where all NAL types described in Tables 4 through 6 are used for the allocation of NAL unit types. As illustrated in Table 7, the NAL unit types include the CRA picture, BLA picture, and IDR picture NAL unit types described with respect to Tables 4 through 6, as well as the VPS, SPS, PPS, and APS NAL unit types described above. Table 7 may be contrasted with Table 2 above because the allocation of NAL unit types provided in Table 7 includes multiple NAL unit types for IDR pictures, CRA pictures, and BLA pictures, whereas the allocation of NAL unit types provided in Table 1 includes a single NAL unit type for each of the IDR pictures, CRA pictures, and BLA pictures.

Table 7: NAL unit type code and NAL unit type category

Encapsulation unit 21 may be configured to receive encoded video data and information indicating a specific coding structure from video encoder 20, and generate a coded video sequence including access units based on the allocation of NAL unit types described in any and all combinations of the NAL unit allocations described in Tables 2 through 7. Additionally, decapsulation unit 29 may be configured to receive the coded video sequence and parse access units and NAL units, where the NAL units are allocated based on any and all combinations of the NAL unit allocations described in Tables 2 through 7.

As described above, according to HEVC WD7, in order to implement the current picture timing SEI message and simultaneously implement both AU-level and DU-level HRD CPB removal to achieve sub-picture delay, the DU must be sent before all AUs are encoded, and the AU-level SEI message cannot be sent before all AUs are encoded. According to the techniques of this disclosure, the encapsulation unit 21 and the decapsulation unit 29 can be configured to modify the sub-picture level or decoding unit level HRD behavior compared to HEVC WD7.

For example, encapsulation unit 21 may be configured so that an AU-level SEI message is sent after all AUs are encoded. This AU-level SEI message may be included in a SEI NAL unit having a distinct NAL unit type. One difference between this SEI NAL unit and the existing definition of a SEI NAL unit (e.g., as defined in HEVC WD7) is that this distinct SEI NAL unit type may be allowed to follow the last VCL NAL unit in the same AU in decoding order and may be constrained not to precede the first VCL NAL unit in the same AU in decoding order. Conventional SEI NAL units and SEI messages may be referred to as prefix SEI NAL units and prefix SEI messages, respectively, while the distinct SEI NAL units and SEI messages described herein may be referred to as suffix SEI NAL units and suffix SEI messages, respectively.

In addition to being configured to generate a coded video sequence based on any and all combinations of the NAL unit assignments described in Tables 2 through 7, encapsulation unit 21 may also be configured to generate a coded video sequence that includes prefix SEI NAL units and suffix SEI NAL units. Similarly, decapsulation unit 29 may be configured to receive a coded video sequence and parse access units and NAL units, where the NAL units include prefix SEI NAL unit types and suffix SEI NAL unit types. That is, decapsulation unit 29 may be configured to extract suffix SEI NAL units from access units. Table 8 illustrates an example of all NAL types described in Tables 4 through 6 being used for the assignment of NAL unit types, as well as prefix SEI NAL units and suffix SEI NAL units.

Table 8: NAL unit type code and NAL unit type category

As described above, in addition to SEI NAL units, non-VCL NAL unit types also include VPS, SPS, PPS, and APS NAL units. According to the parameter set type defined in HEVC WD7, each SPS references a VPS ID, each PPS references an SPS ID, and each slice header references a PPS ID and possibly an APS ID. Video encoder 20 and/or coding structure unit 19 may be configured to generate a parameter set according to the parameter set defined in HEVC WD7. Additionally, video encoder 20 and/or coding structure unit 19 may be configured to generate a parameter set in which the VPS ID and SPS ID may optionally be signaled in the slice header (e.g., where the VPS ID precedes the SPS ID). In one example where the VPS ID and SPS ID are signaled in the slice header, no VPS ID will be located in the SPS and no SPS ID will be located in the PPS. Additionally, in one example, the VPS ID and SPS ID may be present in the slice header of each RAP picture, and each picture may be associated with a recovery point SEI message. Additionally, in other examples, the VPS ID and SPS ID may be present in slice headers of other pictures.

FIG4 is a block diagram illustrating an example encapsulation unit that may implement the techniques described in this disclosure. In the example illustrated in FIG4 , encapsulation unit 21 includes a VCL NAL unit constructor 402, a non-VCL NAL unit constructor 404, an access unit constructor 406, and a bitstream output interface 408. Encapsulation unit 21 receives encoded video data and high-level syntax and outputs an encoded video bitstream. The encoded video data may include residual video data and syntax data associated with a slice. The high-level syntax data may include, for example, syntax elements included in a parameter set, SEI messages, or other syntax elements defined by a video coding standard such as the upcoming HEVC standard. The encoded video bitstream may include one or more coded video sequences and may generally conform to a video coding standard such as the upcoming HEVC standard. As described above, a VCL NAL unit includes a slice of video data. VCL NAL unit constructor 402 may be configured to receive a slice of encoded video data and generate a VCL NAL unit based on the type of picture comprising the slice. The VCL NAL unit constructor 402 may be configured to generate VCL NAL units according to any and all combinations of NAL allocations described above with respect to Tables 2 through 8. The VCL NAL unit constructor 402 may be configured to include a header in the VCL NAL unit, wherein the header identifies the type of the VCL NAL unit.

For example, the VCL NAL unit constructor 402 may be configured to receive a slice of video data contained in an IDR picture and (1) if the IDR picture does not have associated leading pictures, encapsulate the slice of video data in a NAL unit having a type indicating that the IDR picture does not have leading pictures, or (2) if the IDR picture has associated leading pictures, encapsulate the slice of video data in a NAL unit having a type indicating that the IDR picture has leading pictures. The VCL NAL unit constructor 402 may be configured to receive a slice of video data contained in a CRA picture and (1) if the CRA picture does not have associated leading pictures, encapsulate the slice of video data in a NAL unit having a type indicating that the CRA picture does not have leading pictures, or (2) if the CRA picture has associated leading pictures, encapsulate the slice of video data in a NAL unit having a type indicating that the CRA picture has leading pictures. Additionally, if the leading pictures associated with the CRA picture are TFD pictures, the VCL NAL unit constructor 402 may be configured to encapsulate the slice of video data in a NAL unit having a type indicating that the leading pictures associated with the CRA picture are TFD.

Additionally, if the leading picture associated with the CRA picture is not a TFD picture, the VCL NAL unit constructor 402 may be configured to encapsulate the slice of video data in a NAL unit having a type indicating that the leading picture associated with the CRA picture is not a TFD picture. Additionally, the VCL NAL unit constructor 402 may be configured to receive a slice of video data contained in a BLA picture and (1) if the BLA picture does not have an associated leading picture, encapsulate the slice of video data in a NAL unit having a type indicating that the BLA picture does not have a leading picture, or (2) if the BLA picture has an associated leading picture, encapsulate the slice of video data in a NAL unit having a type indicating that the BLA picture has a leading picture. Additionally, if the leading picture associated with the BLA picture is a TFD picture, the VCL NAL unit constructor 402 may be configured to encapsulate the slice of video data in a NAL unit having a type indicating that the leading picture associated with the BLA picture is a TFD picture. Additionally, if the leading picture associated with the BLA picture is not a TFD picture, the VCL NAL unit constructor 402 may be configured to encapsulate the slice of video data in a NAL unit having a type indicating that the leading picture associated with the BLA picture is not a TFD picture.

FIG5 is a flow chart illustrating an example of generating a VCL NAL unit according to the techniques of this disclosure. Although the example of generating a VCL NAL unit illustrated in FIG5 is described as being performed by VCL NAL unit constructor 402, any combination of source device 12, video encoder 20, encapsulation unit 21, and combinations of components thereof may perform the example of generating a VCL NAL unit illustrated in FIG5. As illustrated in FIG5, VCL NAL unit constructor 402 receives a slice of video data (502). The slice of video data may be encoded into encoded video data according to any of the encoding techniques described herein. The slice of video data may be contained in one of the picture types described herein. VCL NAL unit constructor 402 determines whether the slice of video data is contained in an IDR picture or a CRA picture (504).

If the slice of video data is included in an IDR picture ("IDR" branch of 504), VCL NAL unit constructor 402 determines whether the IDR picture has associated leading pictures (506). If the IDR picture does not have associated leading pictures ("No" branch of 506), VCL NAL unit constructor 402 generates a VCL NAL unit indicating that the IDR picture does not have associated leading pictures (508). If the IDR picture has associated leading pictures ("Yes" branch of 506), VCL NAL unit constructor 402 generates a VCL NAL unit indicating that the IDR picture has associated leading pictures (510).

If the slice of video data is contained in a CRA picture, VCL NAL unit constructor 402 determines whether the CRA picture has associated leading pictures (512). If the CRA picture does not have associated leading pictures (the "No" branch of 512), VCL NAL unit constructor 402 generates a VCL NAL unit indicating that the CRA picture does not have associated leading pictures (514). If the CRA picture has associated leading pictures (the "Yes" branch of 512), VCL NAL unit constructor 402 determines whether the associated leading pictures are TFD pictures (516).

If the associated leading picture of the CRA picture is a TFD picture ("yes" branch of 516), the VCL NAL unit constructor 402 generates a VCL NAL unit indicating that the associated leading picture of the CRA is a TFD picture (518). If the associated leading picture of the BLA picture is not a TFD picture ("no" branch of 516), the VCL NAL unit constructor 402 generates a VCL NAL unit indicating that the associated leading picture is not a TFD picture (520).

VCL NAL unit constructor 402 may generate a NAL unit by encapsulating the slice data in a NAL unit and including a NAL unit type value in the NAL unit header. Each NAL unit type value may correspond to a respective NAL unit type. In one example, NAL unit type values may be defined according to Table 7. The generated NAL unit may be output by NAL unit constructor 402 to access unit constructor 406 for inclusion in an access unit (522).

In this way, encapsulation unit 21 represents an example of a device for generating a bitstream including video data, the device including a processor configured to: determine whether a random access point (RAP) picture is of a type that can have associated leading pictures, and whether the RAP picture comprises an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; encapsulate slices of the RAP picture in a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures; and generate a bitstream including the NAL unit.

Likewise, the method of FIG5 represents an example of a method of generating a bitstream comprising video data, the method comprising: determining whether a random access point (RAP) picture is of a type that can have associated leading pictures, and whether the RAP picture comprises an instantaneous decoder refresh (IDR) picture or a clean random access (CRA) picture; encapsulating a slice of the RAP picture in a network abstraction layer (NAL) unit, wherein the NAL unit includes a NAL unit type value indicating whether the RAP picture is of a type that can have associated leading pictures; and generating a bitstream comprising the NAL unit.

4 , the non-VCL NAL unit constructor 404 may be configured to receive high-level syntax elements, such as syntax elements included in parameter sets and SEI messages (as described above), and generate non-VCL NAL units based on any and all combinations of NAL unit allocations described above with respect to Tables 2 through 8. The non-VCL NAL unit constructor 404 may be configured to generate non-VCL NAL units by encapsulating syntax data in NAL units and including a NAL unit type value in a NAL unit header. For example, the non-VCL NAL constructor may be configured to receive syntax elements included in a parameter set and include a NAL unit type value indicating the parameter set type in the NAL unit header.

In addition, the non-VCL NAL unit constructor 404 may be configured to receive AU-level SEI messages and generate SEI message NAL units. In one example, the non-VCL NAL unit constructor 404 may be configured to generate two types of SEI message NAL units, wherein a first type of SEI NAL unit indicates that this SEI NAL unit may follow the last VCL NAL unit in the access unit in decoding order, and a second type of SEI NAL unit indicates that this SEI NAL unit may not follow the last VCL NAL unit in the access unit in decoding order. In addition, the first type of SEI NAL unit may be constrained such that it is not allowed to precede the first VCL NAL unit in the same access unit in decoding order. The first type of NAL unit may be referred to as a suffix SEI NAL unit, and the second type of NAL unit may be referred to as a prefix SEI NAL unit. The non-VCL NAL unit constructor 404 outputs the non-VCL NAL unit to the access unit constructor 406.

The access unit constructor 406 can be configured to receive VCL NAL units and non-VCL NAL units and generate an access unit. The access unit constructor 406 can receive any type of NAL unit defined in Tables 2 to 8. The VCL access unit constructor 406 can be configured to generate an access unit based on any and all combinations of NAL unit types described herein. As described above, according to HEVC WD7, an access unit is a set of NAL units that are consecutive in decoding order and contain a decoded picture. Therefore, the access unit constructor 406 can be configured to receive multiple NAL units and arrange the multiple NAL units according to the decoding order. In addition, the access unit constructor 406 can be configured to arrange the suffix SEI NAL unit, as described above, so that it follows the last VCL NAL unit in the access unit and/or does not precede the first VCL NAL unit in the same access unit.

FIG6 is a flowchart illustrating an example of generating non-VCL NAL units according to the techniques of this disclosure. Although the example of generating non-VCL NAL units illustrated in FIG6 is described as being performed by non-VCL NAL unit constructor 404 and access unit constructor 406, any combination of source device 12, video encoder 20, encapsulation unit 21, and combinations of components thereof may perform the example of generating non-VCL NAL units illustrated in FIG6.

6, non-VCL NAL unit constructor 404 receives a SEI message (602). The SEI message may be any type of SEI message described above with respect to Table 1. Non-VCL NAL unit constructor 404 determines whether the SEI message is a prefix SEI message or a suffix SEI message (604).

If the SEI message is a suffix SEI message (the "Suffix" branch of 604), the non-VCL NAL unit constructor 404 generates a type value for the SEI NAL unit indicating that the SEI NAL unit is a suffix SEI message (606). If the SEI message is a prefix SEI message (the "Prefix" branch of 604), the non-VCL NAL unit constructor 404 generates a type value for the SEI NAL unit indicating that the SEI NAL unit is a regular SEI message (608).

The access unit constructor 406 receives the generated NAL units, which may include any combination of the types of NAL units described above with respect to Tables 2 through 8 (610). The access unit constructor 406 generates an access unit that includes the received NAL units (612). If the generated access unit includes a suffix SEI NAL unit, the NAL units of the access unit may be arranged such that the suffix SEI NAL unit does not precede the first VCL NAL unit in the same access unit, but may follow the last VCL NAL unit in the access unit in decoding order.

In this manner, encapsulation unit 21 represents an example of a processor configured to: determine whether a supplemental enhancement information (SEI) message is a prefix SEI message or a suffix SEI message, wherein the SEI message includes data related to the encoded video data; encapsulate the SEI message in a SEI NAL unit, wherein the SEI NAL unit includes a NAL unit type value that indicates whether the SEI NAL unit is a prefix SEI NAL unit or a suffix SEI NAL unit, and whether the SEI message is a prefix SEI message or a suffix SEI message; and generate a bitstream including at least the SEI NAL unit.

Likewise, the method of FIG. 6 represents an example of a method of generating a bitstream including video data, the method including: determining whether a supplemental enhancement information (SEI) message is a prefix SEI message or a suffix SEI message, wherein the SEI message includes data related to the encoded video data; encapsulating the SEI message in an SEI NAL unit, wherein the SEI NAL unit includes a NAL unit type value, the NAL unit type value indicating whether the SEI NAL unit is a prefix SEI NAL unit or a suffix SEI NAL unit, and whether the SEI message is a prefix SEI message or a suffix SEI message; and generating a bitstream including at least the SEI NAL unit.

4, the bitstream output interface 408 can be configured to receive access units and generate a coded video sequence. The bitstream output interface 408 can be further configured to output the coded video sequence as part of an encoded video bitstream, wherein the encoded video bitstream includes one or more coded video sequences based on any and all combinations of NAL unit types described herein. As described above, according to HEVC WD7, a coded video sequence is a set of access units that are consecutive in decoding order. Therefore, the bitstream output interface 408 can be configured to receive multiple access units and arrange them according to decoding order.

As described above, coding structure unit 19 and/or video encoder 20 may be configured to generate syntax elements for inclusion in a parameter set, including a fixed_pic_rate_flag syntax element that may be included in a set of VUI parameters, which may be included in an SPS, as provided in HEVC WD7. Additionally, coding structure unit 19 and/or video encoder 20 may be configured to generate a fixed_pic_rate_flag syntax element, wherein the fixed_pic_rate_flag syntax element includes semantics modified from those provided in HEVC WD7. For example, according to the current semantics of fixed_pic_rate_flag in HEVC WD7, when fixed_pic_rate_flag is equal to 1, the difference between the presentation times of two pictures consecutive in output order is required to be equal to the clock tick. However, when some of the highest temporal layers are dropped to achieve stream adaptation based on temporal scalability, this may require changes to the values of time_scale or num_units_in_tick.

In one example, the increment (i.e., the difference between the presentation times of two pictures consecutive in output order) need not be exactly equal to the clock tick, but rather the increment may need to be an integer number of multiples of the clock tick. In this way, coding structure unit 19 and/or video encoder 20 may be configured to generate a fixed_pic_rate_flag syntax element such that when fixed_pic_rate_flag is equal to 1, the difference between the presentation times of two pictures consecutive in output order is required to be equal to an integer multiple of the clock tick.

In another example, coding structure unit 19 and/or video encoder 20 may need to signal a fixed_pic_rate_flag for each temporal layer. Additionally, in this example, if fixed_pic_rate_flag for a particular temporal layer is equal to 1, i.e., the temporal layer representation has a constant picture rate, a value N may be signaled, and the increment of the temporal layer representation (between presentation times of two pictures consecutive in output order) may be equal to N times the clock tick.

In another example, coding structure unit 19 and/or video encoder 20 may be configured to optionally signal a fixed_pic_rate_flag for each temporal layer. In this example, if the fixed_pic_rate_flag for a particular layer is present and equal to 1, that is, the temporal layer representation has a constant picture rate, then a value N may be signaled, and the increment of the temporal layer representation (between presentation times of two pictures consecutive in output order) is equal to N times the clock tick. In the case where a fixed_pic_rate_flag is optionally signaled for each temporal layer, assuming that fixed_pic_rate_flag is signaled for the highest temporal layer and has a value equal to 1, then, for each specific temporal layer that does not have a signaled fixed_pic_rate_flag, the value of fixed_pic_rate_flag can be derived to be equal to the fixed_pic_rate_flag signaled for the highest temporal layer, and the value of N is derived to be equal to 2 ^{max_Tid - currTid} , where max_Tid is equal to the highest temporal_id value and currTid is equal to the temporal_id of the specific temporal layer.

FIG7 is a flowchart illustrating an example of signaling presentation time delta values. Although the example of signaling presentation time delta values illustrated in FIG7 is described as being performed by encapsulation unit 21, any combination of source device 12, video encoder 20, encapsulation unit 21, and combinations of components thereof may perform the example of signaling presentation time delta values illustrated in FIG7.

7 , encapsulation unit 21 generates a flag indicating whether the delta between the presentation time (e.g., POC value) of a first picture and the presentation time of a second picture is an integer of a clock tick value (702). In other words, encapsulation unit 21 may generate data indicating whether the difference (e.g., delta) between the presentation times of the first picture and the second picture is an integer multiple of a clock tick value. The flags described in FIG. 7 represent examples of such generated data. In some cases, encapsulation unit 21 may receive the value of the flag from coding structure unit 19 or video encoder 20. The flag may be any of the fixed_pic_rate_flag syntax elements described above.

In one example, encapsulation unit 21 determines whether the value of the flag may indicate that the increment is an integer multiple of the clock tick value (704). When the flag indicates that the increment is an integer value of the clock tick (the "yes" branch of 704), encapsulation unit 21 may generate an integer value N representing an integer multiple of the clock tick value (706). The integer value N may be used by a decoding device, such as destination device 14, to determine the increment value, where the increment is an integer multiple of the clock tick value. In one example, the integer value N may be a value from 0 to 2047 and may indicate a value that is one less than an integer multiple of the clock tick to which the increment is equal. Encapsulation unit 21 may then output the flag and the integer value N as part of the bitstream (708).

On the other hand, when encapsulation unit 21 determines that the flag indicates that the increment value is not an integer multiple of the clock tick (“No” branch of 704 ), encapsulation unit 21 may simply output the flag ( 710 ).

In this way, source device 12 represents an example of a processor configured to generate data indicating whether the difference between the presentation time of the first picture and the presentation time of the second picture is an integer multiple of a clock tick value, and when the data indicates that the difference is an integer multiple of the clock tick value, generate data representing the integer multiple.

Similarly, the method of Figure 7 represents an example of a method for generating a bitstream including video data, the method including: generating data indicating whether the difference between the presentation time of a first picture and the presentation time of a second picture is an integer multiple of a clock tick value, and when the data indicates that the difference is an integer multiple of the clock tick value, generating data representing the integer multiple.

As described above, encapsulation unit 21 receives encoded video data. FIG8 is a block diagram illustrating an example of a video encoder 20 that can generate encoded video data. As shown in FIG8, video encoder 20 receives video data and high-level syntax data. Video encoder 20 typically operates on video blocks within individual video slices to encode the video data. Video blocks may correspond to coding nodes within a CU. Video blocks may have fixed or varying sizes, and the sizes may differ according to a specified coding standard. Video encoder 20 may further generate syntax data, such as block-based syntax data, frame-based syntax data, and GOP-based syntax data, in a frame header, a block header, a slice header, or a GOP header. GOP syntax data may describe the number of frames in the respective GOP, and frame syntax data may indicate the encoding/prediction mode used to encode the corresponding frame.

In the example of FIG8 , video encoder 20 includes a mode select unit 40, a reference picture memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy encoding unit 56. Mode select unit 40, in turn, includes a motion compensation unit 44, a motion estimation unit 42, an intra-prediction unit 46, and a partitioning unit 48. For video block reconstruction, video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and summer 62. A deblocking filter (not shown in FIG8 ) may also be included to filter block boundaries to remove blocking artifacts from the reconstructed video. If necessary, the deblocking filter will typically filter the output of summer 62. In addition to the deblocking filter, additional filters (in-loop or post-loop) may also be used. For simplicity, such filters are not shown, but if necessary, such filters may filter the output of summer 50 (as in-loop filters).

During the encoding process, video encoder 20 receives a video frame or slice to be coded. The frame or slice may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive coding on the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-prediction unit 46 may alternatively perform intra-predictive coding on the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction. Video encoder 20 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

Furthermore, partition unit 48 may partition blocks of video data into sub-blocks based on an evaluation of a previous partitioning scheme in a previous coding pass. For example, partition unit 48 may initially partition a frame or slice into LCUs and, based on rate-distortion analysis (e.g., rate-distortion optimization), partition each of the LCUs into sub-CUs. Mode select unit 40 may further generate a quadtree data structure indicating the partitioning of the LCU into sub-CUs. A leaf node CU of the quadtree may include one or more PUs and one or more TUs.

Mode select unit 40 may select one of the coding modes (intra or inter) (e.g., based on the error result) and provide the resulting intra- or inter-coded block to summer 50 to generate residual block data and to summer 62 to reconstruct the encoded block for use as a reference frame. Mode select unit 40 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy encoding unit 56.

Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are described separately for conceptual purposes. Motion estimation, performed by motion estimation unit 42, is the process of generating motion vectors that estimate the motion of video blocks. For example, a motion vector may indicate the displacement of a PU of a video block within a current video frame or picture relative to a predictive block within a reference frame (or other coded unit) relative to the current block being coded within the current frame (or other coded unit). A predictive block is a block that is found to closely match the block to be coded in terms of pixel difference, which may be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. In some examples, video encoder 20 may calculate values for sub-integer pixel positions of a reference picture stored in reference picture memory 64. For example, video encoder 20 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform motion searches relative to full-pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in reference picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

Motion compensation performed by motion compensation unit 44 may involve extracting or generating a predictive block based on the motion vector determined by motion estimation unit 42. Furthermore, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists. Summer 50 forms a residual video block by subtracting pixel values of the predictive block from pixel values of the current video block being coded, thereby forming pixel difference values, as discussed below. Generally, motion estimation unit 42 performs motion estimation with respect to the luma component, and motion compensation unit 44 uses the motion vector calculated based on the luma component for both chroma and luma components. Mode select unit 40 may also generate syntax elements associated with video blocks and video slices for use by video decoder 30 in decoding video blocks of video slices.

Intra-prediction unit 46 may intra-predict a current block, as an alternative to the inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. In particular, intra-prediction unit 46 may determine an intra-prediction mode to use to encode the current block. In some examples, intra-prediction unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction unit 46 (or, in some examples, mode select unit 40) may select an appropriate intra-prediction mode to use from the tested modes.

For example, intra-prediction unit 46 may calculate rate-distortion values using a rate-distortion analysis for the various intra-prediction modes tested and select the intra-prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the encoded block and the original, unencoded block that was encoded to produce the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. Intra-prediction unit 46 may calculate ratios from the distortion and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

After selecting the intra-prediction mode for the block, intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Video encoder 20 may include definitions of the encoding contexts for the various blocks and indications of the most probable intra-prediction mode, an intra-prediction mode index table, and a modified intra-prediction mode index table in the transmitted bitstream configuration data, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables).

Video encoder 20 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being coded. Summer 50 represents the component(s) that perform this subtraction operation. Transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, producing a video block comprising residual transform coefficient values. Transform processing unit 52 may perform other transforms conceptually similar to the DCT. Wavelet transforms, integer transforms, sub-band transforms, or other types of transforms may also be used. In either case, transform processing unit 52 applies a transform to the residual block, producing a block of residual transform coefficients. The transform may convert the residual information from the pixel value domain to a transform domain (e.g., the frequency domain). Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

After quantization, entropy encoding unit 56 entropy codes the quantized transform coefficients. For example, entropy encoding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. In the case of context-based entropy coding, the context may be based on neighboring blocks. After entropy coding by entropy encoding unit 56, the encoded bitstream may be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the frames of reference picture memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reconstructed video block for storage in reference picture memory 64. The reconstructed video block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block to inter-code a block in a subsequent video frame.

As described above, decapsulation unit 29 may be configured to receive a coded video sequence and parse access units and NAL units, wherein the NAL units are allocated based on any and all combinations of the NAL unit allocations described in Tables 2 through 7. Additionally, decapsulation unit 29 and video decoder 30 may reconstruct video data based on the NAL unit type allocations. In one example, decapsulation unit 29 may be configured to receive a NAL unit, wherein the NAL unit includes a NAL type value, and based on the NAL type value, determine whether the NAL unit encapsulates an encoded slice of video data included in a RAP picture associated with a leading picture, and video decoder 30 may be configured to reconstruct the video data based on whether the NAL unit encapsulates an encoded slice of video data included in a RAP picture associated with a leading picture. In another example, decapsulation unit 29 may be configured to receive a NAL unit, wherein the NAL unit includes a NAL type value, and based on the NAL type value, determine whether the NAL unit encapsulates an AU-level SEI message, and video decoder 30 may be configured to reconstruct the video data based on whether the NAL unit encapsulates the AU-level SEI message. In some cases, reconstructing the video data may include generating a spliced bitstream, as described above, and video decoder 30 may determine the presentation time of pictures in the spliced video stream based on the NAL unit type determination.

As also described above, a source device, such as source device 12, may be configured to signal a delta between the presentation time of a first picture and the presentation time of a second picture, wherein the signaling uses any of the syntax elements, which may be any of the fixed_pic_rate_flag syntax elements described above. Accordingly, destination device 14, decapsulation unit 29, and video decoder 30 may be configured to determine the presentation times of the first picture and the second picture and present the pictures accordingly.

FIG9 is a flowchart illustrating an example method of determining a presentation time delta value. Although the example of signaling a presentation time delta value illustrated in FIG9 is described as being performed by decapsulation unit 29, any combination of destination device 14, video decoder 30, decapsulation unit 29, and combinations of components thereof may perform the example of determining a presentation time delta value illustrated in FIG9. As illustrated in FIG9, decapsulation unit 29 obtains a first picture (902). The first picture may be an encoded picture corresponding to an access unit. Decapsulation unit 29 obtains a second picture (904). The second picture may be an encoded picture corresponding to the access unit. The second picture may be included in the same temporal layer as the first picture. Additionally, the first and second pictures may be included in the highest temporal layer of the video data.

Decapsulation unit 29 may then obtain an integer value N (906). This assumes that decapsulation unit 29 has previously obtained data, such as the value of a flag, indicating the integer value N. The integer value N may be included in a set of VUI parameters, which may be included in the SPS. Decapsulation unit 29 determines a clock tick value (908). Decapsulation unit 29 may determine the clock tick value based on the time_scale and num_units_in_tick syntax elements according to equation (1) described above.

Decapsulation unit 29 may then determine a delta between the presentation time of the first picture and the presentation time of the second picture 910. The delta may be equal to an integer multiple of the clock tick value based on an integer value N. For example, the delta may be equal to (N+1)*clock tick.

Decapsulation unit 29 and video decoder 30 may then present the first picture and the second picture according to the determined delta (912). In one example, decapsulation unit 29 may signal the delta value to video decoder 30, and video decoder 30 may perform a decoding process based on the delta value. In this manner, destination device 14 represents an example of a device including a processor configured to determine a difference between a presentation time of a first picture and a presentation time of a second picture, wherein the difference is equal to an integer value multiplied by a clock tick value, and present the first picture and the second picture according to the determined difference.

Similarly, the method of Figure 9 represents an example of a method that includes the following operations: determining a difference between the presentation time of a first picture and the presentation time of a second picture, wherein the difference is equal to an integer value multiplied by a clock tick value, and presenting the first picture and the second picture according to the determined difference.

FIG10 is a block diagram illustrating an example of a video decoder 30 that may implement techniques for: (1) receiving data including NAL unit types, (2) processing received sub-picture level or decoding unit level HRD behavior, (3) processing data including references to parameter set IDs, (4) processing received data including improved semantics for fixed_pic_rate_flag, or any and all combinations thereof. In the example of FIG10 , the video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intra-prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference picture memory 82, and a summer 80. In some examples, the video decoder 30 may perform a decoding pass that is substantially reciprocal to the encoding pass described with respect to the video encoder 20 ( FIG2 ). Motion compensation unit 72 may generate prediction data based on the motion vector received from entropy decoding unit 70 , while intra-prediction unit 74 may generate prediction data based on the intra-prediction mode indicator received from entropy decoding unit 70 .

During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video slice and associated syntax elements from video encoder 20. Entropy decoding unit 70 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive syntax elements at the video slice level and/or the video block level.

When the video slice is coded as an intra-coded (I) slice, intra-prediction unit 74 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When the video frame is coded as an inter-coded (i.e., B, P, or GPB) slice, motion compensation unit 72 generates predictive blocks for the video block of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 70. The predictive blocks may be generated from one of the reference pictures within one of the reference picture lists. Video decoder 30 may construct reference frame lists: List 0 and List 1, using default construction techniques based on the reference pictures stored in reference picture memory 82. Motion compensation unit 72 determines prediction information for the video block of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate predictive blocks for the current video block being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine a prediction mode (e.g., intra-prediction or inter-prediction) used to code a video block of a video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more of the slice's reference picture lists, motion vectors for each inter-coded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information used to decode video blocks in the current video slice.

Motion compensation unit 72 may also perform interpolation based on interpolation filters. Motion compensation unit 72 may calculate interpolated values for sub-integer pixels of a reference block using interpolation filters as used by video encoder 20 during encoding of the video block. In this case, motion compensation unit 72 may determine the interpolation filters used by video encoder 20 from the received syntax elements and use the interpolation filters to produce the predictive blocks.

Inverse quantization unit 76 inverse quantizes, i.e., dequantizes, the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 70. The inverse quantization process may include using a quantization parameter _QPY for each video block in a video slice calculated by video decoder 30 to determine a degree of quantization and, likewise, a degree of inverse quantization that should be applied.

Inverse transform unit 78 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to produce residual blocks in the pixel domain.

After motion compensation unit 72 generates a predictive block for the current video block based on the motion vector and other syntax elements, video decoder 30 forms a decoded video block by summing the residual block from inverse transform unit 78 with the corresponding predictive block generated by motion compensation unit 72. Summer 80 represents the component(s) that perform this summing operation. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. Other loop filters (in the coding loop or after the coding loop) may also be used to smooth pixel transitions or otherwise improve video quality. The decoded video blocks in a given frame or picture are then stored in reference picture memory 82, which stores reference pictures used for subsequent motion compensation. Reference picture memory 82 also stores the decoded video for later presentation on a display device (e.g., display device 32 of FIG. 3 ).

It should be appreciated that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary to practice the techniques). Furthermore, in some examples, actions or events may be performed simultaneously rather than sequentially, for example, via multithreading, interrupt processing, or multiple processors.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media (which corresponds to tangible media such as data storage media) or communication media, which includes, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) a non-transitory, tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and/or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Moreover, any connection may be properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (e.g., infrared, radio, and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (e.g., infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather are directed to non-transitory, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or collections of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Specifically, as described above, the various units can be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in conjunction with appropriate software and/or firmware.

Various examples have been described. These and other examples are within the scope of the following claims.

Claims (12)

1. A method for decoding video data, the method comprising: A first value of the first NAL unit type of the first supplementary enhancement information SEI network abstraction layer NAL unit of the bit stream and a second value of the second NAL unit type of the second SEI NAL unit of the bit stream are determined, wherein the first NAL unit type value of the first SEI NAL unit indicates that the first SEI NAL unit includes a prefix SEI NAL unit containing a prefix SEI message, and wherein the second NAL unit type value of the second SEI NAL unit indicates that the second SEI NAL unit includes a suffix SEI NAL unit containing a suffix SEI message; The video data of the bitstream is decoded based on the data of the first SEI NAL unit, which is the prefix SEI NAL unit, and based on the data of the second SEI NAL unit, which is the suffix SEI NAL unit. Decoding the video data further includes: The prefix SEI NAL unit is extracted from a first access unit including the prefix SEI NAL unit, and subsequently, one or more first video decoding layer VCL NAL units are extracted, the first or more VCL NAL units including the last VCL NAL unit, and in the first access unit, the first or more VCL NAL units are following the prefix SEI NAL unit in the decoding order; and Extract the second one or more VCL NAL units including the last VCL NAL unit from the second access unit and then extract the suffix SEI NAL unit, wherein the suffix SEI NAL unit is in the decoding order following the second one or more VCL NAL units including the last VCL NAL unit in the second access unit. 2. An apparatus for decoding video data, the apparatus comprising: A memory configured to store video data of a bitstream; and Processor, configured to: A first value of the first NAL unit type of the first supplementary enhancement information SEI network abstraction layer NAL unit of the bit stream and a second value of the second NAL unit type of the second SEI NAL unit of the bit stream are determined, wherein the first NAL unit type value of the first SEI NAL unit indicates that the first SEI NAL unit includes a prefix SEI NAL unit containing a prefix SEI message, and wherein the second NAL unit type value of the second SEI NAL unit indicates that the second SEI NAL unit includes a suffix SEI NAL unit containing a suffix SEI message; The video data of the bitstream is decoded based on the data of the first SEI NAL unit, which is the prefix SEI NAL unit, and based on the data of the second SEI NAL unit, which is the suffix SEI NAL unit. To decode the video data, the processor is further configured to: The prefix SEI NAL unit is extracted from a first access unit including the prefix SEI NAL unit, and subsequently, one or more first video decoding layer VCL NAL units are extracted, the first or more VCL NAL units including the last VCL NAL unit, and in the first access unit, the first or more VCL NAL units are following the prefix SEI NAL unit in the decoding order; and Extract the second one or more VCL NAL units including the last VCL NAL unit from the second access unit and then extract the suffix SEI NAL unit, wherein the suffix SEI NAL unit is in the decoding order following the second one or more VCL NAL units including the last VCL NAL unit in the second access unit. 3. The apparatus of claim 2, wherein the apparatus comprises at least one of the following: One or more integrated circuits; One or more microprocessors; One or more digital signal processors (DSPs); One or more Field Programmable Gate Arrays (FPGAs); Desktop computers; Laptop computers; Tablet computer; Telephone; TV set; camera; Display device; Digital media player; Video game console; Video game devices; Video streaming device; or Wireless communication device. 4. The apparatus of claim 2, further comprising a display device configured to display at least a portion of the video data. 5. An apparatus for decoding video data, the apparatus comprising: A means for determining a first value of a first NAL unit type of a first SEI NAL unit of a first supplemental enhancement information (SEI) network abstraction layer (NAL) unit of a bit stream and a second value of a second NAL unit type of a second SEI NAL unit of the bit stream, wherein the first NAL unit type value of the first SEI NAL unit indicates that the first SEI NAL unit includes a prefix SEI NAL unit containing a prefix SEI message, and wherein the second NAL unit type value of the second SEI NAL unit indicates that the second SEI NAL unit includes a suffix SEI NAL unit containing a suffix SEI message; A means for decoding video data of the bitstream based on data of the first SEI NAL unit, wherein the first SEI NAL unit is the prefix SEI NAL unit, and based on data of the second SEI NAL unit, wherein the means for decoding the video data further comprises: A means for retrieving the prefix SEI NAL unit from a first access unit including the prefix SEI NAL unit and subsequently retrieving one or more first video decoding layer VCL NAL units, the first or more VCL NAL units including a last VCL NAL unit, and in the first access unit, the first or more VCL NAL units are sequentially following the prefix SEI NAL unit in decoding order; and A means for retrieving a second one or more VCL NAL units including the last VCL NAL unit from a second access unit and subsequently retrieving the suffix SEI NAL unit, wherein the suffix SEI NAL unit is in the decoding order following the second one or more VCL NAL units including the last VCL NAL unit in the second access unit. 6. The apparatus of claim 5, wherein the apparatus comprises at least one of the following: One or more integrated circuits; One or more microprocessors; One or more digital signal processors (DSPs); One or more Field Programmable Gate Arrays (FPGAs); Desktop computers; Laptop computers; Tablet computer; Telephone; TV set; camera; Display device; Digital media player; Video game console; Video game devices; Video streaming device; or Wireless communication device. 7. The apparatus of claim 5, further comprising a display means for displaying at least a portion of the video data. 8. A computer-readable storage medium having instructions stored thereon, said instructions, when executed, causing a processor to perform the following operations: A first value of the first NAL unit type of the first supplementary enhancement information SEI network abstraction layer NAL unit of the bit stream and a second value of the second NAL unit type of the second SEI NAL unit of the bit stream are determined, wherein the first NAL unit type value of the first SEI NAL unit indicates that the first SEI NAL unit includes a prefix SEI NAL unit containing a prefix SEI message, and wherein the second NAL unit type value of the second SEI NAL unit indicates that the second SEI NAL unit includes a suffix SEI NAL unit containing a suffix SEI message; The video data of the bitstream is decoded based on the data of the first SEI NAL unit, which is the prefix SEI NAL unit, and based on the data of the second SEI NAL unit, which is the suffix SEI NAL unit. The instructions causing the processor to decode the video data further include instructions causing the processor to perform the following operations: The prefix SEI NAL unit is extracted from a first access unit including the prefix SEI NAL unit, and subsequently, one or more first video decoding layer VCL NAL units are extracted, the first or more VCL NAL units including the last VCL NAL unit, and in the first access unit, the first or more VCL NAL units are following the prefix SEI NAL unit in the decoding order; and Extract the second one or more VCL NAL units including the last VCL NAL unit from the second access unit and then extract the suffix SEI NAL unit, wherein the suffix SEI NAL unit is in the decoding order following the second one or more VCL NAL units including the last VCL NAL unit in the second access unit. 9. A method for generating a bitstream containing video data, the method comprising: The encoding includes a first SEI message containing prefix supplemental enhancement information (SEI) and a second SEI message containing a suffix SEI, wherein each of the prefix SEI message and the suffix SEI message contains data related to the encoded video data. The prefix SEI message is encapsulated in the first SEI network abstraction layer NAL unit; Set the first NAL unit type value of the first SEI NAL unit to a value that indicates the first SEI NAL unit is a prefix SEI NAL unit; The first access unit is formed such that the first access unit includes the first SEI NAL unit, and the final video decoding layer VCL NAL unit is connected after the first SEI NAL unit in the decoding order. The suffix SEI message is encapsulated in a second SEI NAL unit; Set the second NAL cell type value of the second SEI NAL cell to a value that indicates that the second SEI NAL cell is a suffix SEI NAL cell; A second access unit is formed such that the second access unit includes one or more VCLNAL units containing the last VCL NAL unit, the suffix SEI NAL unit being sequentially appended to the last VCL NAL unit in the decoding order, such that the suffix SEI NAL unit is sequentially appended to all VCL NAL units in the second access unit in the decoding order; and A bit stream is generated that includes at least the first access unit and the second access unit. 10. An apparatus for generating a bitstream comprising video data, the apparatus comprising a processor configured to: The encoding includes a first SEI message containing prefix supplemental enhancement information (SEI) and a second SEI message containing suffix SEI information, wherein the prefix SEI information and the suffix SEI information each contain data related to the encoded video data. The prefix SEI message is encapsulated in the first SEI network abstraction layer NAL unit; Set the first NAL unit type value of the first SEI NAL unit to a value that indicates the first SEI NAL unit is a prefix SEI NAL unit; The first access unit is formed such that the first access unit includes the first SEI NAL unit, and the final video decoding layer VCL NAL unit is connected after the first SEI NAL unit in the decoding order. The suffix SEI message is encapsulated in a second SEI NAL unit; Set the second NAL cell type value of the second SEI NAL cell to a value indicating that the second SEI NAL cell is a suffix SEI NAL cell; A second access unit is formed such that the second access unit includes one or more VCLNAL units containing the last VCL NAL unit, the suffix SEI NAL unit being sequentially appended to the last VCL NAL unit in the decoding order, such that the suffix SEI NAL unit is sequentially appended to all VCL NAL units in the second access unit in the decoding order; and A bit stream is generated that includes at least the first access unit and the second access unit. 11. An apparatus for generating a bitstream comprising video data, the apparatus comprising: A means for encoding a first SEI message containing a prefix supplemental enhancement information (SEI) message and a second SEI message containing a suffix SEI message, wherein the prefix SEI message and the suffix SEI message each contain data related to the encoded video data; A means for encapsulating the prefixed SEI message in a first SEI network abstraction layer (NAL) unit; A means for setting the first NAL cell type value of the first SEI NAL cell to a value indicating that the first SEI NAL cell is a prefix SEI NAL cell; The means for forming a first access unit such that the first access unit includes the first SEI NAL unit, and the final video decoding layer VCL NAL unit is connected after the first SEI NAL unit in the decoding order. A means for encapsulating the suffix SEI message in a second SEI NAL unit; means for setting the second NAL cell type value of the second SEI NAL cell to indicate that the second SEI NAL cell is a suffix SEI NAL cell; Means for forming a second access unit such that the second access unit includes one or more VCL NAL units comprising a last VCL NAL unit, the suffix SEI NAL unit being sequentially appended to the last VCL NAL unit in decoding order, such that the suffix SEI NAL unit is sequentially appended to all VCL NAL units in the second access unit in decoding order; and A means for generating a bit stream that includes at least the first access unit and the second access unit. 12. A computer-readable storage medium having instructions stored thereon, said instructions, when executed, causing a processor to perform the following operations: The encoding includes a first SEI message containing a prefix supplemental enhancement information (SEI) message and a second SEI message containing a suffix SEI message, wherein the prefix SEI message and the suffix SEI message each contain data related to the encoded video data; The prefix SEI message is encapsulated in the first SEI network abstraction layer NAL unit; Set the first NAL unit type value of the first SEI NAL unit to a value that indicates the first SEI NAL unit is a prefix SEI NAL unit; The first access unit is formed such that the first access unit includes the first SEI NAL unit, and the final video decoding layer VCL NAL unit is connected after the first SEI NAL unit in the decoding order. The suffix SEI message is encapsulated in a second SEI NAL unit; Set the second NAL cell type value of the second SEI NAL cell to a value that indicates that the second SEI NAL cell is a suffix SEI NAL cell; A second access unit is formed such that the second access unit includes one or more VCLNAL units containing the last VCL NAL unit, the suffix SEI NAL unit being sequentially appended to the last VCL NAL unit in the decoding order, such that the suffix SEI NAL unit is sequentially appended to all VCL NAL units in the second access unit in the decoding order; and A bit stream is generated that includes at least the first access unit and the second access unit.