HK1234565B - Method and system for selectively breaking prediction in video coding - Google Patents

Description

Method and system for selectively destroying prediction in video coding

This application is a divisional application of the invention patent application with application number 201180062300.7, application date December 28, 2011, and invention name “Method and system for selectively destroying prediction in video coding”.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/427,569, filed on December 28, 2010, entitled “PICTURE SEGMENTATION USING GENERALIZED SLICES,” and U.S. Patent Application No. 13/336,475, filed on December 23, 2011, entitled “METHOD AND SYSTEM FOR SELECTIVELY BREAKING PREDICTION IN VIDEO CODING,” each of which is incorporated herein by reference in its entirety.

Technical Field

Embodiments of the present invention relate to video compression, and more particularly, to selective use of prediction and in-loop filter mechanisms at picture segment boundaries of video pictures.

Background Art

Digital video capabilities may be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, camcorders, digital recording devices, video game devices, video game consoles, cellular or satellite radiotelephones, etc. Digital video devices may implement video compression techniques, such as those described in standards like MPEG-2, MPEG-4, available from the International Standards Organization ("ISO"), PO Box 56, 1st Avenue Voie-Creuse, CH-1211 Geneva, Switzerland, or www.iso.org, or ITU-T H.264/MPEG-4 Part 10, Advanced Video Coding ("AVC"), available from the International Telecommunication Union ("ITU"), 20 Place des Palais, CH-1211 Geneva, Switzerland, or www.itu.int, each of which is incorporated herein by reference in its entirety, or according to other standard or non-standard specifications, to efficiently encode and/or decode digital video information. Other compression technologies may still be developed in the future or are currently under development. For example, a new video compression standard called HEVC/H.265 is under development by the JCT-VC committee. A working draft of HEVC/H.265 was proposed in "WD3: Working Draft 3 of High-Efficiency Video Coding, JCTVC-E603" by Wiegand et al., March 2011, which is hereinafter referred to as "WD3" and is incorporated herein by reference in its entirety.

A video encoder may receive unencoded video information for processing into any suitable format, which may be a digital format compliant with ITU-R BT601 (available from the International Telecommunication Union ("ITU") at 20 Place des Palais, Geneva, Switzerland, CH-1211, or www.itu.int, and incorporated herein by reference in its entirety), or some other digital format. The unencoded video may be spatially organized into pixel values arranged in one or more two-dimensional matrices and temporally organized into a series of unencoded images, each unencoded image comprising one or more of the aforementioned two-dimensional matrices of pixel values. Furthermore, each pixel may comprise a number of independent components for representing color in a digital format. One common format for unencoded video input to a video encoder has, for each group of four pixels, four luminance samples (which include information about the brightness/lightness or darkness of the pixel) and two chrominance samples (which include color information (e.g., YCrCb 4:2:0)).

One function of a video encoder is to translate (more generally "convert") an uncoded image into a bitstream, packet stream, NAL unit stream, or other suitable transmission format (hereinafter referred to as a "bitstream"), with the goal of, for example, reducing the amount of redundancy encoded into the bitstream to thereby increase the transmission rate, increasing the bitstream's recovery to mitigate bit errors or packet erasures that may occur during transmission (collectively referred to as error recovery), or other specialized goals. Embodiments of the present invention provide at least one of removal or reduction of redundancy, increased error resilience, and implementation of a video encoder and/or associated decoder in a parallel processing architecture.

One function of a video decoder is to receive as its input an encoded video in the form of a bitstream produced by a video encoder that conforms to the same video compression standard. The video encoder then translates (more commonly "converts") the received encoded bitstream into unencoded video information that can be displayed, stored, or otherwise processed.

Video encoders and video decoders can be implemented using hardware and/or software configurations (comprising a combination of both hardware and software). The implementation of any one or both of a video encoder and a video decoder can include the use of programmable hardware components such as (e.g., those found in a personal computer (PC)) a general-purpose central processing unit (CPU), an embedded processor, a graphics card processor, a digital signal processor (DSP), a field programmable gate array (FPGA), etc. In order to implement at least a portion of video encoding or decoding, instructions may be required, and one or more non-transient computer-readable media can be used to store and distribute those instructions. Computer-readable media selections include compact disc read-only memory (CD-ROM), digital video disc read-only memory (DVD-ROM), memory stick, embedded ROM, etc.

Video compression and decompression refer to specific operations performed in a video encoder and/or decoder. A video decoder may perform all or a subset of the inverse operations of the encoding operations. Unless otherwise specified, the video encoding techniques described herein are also intended to include the inverse operations of the described video encoding techniques (i.e., the related video decoding techniques).

Uncompressed, digitally represented video can be viewed as a stream of samples that can be processed in scan order by a video display. One type of boundary that commonly occurs in this sample stream is the boundary between pictures in the sample stream. Many video compression standards recognize these boundaries and typically split the coded bitstream at these boundaries, for example, by inserting a picture header or other metadata at the beginning of each uncoded picture. Other boundaries that may occur in a sample stream include slice boundaries and tile boundaries, which may occur within uncoded pictures, as described below.

Prediction in video coding can occur at several levels.

A level is hereinafter referred to as the "entropy coding level," and prediction at this level is referred to as "coding prediction." Within this level, decoding of an entropy coded symbol may require successful decoding of a previous entropy coded symbol. All or nearly all current compression standards break coding prediction at the picture and slice levels. That is, upon detection of a picture or slice header in the bitstream (or equivalent), the entropy coding-related states used in entropy coding are reset to an initialized state. An example of entropy coding prediction is the resetting of the CABAC state in ITU-T Rec. H.264.

Furthermore, there may be an encoding mechanism where the decoding mechanism does not fall within the common understanding of prediction associated with entropy coding as described above, but still involves reconstruction control information associated with the bitstream rather than pixel values. As an example, even some older standards (e.g., ITU-T Rec. H.261) allow motion vectors to be encoded relative to one or more previously encoded motion vectors. The detection of a group of blocks (GOB), slice, or picture header resets the prediction vector to (0, 0).

There are also prediction mechanisms that span multiple pictures. For example, motion compensation can use (possibly motion-compensated) pixel values from one or more reference pictures for prediction. This type of prediction is broken down by macroblock type (or equivalent). For example, intra macroblocks typically do not use prediction from a reference picture, whereas inter macroblocks may. In this sense, intra and inter slices are simply accumulations of macroblocks belonging to those different macroblock types.

There is also a level of prediction that includes predictions based on pixel values that have already been reconstructed during the reconstruction process of the image being coded. An example is an intra-frame prediction mechanism, such as that described in Annex I of ITU-T Rec. H.263 (similar mechanisms are also available in other video coding standards).

In addition to prediction mechanisms, several video coding standards specify filters for performing in-loop filtering. One example is the in-loop filter specified in Annex J of ITU-T Rec. H.263.

For some applications, it may be advantageous to segment the image being encoded into smaller data blocks, where the segmentation may occur before or during encoding. Two use cases that may benefit from image segmentation are described below.

The first use case involves parallel processing. In the past, standard definition video (e.g., 720x480 or 720x576 pixels) was the largest format in widespread commercial use. More recently, HD formats (up to 1920x1080 pixels) have emerged and are used in a variety of application spaces, as well as 4k (4096x2048 pixels), 8k (8192x4096 pixels), and larger formats. Despite the increase in affordable computing power in recent years, due to the very large image sizes associated with some of these newer or larger formats, it is generally advantageous to leverage the efficiency of parallel processing for encoding and decoding processes. Parallel encoding and decoding can occur, for example, at the instruction level (e.g., using SIMD), in a pipeline that can process multiple video encoding units simultaneously at different stages, or on a large-scale architecture where a collection of video encoding subunits is processed by independent computing engines as independent entities (e.g., a multi-core general-purpose processor). The last format for parallel processing may require image segmentation.

The second use case involves image segmentation to create a bitstream suitable for efficient transmission over a packet network. Codecs used to transmit coded video over IP or other packet network protocols may be limited by maximum transmission unit ("MTU") size constraints. With respect to coded slice size, it is sometimes advantageous for the resulting packets containing coded slices to be as close to the MTU size as possible without exceeding it, in order to maintain a high payload/packetization overhead ratio while avoiding network fragmentation (and the resulting higher probability of loss).

The MTU size varies widely from one network to another. The MTU size of many Internet connections may be set, for example, by the minimum MTU size of the network infrastructure typically used for the Internet, which typically corresponds to the limit in Ethernet and may be approximately 1500 bytes.

The number of bits in a coded picture depends on many factors, such as the size of the source picture, the desired quality, the complexity of the content in terms of prediction suitability, the coding efficiency of the video coding standard, and other factors. However, even at moderate settings of quality and content complexity, the average coded picture size easily exceeds the MTU size for sequences at HD resolution and higher. A video conferencing encoder might, for example, require approximately 2 Mbit/sec to encode a 720p60 video sequence. This results in an average coded picture size of approximately 33,000 bits or 4,215 bytes, which is considerably more than the Internet's MTU size of 1,500 bytes. At higher resolutions, the average picture size increases to values significantly above the Internet's MTU size. Assuming a similar compression ratio as in 720p60 above, 4096x2048 (4k) video at 60 fps (4kp60) might require more than 300,000 bits or 25 MTU-sized packets per coded video picture.

In many previous video coding standards (e.g., up to and including WD3), a picture segment (or at least one form of picture segment) was referred to as a "slice." In the following description, any kind of picture segmentation (e.g., based on video coding) that is used to disrupt at least one form of intra-picture prediction, in-loop filtering, or other coding mechanism may be generally referred to as a "slice." Similarly, structures such as the group of blocks ("GOB") in ITU-T Rec. H.261 or ITU Rec. H.263 (available from the ITU for H.264, see above), slices in H.264 or the MPEG family of standards may each constitute the term "slice" as used throughout this document. However, the segmentation units of RFC3984 or the data portions of H.264 do not constitute the term "slice" as used throughout this document because they do not disrupt intra-picture prediction, in-loop filtering, or another coding mechanism.

Referring to FIG. 1 , an example 100 of image segmentation using slices is shown. An image 101 is divided into two scan-order slices 102 and 103. The slice boundary is shown as a thick line 104. The first macroblock 105 of the second slice 103 has an address 11. For example, when a corresponding bitstream 106 for transmitting image 101 is generated using the H.264 standard, bitstream 106 may include one or more parameter sets 107 followed by slice headers 108 and 110 and slice data 109 and 111 for both slices 102 and 103. The one or more parameter sets 107 do not include information regarding slice boundaries. The slice header 110 of the second slice 103 is shown enlarged. For example, the size of an uncoded slice 103 is determined by a decoder based on a combination of at least two factors. First, slice header 110 includes the address of the first macroblock 105 of slice 103. Next, the end of the slice is determined, for example, by detecting a new slice header in the bitstream, or in the example described, by the end of the coded picture in the bitstream 112 (i.e., after macroblock 24). All macroblocks between the first macroblock and the end of the slice constitute the slice. It should be noted that scan order modifications (e.g., H.264's flexible macroblock ordering) can change the number of macroblocks in a slice by creating gaps.

One advantage of using slices over a media-unaware segmentation mechanism, such as those provided by IP at the routing layer, is that slices can be decoded at least to a certain degree independently by breaking a certain type of prediction at the boundaries between slices (as discussed in more detail below). Thus, the loss of one slice does not necessarily render other slices of the coded picture unusable or undecodable. Depending on the implementation of the segmentation mechanism, the loss of a segment may instead render many other segments unusable, since the term segmentation, as used throughout this document, does not break any form of prediction.

WD4 ("WD4: Working Draft 4 of High-Efficiency Video Coding" by B. Bross et al., available from http://wftp3.itu.int/av-arch/jctvc-sit/2011_07_f_Forino/) is a draft specification associated with a developing digital video coding standard, which may be referred to as High Efficiency Video Coding (HEVC) or H.265. In addition to slices, WD4 also includes an image segmentation mechanism called "tiles". According to WD4, a source image can be divided into rectangular units called tiles, so that each pixel of the source image is part of a tile (other constraints may also apply). Thus, a tile is a rectangular portion of an image. Tiles have tile boundaries determined by a coordinate system available in a high-level syntax structure, which is called a parameter set in WD4. Tiles are described in more detail below.

Each of the above inter-picture prediction mechanisms or coding mechanisms, with the exception of inter-picture prediction, may be violated by decoding of a picture header (or equivalent, e.g., decoding of a slice with a different frame number than the previous slice). Whether those prediction mechanisms are violated across slice boundaries or tile boundaries depends on the video compression standard and the slice type used.

In H.264, slices can be decoded independently of motion vector prediction, intra-frame prediction, CA-VLC and CABAC states, and other aspects of the H.264 standard. Only inter-picture prediction (including the input of pixel data beyond slice boundaries via motion compensation) is allowed. While this decoding independence improves error resilience, disabling prediction across slice boundaries reduces coding efficiency.

In H.263, video encoders have more flexibility in choosing which prediction mechanisms to override by using slices or GOBs with non-empty headers. For example, there is a bit included in the picture header, selectable when using Annex R, that signals to the decoder that no prediction occurs across slice/GOB (with non-empty header) boundaries. Certain prediction mechanisms (e.g., motion vector mechanisms) are violated across GOBs with non-empty headers and across slice boundaries, regardless of the state of Annex R. Other prediction mechanisms are controlled by Annex R. For example, if this bit is not set, motion vectors can point outside the spatial region co-located with a slice/GOB with non-empty header in the current reference picture, potentially "importing" sample values used for motion compensation from areas outside the geometric region of the slice/GOB in the reference picture into the current slice. Furthermore, unless Annex R is active, loop filtering can include sample values outside of the slice/GOB. Similarly, another picture header exists in the picture header that enables or disables intra prediction.

However, in most standards, the decision to break picture prediction is made at the granularity of at least one picture, and in some cases at the sequence granularity. In other words, using H.263 as an example, it is not possible to mix slices that have the deblocking filter enabled or disabled (respectively) in a given picture, nor is it possible to enable/disable intra prediction at the slice level.

As already described, image segmentation allows an image to be divided into spatial regions that are smaller than the entire image. While the most common applications for image segmentation appear to be matched to the MTU size and parallelized, image segmentation can also be used for many other purposes, including those that adapt the size and shape of the segments to the content. Coding of regions of interest is one of several examples. In this case, when different coding tools (including different prediction mechanisms) are applied, certain parts of the image may be more efficiently encoded than other parts (in the sense of spending a lower number of bits to encode while producing a similar visual experience). For example, some content may benefit from deblocking filtering and may not respond well to intra-frame prediction, while other content in the same image may be better encoded without deblocking filtering, but can benefit from intra-frame prediction. By enabling deblocking filtering and intra-frame prediction, third content can be best encoded. When the image is divided into tiles, all of this content can be located in the same image, which occurs, for example, in an interview situation or video conferencing.

One drawback of existing mechanisms for prediction breaking at segment boundaries is that enabling and/or disabling of prediction breaking is generally hard-coded into existing video coding standards, making it difficult or impossible to selectively break the prediction mechanism at segment boundaries, for example based on characteristics of the coded content.

Therefore, there is a need for an improved method and system for enabling or disabling prediction and in-loop filtering mechanisms individually or globally on a per-slice basis.Therefore, a solution that at least partially addresses the above or other drawbacks is desired.

Furthermore, there is a need to enable or disable prediction mechanisms and/or in-loop filtering mechanisms across headerless (or equivalent) picture segment boundaries (e.g., tile boundaries) on a per-picture (or group of pictures, sequence group, etc.) or global basis. Therefore, a solution that at least partially addresses the above and other drawbacks is desired.

Summary of the Invention

Embodiments of the present invention provide methods and systems for encoding and/or decoding a video image in which multiple prediction and in-loop filtering tools for image segments can be selectively enabled or disabled.

According to one aspect of the present invention, an encoder can indicate to one or more prediction tools that the tool can access information external to the image segment currently being processed as reference information for processing within that image segment. The encoder can provide this indication for a single prediction tool (e.g., entropy prediction, intra prediction, motion compensated prediction, motion vector prediction, hereinafter referred to as a prediction tool) and/or a single filtering tool (e.g., adaptive interpolation filtering, adaptive loop filtering, deblocking, filtering, sample adaptive offset, hereinafter referred to as a loop filtering tool), as well as other indications. Alternatively, the encoder can provide this indication for multiple predefined tools, or a predefined tool set that can include any of the above prediction and loop filtering tools, as well as other indications. This can be used to support parallelization of encoders and decoders and specific application cases, such as soft persist rendering (stitching coded images together in the compressed domain).

According to one aspect of the present invention, when using headerless picture segments (e.g., tiles), an encoder can instruct a prediction tool, a loop filtering tool, or a plurality of predefined groups of tools, regardless of whether the tool may use information across horizontal, vertical, or both horizontal and vertical tile boundaries as reference information.

As an example, in the specific case of H.264 or HEVC, the encoder can set the value of the "coding interruption indication" flag for prediction and in-loop filtering tools, such as: intra-frame prediction reference sample values outside slice/tile boundaries; vector reference sample values outside slice/tile boundaries (i.e., through motion compensation); use of CABAC state outside slice/tile boundaries; use of CA-VLC state outside slice/tile boundaries; use of PIPE or similar V2V entropy coding state outside slice/tile boundaries (HEVC only); and use of states and sample values outside slice/tile boundaries for in-loop filters (such as adaptive interpolation filter, adaptive loop filter, deblocking loop filter, or sample adaptive offset).

According to one aspect of the invention, use or other enabling of an encoding tool is indicated not in the form of a flag but through the operation of other data structures, e.g., in some cases a "coding interrupt indication" integer may combine multiple aforementioned flags or preferred permutations of those flags into a single symbol.

According to one aspect of the invention, the maximum length of a motion vector pointing outside a slice boundary may be encoded as a suitable entropy coded representation of an integer, thereby indicating not only that motion compensation is not used up to the distance allowed by the used level, but also the maximum value allowed, which may, for example, aid resource allocation in a decoder implementation.

According to one aspect of the present invention, at least one of the aforementioned encoding interruption indication flags or other data encoding interruption indication structures may be stored in a slice header, a picture header, a parameter set, or the like.

According to one aspect of the invention, a decoder may react to the presence of a flag or other data structure by breaking the indicated prediction tool across slice/tile boundaries rather than other boundaries that might be appropriate.

In one broad aspect, a method for decoding a coded video picture comprising a plurality of segments is provided. For at least one segment of the coded video picture that does not have an associated segment header, the method may include obtaining, from the coded video picture, at least one indication of at least one prediction or in-loop filtering operation to be applied to the coded video picture, and controlling the at least one prediction or in-loop filtering operation in response to the at least one indication. In some cases, the coded video picture may include at least two segments that do not have associated segment headers.

In another broad aspect, a method for encoding a video picture comprising a plurality of segments is provided. The method includes: obtaining, for at least one segment of the video picture that does not have an associated segment header, at least one indication of at least one prediction or in-loop filtering operation to be applied to the at least one segment that does not have an associated segment header; and controlling, during encoding of the video picture, the at least one prediction or in-loop filtering operation in response to the at least one indication. In some cases, the video picture may include at least two segments that do not have an associated segment header.

In yet another broad aspect, a non-transitory computer-readable medium having computer-executable instructions stored thereon is provided for programming one or more processors to perform a method for encoding a video image comprising a plurality of segments. The method may include, for at least one segment of the encoded video image that does not have an associated segment header, obtaining from the encoded video image at least one indication of at least one prediction or in-loop filtering operation to be applied to the encoded video image, and controlling the at least one prediction or in-loop filtering operation in response to the at least one indication. In some cases, the encoded video image may include at least two segments that do not have an associated segment header.

In yet another broad aspect, a non-transitory computer-readable medium having computer-executable instructions stored thereon is provided for programming one or more processors to perform a method for encoding a video image comprising a plurality of segments. The method may include, for at least one segment of the video image that does not have an associated segment header, obtaining at least one indication of at least one prediction or in-loop filtering operation to be applied to the at least one segment that does not have an associated segment header, and controlling the at least one prediction or in-loop filtering operation during encoding of the video image in response to the at least one indication. In some cases, the video image may include at least two segments that do not have an associated segment header.

In yet another broad aspect, a data processing system is provided that includes at least one of a processor and accelerator hardware configured to perform a method for encoding a video image comprising a plurality of segments. The method may include, for at least one segment of the encoded video image that does not have an associated segment header, obtaining from the encoded video image at least one indication of at least one prediction or in-loop filtering operation to be applied to the encoded video image, and controlling the at least one prediction or in-loop filtering operation in response to the at least one indication. In some cases, the encoded video image may include at least two segments that do not have an associated segment header.

In yet another broad aspect, a data processing system is provided that includes at least one of a processor and accelerator hardware configured to perform a method for encoding a video image comprising a plurality of segments. The method may include, for at least one segment of the video image that does not have an associated segment header, obtaining at least one indication of at least one prediction or in-loop filtering operation to be applied to the at least one segment that does not have an associated segment header, and controlling the at least one prediction or in-loop filtering operation during encoding of the video image in response to the at least one indication. In some cases, the video image may include at least two segments that do not have an associated segment header.

In some embodiments, according to any of the above schemes, at least one prediction or in-loop filtering operation may include at least one of entropy prediction, intra-frame prediction, motion vector prediction, motion compensated prediction, adaptive loop filtering, adaptive interpolation filtering, deblocking filtering or sample adaptive offset.

In some embodiments, according to any of the above aspects, at least one aspect of the plurality of indications may be obtained from at least one combined indication.

In some embodiments, according to any of the above schemes, the at least one indication may be encoded as a vector indicating a maximum length of a motion vector.

In some embodiments, at least one indication may be encoded into a parameter set according to any of the above aspects.

According to a further aspect of the present invention, a device (e.g., a data processing system), a method for changing the device, and an article of manufacture (e.g., a non-transitory computer-readable medium or product having recorded and/or stored thereon program instructions for executing any method described herein) are provided.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features and advantages of embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

1 is a diagram showing an exemplary picture with scan order slices and a bitstream representing an encoded picture according to one embodiment of the present invention;

FIG2 is a diagram showing tiles and slices according to one embodiment of the present invention;

FIG3 is a block diagram illustrating an encoded bit stream according to an embodiment of the present invention;

FIG4 is a block diagram illustrating an encoded bit stream according to an embodiment of the present invention;

FIG5 is a block diagram illustrating an encoded bit stream according to an embodiment of the present invention;

FIG6 is a flow chart illustrating the operation of an exemplary decoder according to one embodiment of the present invention;

7 is a flowchart illustrating the operation of an exemplary decoder when decoding slices according to one embodiment of the present invention;

8 is a block diagram illustrating an implementation based on a data processing system (eg, a personal computer ("PC")) according to an embodiment of the present invention;

It should be noted that throughout the drawings, like features are identified by like reference numerals.

DETAILED DESCRIPTION

In the following, details are set forth to provide an understanding of the present invention. In some instances, specific software, circuits, structures, and methods are not described or shown in detail to avoid obscuring the present invention. The term "data processing system" is used herein to refer to any machine for processing data, including the computer systems, wireless devices, and network configurations described herein. Embodiments of the present invention can be implemented in any computer programming language, as long as the operating system of the data processing system provides facilities that can support the requirements of these embodiments. Embodiments of the present invention can also be implemented in hardware or a combination of hardware and software.

At least some embodiments of the present invention are directed to selectively disrupting prediction mechanisms and/or selectively disabling in-loop filtering mechanisms in conjunction with image segmentation in video compression.

Terms such as "segment" or "picture segment" are used below to refer to any one or more macroblocks or equivalent (e.g., treeblocks in WD4) that are smaller than an entire picture, and at a "segment" or "picture segment" boundary, at least one form of prediction is broken and/or at least one form of in-loop filtering is disabled. H.264-like slices and WD4-like tiles (where tile_boundary_independence_idc is equal to 1) as described below are non-limiting examples of segments.

FIG2 illustrates an example 200 in which an image 201 is divided into two tiles 202 and 203 by a vertically dividing tile boundary 204 depicted as a solid bold line. Tiles can coexist with slices within image 201. For example, image 201 can be divided into two slices by slice boundary 205, just as it is divided into tiles 202 and 203 by tile boundary 204. Tiles (tile_boundary_independence_idc equal to 1) as described in WD4 can, in one or more schemes, generalize to another type of image segment called a column, which is further described in U.S. patent application Ser. No. 13/336,675, filed Dec. 23, 2011, entitled “METHOD AND SYSTEM FOR SELECTIVELY BREAKING PREDICTION IN VIDEO CODING,” which is incorporated herein by reference in its entirety.

The transmitted bitstream 206 corresponding to the image 201 may include, for example, a parameter set 207 or other high-level syntax elements including tile boundary information 208 for identifying tile boundaries 204. However, portions of the bitstream other than the parameter set 207 do not include any information about tile boundaries. The decoder can identify the tile to which a coded macroblock (also called a largest coding unit (LCU) or a treeblock (in WD4)) belongs by associating internal state information of the macroblock currently being processed with information about the tile size known from the parameter set 207.

One difference between tiles and other rectangular picture partitioning mechanisms, such as rectangular slices (a submode of Annex K of ITU-T Rec. H.263), is that tiles (unlike rectangular slices) do not require a header. In the case where a header is not included, the physical size of the tile may instead be defined in a parameter set. In some cases (tile_boundary_independence_idc is equal to 1), tile boundaries according to WD4 interrupt all intra-picture prediction mechanisms, but allow referencing a sample in a reference picture that is at a different location than the sample associated with the tile for which motion compensation is being performed. Furthermore, tile boundaries do not interrupt in-loop filtering including the deblocking filter, sample adaptive offset filter, and adaptive loop filter.

However, it may also be convenient or desirable for an encoder or decoder to use tiles to break up different sets of prediction mechanisms. For example, at very high resolutions, it may be advantageous to divide the video image into tiles that are subject to a requirement that motion vectors are not allowed to point outside tile boundaries and/or the encoder and decoder treat tile boundaries as picture boundaries (similar to Annex R of H.263) or similar, thus, for example, avoiding not only motion compensation across tile boundaries but also in-loop filtering.

In other cases, it may be convenient or desirable for the encoder or decoder to be able to handle full-resolution video encoding in addition to entropy coding of symbols. The encoder or decoder may, for example, include dedicated signal processing hardware based on sample processing, but a general-purpose multi-core CPU may be used for entropy encoding and/or decoding in cases where a single core is unable to handle the load (given that CABAC entropy coding in HEVC is computationally demanding, in particular). Therefore, to support this use case, entropy coding may need to be broken at tile boundaries, whereas other intra- or inter-image prediction mechanisms may be able to cross slice and/or tile boundaries.

In other cases, it may be convenient or desirable for the encoder or decoder to allow limited inter-processor coordination across tile boundaries. In this case, reference to pixel values is not possible, but the communication channel between the processors can obtain reference to control information (such as information necessary for motion vector prediction). In this case, intra prediction is not possible, but motion vector prediction can be used.

There may be a coding tool that is not directly related to prediction but that can still be advantageously interrupted across slice or tile boundaries. For example, co-pending U.S. patent application No. 13/286,828, filed on November 1, 2011, entitled "ADAPTIVE INTERPOLATION IN DIGITAL VIDEO CODING," which is incorporated herein by reference in its entirety, discloses an adaptive interpolation filter whose properties and coefficients can be selected by an encoder. It is advantageous to limit the use of samples outside the slice for interpolation filtering. Similarly, WD4 includes an adaptive interpolation filter where control of the adaptive interpolation filter is at least partially derived from specific pixels. It is advantageous to limit the acquisition to pixels only within the slice or tile boundaries. It is also advantageous to limit the filtering itself (as opposed to the acquisition of filter control information) to pixels within the slice or tile boundaries. WD4 also includes other loop filters, such as an adaptive loop filter (related to filtering all samples), a deblocking filter (related to filtering block boundaries), and a filtering mechanism called adaptive sample offset. All of these filters can share the same performance as AIF. For example, in the case of an adaptive loop filter as specified in WD4, it is advantageous to (possibly independently) disable the access to the information for obtaining filter taps across tile boundaries and disable filtering across tile boundaries themselves.

Segment boundaries may be defined by picture (or higher) level syntax structures (e.g., parameter sets when WD4 tiles are used), by segment header information (e.g., rectangular slices of H.263 Annex K), by a combination of the arrangement of segment headers in the bitstream and encoder/decoder state (e.g., H.264 slices when flexible macroblock ordering is not used), or by a combination of two or more of the foregoing mechanisms (i.e., FMO defines slice groups, and within a slice group, picture segments are defined by a combination of the arrangement of slice headers in the bitstream (identifying the first macroblock of a slice by its address) and the implicit dominance of macroblock addresses within the slice group, until the end of the slice is detected by bitstream parsing or other means).

We now describe the first mechanism that allows the prediction tool to select tile boundaries, followed by the mechanism that allows the prediction tool to select slice boundaries. Finally, we describe the interaction of the two mechanisms.

Referring to example 300 in FIG3 , FIG3 shows a coded bitstream 301 including a parameter set 302 and two coded slices 304, 305. The coded slices 304, 305 may belong to one or two coded pictures. In WD4, picture boundaries may be identified by a slice header with an LCU address of 0. The parameter set 302 may include tile control information 303 (e.g., tile boundaries), and in this example it is assumed that the information in the parameter set 302 relates to two coded slices (i.e., the parameter set reference in the slice header includes the same index). In many WD4 and H.264 based systems, parameter sets relate to tens, hundreds, or more slices.

According to one embodiment, parameter set 302 may include multiple prediction tool indication flags (PTIs). For example, when PTI is set (i.e., enabled), prediction across segment boundaries may be allowed regardless of which encoding or decoding tool is associated with the flag; otherwise, when PTI is not set (i.e., disabled), such prediction may be prohibited. Flags may be defined, for example, for entropy coded prediction 306, intra-frame prediction 307, motion vector prediction 308, motion compensated prediction 309, adaptive loop filtering 310, adaptive interpolation filtering 311, deblocking filtering 312, sample adaptive offset 313, and possibly other prediction and in-loop filtering tools defined in the video coding mechanism.

Including PTI for separate prediction and in-loop filtering mechanisms for all slices and pictures that refer to the parameter set may help adapt the bitstream to the encoding and/or decoding environment, such as the hardware architecture of the encoder or decoder. Since the flag may be part of a parameter set that can be applied to many slices or pictures, the overhead of PTI in the parameter set may be negligible compared to the benefits provided by PTI.

Referring to the example 400 depicted in Figure 4, there is shown a coded bitstream 401 comprising a parameter set 402 and a coded picture comprising two slices 403, 404. Each slice starts with a header 405, 406. The slice header 405 is enlarged to show part of its information.

According to an embodiment, the slice header 405 may include multiple prediction tool indicator flags (PTIs). For example, when one or more PTIs are set, prediction and/or in-loop filtering across segment boundaries may be allowed, regardless of which encoding or decoding tool is associated with the flag; otherwise, when the PTI is set (i.e., disabled), the prediction may be prohibited. Flags may be defined for, for example, entropy prediction 407, intra-frame prediction 408, motion vector prediction 409, motion compensated prediction 410, adaptive loop filtering 411, adaptive interpolation filtering 412, deblocking filtering 413, sample adaptive offset 414, and possibly other prediction and in-loop filtering tools defined in the video coding mechanism.

Including PTI for separate prediction and in-loop filtering mechanisms involving a given slice may help adapt the bitstream to the content, thus improving coding efficiency.

We now describe how the two mechanisms described above can interact.

Referring to the example 500 shown in FIG. 5 , FIG. 5 shows an encoded bitstream 501 comprising a parameter set 502 and two slices 503 , 504 , each of the two slices 503 , 504 starting from a corresponding slice header 505 , 506 .

The parameter set 502, shown enlarged at 507, includes, for example, tile control information 508 or other information related to a headerless segment boundary, which can, for example, indicate a vertical tile boundary 204 as shown in FIG. 2 . Furthermore, the parameter set 502 may include one or more PTIs. Three PTIs are shown here, one associated with entropy prediction 509, one associated with intra-frame prediction 510, and one associated with motion compensation 511. These flags can control decoder prediction at the tile boundary 204. The tile boundary 204 can be set, for example, by the tile control information 508 so that the image 201 is vertically divided into two tiles 202, 203. The mechanisms described herein can also be used for other configurations of tile boundaries (including combinations of vertical and horizontal boundaries).

The coded image may also include, for example, two coded slices 503 and 504, each beginning with a corresponding slice header 505 and 506. As shown in FIG2, the (uncoded) slices corresponding to the coded slices 503 and 504 may, for example, include spatial regions of macroblock addresses 1 to 14 and 15 to 24, respectively. The slice header 506 is shown enlarged at 512 and may include multiple PTIs. Two PTIs are shown, one associated with intra-frame prediction 513 and the other associated with adaptive loop filtering (ALF) 514. However, it should be noted that overlap between the parameter sets 502 or the PTIs of the slice header 506 may exist but is not required.

According to an embodiment, the PTIs 509 , 510 , 511 of the parameter set 502 control prediction and in-loop filtering across tile boundaries 204 defined by the tile control information 508 .

According to one embodiment, the PTI 513, 514 of the slice header 512 controls prediction and in-loop filtering across the boundary between slices 503, 504. For example, the slice boundary of slice 504 has a boundary in addition to the picture boundary marked by the dashed bold slice boundary line 205.

Thus, in example 200, some prediction and in-loop filtering mechanisms are interrupted at tile boundaries (so as to allow image encoding to be distributed among several processors), while other prediction and in-loop filtering mechanisms are selectively interrupted at slice boundaries under the control of the slice header 506 (thereby giving the encoder full control by breaking up the prediction and in-loop filtering mechanisms so that any particular combination of prediction and in-loop filtering mechanisms can be selected for the content being encoded, including combinations that are desirable or convenient for a given application or use).

If PTIs involving the same prediction or in-loop filtering mechanism appear in the parameter set 502 and the slice header 506, and if the corresponding tile and slice boundaries are aligned, then at least two decoder reactions are possible. This can be selected by the profile/level so that the selection is statically specified in the flags, or dynamically specified in the flags based on control information in the parameter set or other high-level syntax content.

One option is for the PTI in the parameter set 502 to override the conflicting information in the slice header 506. This option may have the effect of enabling the decoder to determine that segments can be allocated to individual processors or cores without having to implement a mechanism to allow information to be shared between these segments.

Another option is for the PTI in the slice header 508 to override the conflicting information in the parameter set 502. This option may allow the encoder greater flexibility in its choice of tools. Other responses are also possible.

To optimize the coding flag (if the optimized coding flag is in the slice header 508 or parameter set 502), in some cases it may be advantageous to specify any of the following criteria:

(1) If a specific class and/or level is indicated, some PTIs may not be part of the parameter set or slice header because in that class and/or level, prediction or in-loop filtering tools are not available.

(2) If, for example, it is determined in a particular profile that the flexibility to independently switch these individual PTIs on/off is not needed or even desired, then two or more PTIs are "bundled" into a single combined PTI. [93] (3) In some cases, it may be better not to encode the PTI as a Boolean (i.e., binary) parameter. For example, the need for inter-processor coordination in the case of motion compensation may be determined at least in part by the length of a motion vector that points outside the co-located spatial region covered by a slice or tile. Therefore, in one embodiment, the PTI information may also be encoded as an integer or other non-Boolean parameter, thereby indicating an appropriate prediction value range, such as the maximum length of a motion vector that points outside a segment boundary.

(4) In some cases, PTI values may not need to be physically present in the bitstream, as their values can be derived from other characteristics of the bitstream. For example, PTI related to motion compensation may not need to be included within a slice, as motion compensation cannot occur within a slice, as designed by the standard.

The operation of an encoder according to an embodiment, which may be applicable to any of the aforementioned configurations of PTI information, will now be described.

6 , in one embodiment, an encoder may operate according to a flowchart 600. Before encoding the first slice of a video sequence, the encoder may determine (601) sequence-dependent settings for PTI and tile layout of video images in the image sequence. This determination may take into account the hardware architecture of the encoder, the hardware architecture of the decoder, possible tile layouts suggested or indicated by the hardware architecture, knowledge about the transmission network (if any), such as the size of the MTU, and the like. In some cases, the PTI value may be sanctioned by a system-level standard that the encoder may take into account in this determination. For example, future digital TV standards may envision the need for specific tile layouts and specific PTI settings that control prediction and in-loop filtering across tile boundaries for specific (higher) resolutions to enable cost-effective multi-processor/multi-core implementations. It may be necessary to determine only a subset of all PTIs at the sequence level.

Several options for those settings have been described previously.

After determining, the encoder can (602) encode the PTI associated with the sequence into a suitable high-level syntax structure such as a sequence or picture parameter set, sequence, GOP or picture header. The encoder can also choose not to define PTI during the encoding (through the syntax structure of the video coding standard).

The sequence-dependent PTI can remain constant for at least one complete video picture (unless overridden by a slice header-based PTI, as described later), but in many cases can remain constant for at least one "sequence" (all pictures between two IDR pictures and the preceding IDR picture in the video stream) and possibly during the entire video encoding session. For example, the sequence-dependent PTI can be driven, at least in part, by hardware limitations that cannot be changed during a session. This latter case is assumed hereafter for convenience.

The encoder continues by encoding the slice. To do so, the encoder may determine (603) a slice-level PTI, which may interact with the sequence-related PTI as already described. The slice-level PTI may be encoded as part of the encoding of the (604) slice header.

The slice may then be encoded (605) depending on which coding standard is being applied (eg, WD4 or H.264), while taking into account the breakdown of prediction and/or in-loop filtering mechanisms across slice and tile boundaries as indicated by the PTI.

The encoding continues (606) with the next slice.

The operation of a decoder according to an embodiment applicable to any of the aforementioned PTI information configurations will now be described.

FIG7 is a flowchart 700 of a decoder that may be used in accordance with one embodiment of the present invention. The decoder may receive (701) a NAL unit from a bitstream and determine the type of the NAL unit. If the NAL unit type indicates a parameter set (702), the decoder may perform parameter set parsing and storage (703) according to the employed video coding standard (other high-level syntax structures, such as sequence, GOP, or picture headers, may also be used for this purpose).

If the NAL unit type is used to indicate slice data (704) (other cases are not described), the decoder may parse the slice header (705) and then respond based on the information encoded therein, such as PTI information. For example, the slice header may include a parameter set reference, and the parameter may be "activated" (706), as described in video coding standards, indicating that the value of the referenced parameter set becomes valid. Since the PTI may be part of a parameter set, the value of the PTI may also become valid through this activation (706).

The slice header may also include its own PTI, which, as already described, may be different from the PTI included in the parameter set. It has been described how the selection of how to arbitrate between the PTI information in the slice header and the PTI information encoded in the parameter set during encoding is performed. For example, by associating the slice header-based PTI (if present) with the parameter set header PTI (if present), and taking into account any constraints that may exist in other parts of the video coding standard (e.g., constraints and/or default settings for PTI made by profiles and levels), the decoder may determine (707) the final PTI setting to be used for decoding the subject slice. It should be noted that depending on the PTI setting of the parameter set and the PTI setting of the slice header (including the special case where slice boundaries are aligned with tile boundaries), the PTI may be different for different edges of the slice.

Given the final PTI setting, the decoder may decode (708) the slice using prediction and/or in-loop filtering techniques across slice or tile boundaries as indicated by information encoded into the PTI.

The process continues (709) with the next NAL unit.

FIG7 does not illustrate the processing of NAL units other than slice or parameter set NAL units.

FIG8 is a block diagram illustrating an implementation based on a data processing system (e.g., a personal computer (“PC”)) 800 according to an embodiment of the present invention. Thus far, for convenience, the description has not been specifically related to possible physical implementations of the encoder and/or decoder. Many different physical implementations based on software and/or combinations of components are possible. In some implementations, the video encoder and/or decoder may be implemented, for example, using custom or gate array integrated circuits, in many cases for reasons related to cost efficiency and/or power efficiency.

In addition, software-based implementations using a general-purpose processing architecture (one example of which is a data processing system 800) are possible. For example, this implementation strategy may be possible using a personal computer or similar device (e.g., a set-top box, laptop, mobile device). As shown in FIG8 , according to the embodiment, an encoder and/or decoder for a PC or similar device may be provided in the form of a computer-readable medium 801 (e.g., a CD-ROM, semiconductor ROM, memory stick) comprising instructions, wherein the instructions are configured to allow a processor 802 to perform encoding or decoding alone or in combination with accelerator hardware (e.g., a graphics processor) 803 in conjunction with a memory 804 coupled to the processor 802 and/or accelerator hardware 803. The processor 802, memory 804, and accelerator hardware 803 may be coupled to a bus 805 that can be used to transfer bitstreams and uncompressed video to/from the aforementioned devices. Depending on the implementation, peripheral devices for input/output of bitstreams and uncompressed video may be coupled to the bus 805. A camera 806 can be attached to bus 805, for example, via a suitable interface such as a frame grabber 807 or a USB link 808, for real-time input of uncompressed video. Similar interfaces can be used for uncompressed video storage devices such as VTRs. Uncompressed video can be output via a display device such as a computer monitor or TV screen 809. A DVDRW drive or equivalent (e.g., a CD-ROM, CD-RW Blu-ray disc, memory stick) 810 can be used to enable input and/or output of bitstreams. Finally, for real-time transmission over a network 812, a network interface 811 can be used to deliver bitstreams and/or uncompressed video, depending on the capacity of the access link to the network 812 and the network 812 itself.

According to various embodiments, the above methods may be implemented by respective software modules, according to other embodiments, the above methods may be implemented by respective hardware modules, or according to other embodiments, the above methods may be implemented by a combination of software modules and hardware modules.

Although the embodiments are described primarily with reference to an exemplary method for convenience, the apparatus discussed above with reference to the data processing system 800 can be programmed to implement the method according to the embodiments. In addition, an article of manufacture for the data processing system 800, such as a pre-recorded storage medium or other similar computer-readable medium or product including program instructions recorded thereon, can direct the data processing system 800 to facilitate implementation of the method. It will be understood that, in addition to the method, such apparatus and articles of manufacture also fall within the scope of the embodiments.

Specifically, according to one embodiment of the present invention, a sequence of instructions that, when executed, causes the data processing system 800 to perform the method described herein may be included in a data carrier product. The data carrier product may be loaded into the data processing system 800 and run by the data processing system 800. In addition, according to one embodiment of the present invention, a sequence of instructions that, when executed, causes the data processing system 800 to perform the method described herein may be included in a computer program or software product. The computer program or software product may be loaded into the data processing system 800 and run by the data processing system 800. In addition, according to one embodiment of the present invention, a sequence of instructions that, when executed, causes the data processing system 800 to perform the method described herein may be included in an integrated circuit product (e.g., a hardware module or modules) that may include a coprocessor or memory. The integrated circuit product may be installed in the data processing system 800.

The above embodiments may contribute to an improved system and method for selectively disrupting prediction and/or in-loop filtering in video coding, and may provide one or more advantages. For example, including a PTI for individual prediction and in-loop filtering mechanisms, where the PTI relates to all slices and pictures that reference the parameter set, may help adapt the bitstream to the encoding and/or decoding environment, such as the hardware architecture of an encoder or decoder. Furthermore, including a PTI for individual prediction and in-loop filtering mechanisms that relates to a given slice may help adapt the bitstream to the content, thereby improving coding efficiency.

The embodiments of the invention described herein are intended to be examples only. Therefore, various changes and/or modifications to the details may be made to these embodiments, all of which fall within the scope of the invention.

Claims (6)

1. A method for decoding an encoded video image comprising multiple tiles, the method comprising: For at least one tile of the encoded video image that does not have an associated tile header, obtain at least one indication from the encoded video image of a sample adaptive offset operation to be applied to the boundary between two tiles in the encoded video image; and In response to the at least one indication, the sample adaptive offset operation between the two tiles is controlled. The encoded video image comprises at least two tiles, and at least one of the tiles comprises at least a portion of two tiles without an associated tile header. 2. A method for encoding a video image comprising multiple tiles, the method comprising: For at least one tile of the video image that does not have an associated tile header, obtain at least one indication of a sample adaptive offset operation to be applied to the boundary between two tiles in the video image; and In response to the at least one instruction, the sample adaptive offset operation between two tiles is controlled during the encoding of the video image. The video image comprises at least two tiles, and at least one of the tiles comprises at least a portion of two tiles without an associated tile header. 3. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, configure the processor to perform the method according to any one of claims 1-2. 4. An apparatus for encoding or decoding video images, comprising: Computer-readable storage media storing computer-executable instructions; and One or more processors, the processors being coupled to the computer-readable storage medium and configured to execute the computer-executable instructions to cause the device to perform the method according to any one of claims 1-2. 5. An apparatus for decoding an encoded video image comprising multiple tiles, the apparatus comprising: A component for obtaining at least one indication of a sample adaptive offset operation to be applied to the boundary between two tiles in the encoded video image, for at least one tile of the encoded video image that does not have an associated tile header; and A component for controlling the sample adaptive offset operation between two tiles in response to the at least one indication. The encoded video image comprises at least two tiles, and at least one of the tiles comprises at least a portion of two tiles without an associated tile header. 6. An apparatus for encoding a video image comprising a plurality of tiles, the apparatus comprising: A component for obtaining at least one indication of a sample adaptive offset operation to be applied to the boundary between two tiles in the video image, for at least one tile of the video image that does not have an associated tile header; and A component for controlling the sample adaptive offset operation between two tiles during the encoding of the video image in response to the at least one instruction. The video image comprises at least two tiles, and at least one of the tiles comprises at least a portion of two tiles without an associated tile header.