GB2505408A

GB2505408A - Video Encoding and Decoding with Chrominance Sub-sampling

Info

Publication number: GB2505408A
Application number: GB201215164A
Authority: GB
Inventors: Marta Mrak; Andrea Gabriellini
Original assignee: British Broadcasting Corp
Current assignee: British Broadcasting Corp
Priority date: 2012-08-24
Filing date: 2012-08-24
Publication date: 2014-03-05
Also published as: GB201215164D0

Abstract

A method of video encoding utilising a transform operation on separate luminance and chrominance transform blocks. Luminance and chrominance information relating to a common image region is received, the chrominance information being horizontally sub-sampled with respect to the luminance information (such as in the 4:2:2 sub-sampling format.) Square luminance transform blocks are formed by quad-tree splitting of the received luminance information and square chrominance transform blocks are formed by binary splitting and quad-tree splitting of the received chrominance information. Entropy encoding is then performed on each luminance and chrominance transform block, optionally preceded by transform and/or quantisation. The luminance and chrominance information may optionally represent residual luminance and chrominance after deduction of a prediction for the common image region or part of the common image region. The binary splitting of the chrominance information may be conducted in a root block preceding the quadtree splitting. Also disclosed in an equivalent method of video decoding.

Description

VIDEO ENCODING AND DECODING WITH CHROMINANCE SUB-SAMPLING

FIELD OF THE INVENTION

This invention is related to video compression and decompression systems in which chrominance information is sub-sampled relative to luminance information and in particular to such systems where the chrominance sub-sampling varies between horizontal and vertical directions.

BACKGROUND OF THE INVENTION

It is well understood that since the human visual system is much more sensitive to variations in brightness than colour, a video compression system need devote less bandwidth to chrominance information (typically colour difference components Cb and Cr) than to luminance information, the luminance component being usually denoted Y. Using the standard format notation in which 4:4:4 indicates no chrominance sub-sampling, video compression systems commonly utilise 4:2:0 in which Cb and Cr are each sub-sampled at a factor of 2 both horizontally and vertically. In for example H.2621MPEG2 or H.264/AVC, a macro-block may contain four 8X8 luminance blocks but only one Cb block and only one Cr block.

In 4:2:0, (as well as of course in 4:4:4) there is uniform sampling of chrominance in both horizontal and vertical directions. In high quality professional applications (for example CCIR 601) it has long been common to employ 4:2:2 in which Cb and Cr are each sub-sampled at a factor of 2 in only the horizontal direction. Interest has been expressed in new compression systems -for example High Efficiency Video Coding (HEVC) -in accommodating video content such as 4:2:2 where the chrominance sub-sampling varies between horizontal and vertical directions.

An important advantage of HEVC is the flexibility it offers in block structures. Block structures can be defined separately for the purposes of prediction and for transformation. A quadtree block splitting approach is employed, with the depth or level of block splitting being a parameter available to the encoder to optimise its performance depending upon video content and other relevant constraints. Because in 4:2:2 the chrominance information is sub-sampled only in the horizontal direction, a chrominance block corresponding with a square luminance block will now be rectangular. It remains possible to use for chrominance the same block splitting structure as used for luminance. Transform and related operations can be modified to accommodate rectangular chrominance blocks.

It is an object of this invention to provide more efficient techniques in encoding and decoding to accommodate video formats such as 4:2:2 where chrominance sub-sampling varies between horizontal and vertical directions.

SUMMARY OF THE INVENTION

According to one aspect of the invention there is provided a method of video encoding utilising a transform or other operation on separate luminance and chrominance transform blocks, comprising the steps of: receiving luminance and chrominance information relating to a common image region, the chrominance information being horizontally sub-sampled with respect to the luminance information, the luminance and chrominance information optionally representing residual luminance and chrominance after deduction of a prediction for that region or for part of that image region; forming square luminance transform blocks by quadtree splitting of received luminance information; forming square chrominance transform blocks by binary splitting and quadtree splitting of received chrominance; and performing, on each luminance transform block and on each chrominance transform block, entropy coding of block coefficients optionally preceded by transform and/or quantisation.

The binary splitting of chrominance information may be conducted in a root block preceding quadtree splitting; alternatively the binary splitting of received chrominance information may be conducted in leaves of the quadtree splitting A rectangular chrominance block corresponding to the same image region as a square luminance block may be split using a binary splitting of the root and where its quad-split branches are derived from the block splitting structure used in that square luminance block. A rectangular chrominance block corresponding to the same image region as a square luminance block may be split according to the block splitting structure employed in a selected half of that square luminance block.

The manner of chrominance splitting may be signalled to a decoder, for example by reference to the manner of luminance splitting.

In entropy coding, the coefficient scanning orders for chrominance may differ between uniformly sampled chrominance (for example 4:4:4 or 4:2:0) and non-uniformly sampled chrominance (for example 4:2:2). Similarly, the contexts may differ between blocks representing uniformly sampled chrominance and blocks representing non-uniformly sampled chrominance.

According to another aspect of the invention there is provided a method of video decoding utilising an inverse transform or other operations on separate luminance and chrominance transform blocks, comprising the steps of: receiving coefficients representing square luminance transform blocks formed by quadtree splitting of luminance information; receiving coefficients representing square chrominance transform blocks formed by binary splitting and quadtree splitting of chrominance, said luminance and chrominance information relating to a common image region, the chrominance information being horizontally sub-sampled with respect to the luminance information, the luminance and chrominance information optionally representing residual luminance and chrominance after deduction at an encoder of a prediction for that region or for part of that image region; and performing entropy decoding of block coefficients optionally followed by dequantisation and/or inverse spatial transform for each luminance transform block and for each chrominance transform block.

The manner of chrominance splitting may be signalled to a decoder, for example by reference to the manner of luminance splitting. Alternatively, the decoder decodes the manner of chrominance splitting, for example by reference to the manner of luminance splitting.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1: 4:2:2 digital video sampling with co-sited samples of different components Figures 2 to 4: Examples of block splitting in HEVC illustrating an embodiment of the invention Figures 5 and 6: Examples of block splitting illustrating another embodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The example will be taken of the handling of rectangular chrominance blocks for 4:2:2 in HEVC. As noted above, 4:2:2 format refers to the case when chrominance components (Cb and Cr) are sampled at half the horizontal resolution of the luminance (Y). Figure 1 gives an example which demonstrates such sub-sampling.

Since 4:2:2 format is usually used in professional applications, it is associated with high bit-rates and high qualities of compressed videos.

Different from earlier video compression standards, HEVC uses recursive block splitting structures. This more flexible structure allows better adaptation of pixel-level compression parameters. However, solutions available for encoding 4:2:2 formats are not currently defined in such coding design.

It may be helpful first to view the recursive block splitting structures of HEVC.

Double recursive block splitting, with an additional prediction split is the base for HEVC pixel-level processing framework. First recursive splitting is performed on blocks of the Coding Tree Units, to define Coding Units. In the second recursive split such blocks are divided to define blocks on which actual transform is performed. The decisions for each split are sent in the bitstream, while a number of normative restrictions apply.

In the following explanation, uniform sampling in both directions is used, as used for the luminance on which the main partitioning rules are applied.

Coding Tree Unit (CTU) -In HEVC frames are divided into rows of square Coding Tree Units (CTU5). Typically, a CTU consists of an N x N block of single component samples (e.g. luminance samples). Additionally, other components are typically related to a CTU, like chrominance samples (U and V, i.e. Cb and Cr). In 4:2:0 format, each chrominance component related to a CTU consists of N/2 x N/2 samples. In the main profile of the standard, the maximum allowed N is 64. Differently from macro blocks used in earlier standards, CTUs in HEVC are not just larger, but also have different possibilities for their divisions into smaller blocks.

Coding Unit (CU) -A CTU consist of a number of Coding Units (CUs). CU is a square block that can be the size of related CTU, or smaller (up to 8 x 8 luminance samples).

CTU split into a number of CUs is defined by a quadtree, i.e. such split has a recursive structure. Coding (prediction) inside a CU can be intra or inter. Several examples of CTU split into CUs are given in Figure 2.

Prediction Unit (PU) -Each CU is associated with 1, 2 014 Prediction Units (PUs), Figure 3. The main information related to each PU is the prediction. For example, if a CU is an intra coded CU then a prediction direction is defined for each PU (e.g. horizontal, vertical, or other angular prediction direction). All options are used for inter prediction CU, while only PART_2Nx2N and PART_NxN partition modes are used for intra CUs.

Additional restrictions apply depending on given CU size.

Transform Unit (TU) -Each CU contains one or more Transform Units (TU5). While the PU split defines prediction in different parts of CU, TU split defines transforms within a CU. Each TU contains a square block of luma samples, ranging from 32x32 to 4x4 pixels. Similar to the CTU split into CUs, a CU split into TUs is described by a quadtree structure. In this case the split is described by a Residual QuadTree (RQT) structure.

Some specific restrictions are applied, depending on the prediction mode, PU split, given component (YUV) and block sizes. An example that shows where the ROT stands relatively to CTU/PU splits) is given in Figure 4 with up-right CU split into two PUs using PART_2NxN partitioning and one level of TU split (in black). The actual transforms and quantisation are performed on blocks of pixels related to each TU.

Turning now to the handling of 4:2:2, each rectangular chrominance block in 4:2:2 format (like in e.g. Figure 1), can be linked to a collocated square luminance block when it comes to CU and PU blocks. The processing required at the levels of CU and PU blocks is usually of limited precision. Therefore handling larger rectangular blocks is not a burden for a codec. However, when it comes to the transform level, an alternative solution would be desirable to support 4:2:2 chrominance sampling. This is because at the transform level several operations on a pixel level are required, each operating with high precision and larger complexity requirements (such as pixel-level multiplying).

These operations include transforms, quantisation and entropy coding. Complex implementations of such operations for square blocks will already be present in pre-existing hardware and software. Re-design of such implementations in order to support rectangular blocks is undesirable.

This invention avoids the need to support rectangular transform blocks, by application of a transform tree that is not purely a quadtree. Now it is mandatory to include binary splits within the transform quadtree structure, to achieve square transform blocks of chrominance at some level of RQT.

Note that even in the cases when transform is not used (e.g. transform skip or lossless coding) defining chroma structures into "transform blocks" is still needed since entropy coding of block coefficients is used on such blocks.

According to one embodiment of the present invention, chrominance rectangular coding blocks are split into square transform blocks with binary root-ROT (BR-ROT). Thus, the first level of ROT is actually a binary split, i.e. has two branches; in this way each rectangular coding block is split into two square blocks. Other levels of ROT are regular ROT splits. In some cases, the binary split can be assumed without need for signalling to the decoder; e.g. for 32x64 blocks it is needed to have some sort of split to get to a size supported by available transforms. With such an assumed binary split, it is necessary to signal only any further RQT splitting. In other cases, both the binary split and any further ROT splitting will require to be signalled.

The BR-ROT chrominance split may be defined independently; alternatively it may be defined with respect to the luminance split. Thus, BR-ROT may be taken from the left or right part of the luminance block as shown in FigureS. It can be signalled in the bit-stream which one to use, or it can be pre-defined which part of luminance ROT is used (e.g. only left part, or only right part). It may be appropriate to select between the left or right part of the luminance block in dependence upon pre-defined rules; for example the most or the least divided part could be selected. The chrominance split may be defined with respect to the luminance split in other ways. For example, the chrominance split may be at one or a defined number of levels higher or lower than the luminance split in a corresponding region. Maximum or minimum block size constraints may also be applied with any of these techniques.

In another embodiment of the present invention, there is a binary split of leaves of quadtrees related to chrominance, as shown in Figure 6. Because of correlation between luminance and chrominance components it may be desirable that splitting of coded blocks for all components is spatially equivalent. E.g. if a coded block has more detailed content in its bottom right corner, then it is likely that the splitting into smaller blocks, using quadtree, is more utilised there than in other parts of the block, for all components. Therefore in this case the quadtree is used from the root of the new tree.

At the end of the quadtree splitting, each rectangular block of chrominance is split into square blocks using binary split, enabling the required square format for transform-level operations (transform, quantisation, entropy coding).

Another important aspect in accommodating 4:2:2 is the coding applied to the coefficients which result from the spatial transform. Since in 4:2:2 format chrominance blocks are down sampled (with respect to luminance) in the horizontal direction only, on average that direction is likely to carry more information than the vertical direction. This property might also apply to properties of residuals. Therefore, after the transform, it may not be appropriate to assume equal distribution of all transform coefficients as for uniformly sampled signals (e.g. like in luminance or chrominance in 4:2:0 and 4:4:4 formats). It may be appropriate to change scanning orders in some of these cases, e.g. applying horizontal scans instead of the diagonal scans that are appropriate where there is uniformity between horizontal or vertical directions. Additionally, because of different sampling strategies used for forming 4:2:2 signals, other scanning strategies (like vertical scanning) may be defined for chrominance components in 4:2:2 signal. It may also be appropriate to readjust the contexts for the binary arithmetic coding of the quantised transform coefficients reflecting the different statistical properties of the underlying signal. The coefficient scanning or entropy coding contexts for a square chrominance block representing 4:2:2 may then be different from those used for a square chrominance block representing 4:2:0 or 4:4:4. Similar considerations may lead to the choice of different quantisation matrices.

It will be understood that the present invention has been described by way of example only and that a wide variety of modifications are possible without departing from the scope of the invention as claimed. For example, the invention may find application in coding schemes other than HEVC

Claims

CLAIMSA method of video encoding utilising a transform or other operation on separate luminance and chrominance transform blocks, comprising the steps of: receiving luminance and chrominance information relating to a common image region, the chrominance information being horizontally sub-sampled with respect to the luminance information, the luminance and chrominance information optionally representing residual luminance and chrominance after deduction of a prediction for that region or for part of that image region; forming square luminance transform blocks by quadtree splitting of received luminance information; forming square chrominance transform blocks by binary splitting and quadtree splitting of received chrominance; and performing, on each luminance transform block and on each chrominance transform block, entropy coding of block coefficients optionally preceded by transform and/or quantisation.
2. A method according to Claim 1, wherein the binary splitting of chrominance information is conducted in a root block preceding quadtree splitting.
3. A method according to any one of the preceding claims, in which a rectangular chrominance block corresponding to the same image region as a square luminance block is split using a binary splitting of the root and where its quad-split branches are derived from the block splitting structure used in that square luminance block.
4. A method according to any one of the preceding claims, in which a rectangular chrominance block corresponding to the same image region as a square luminance block is split according to the block splitting structure employed in a selected half of that square luminance block.
5. A method according to Claim 1, wherein the binary splitting of received chrominance information is conducted in leaves of the quadtree splitting.
6. A method according to any one of the preceding claims, wherein the manner of chrominance splitting is signalled to a decoder.
7. A method according to Claim 6, wherein the manner of chrominance splitting is signalled to a decoder by reference to the manner of luminance splitting.
8. A method according to any one of the preceding claims, where in entropy coding the coefficient scanning orders for chrominance differ between uniformly sampled chrominance (for example 4:4:4 or 4:2:0) and non-uniformly sampled chrominance (for example 4:2:2).
9. A method according to any one of the preceding claims, where in binary entropy coding the contexts differ between blocks representing uniformly sampled chrominance and blocks representing non-uniformly sampled chrominance.
10. A method of video decoding utilising an inverse transform or other operations on separate luminance and chrominance transform blocks, comprising the steps of: receiving coefficients representing square luminance transform blocks formed by quadtree splitting of luminance information; receiving coefficients representing square chrominance transform blocks formed by binary splitting and quadtree splitting of chrominance, said luminance and chrominance information relating to a common image region, the chrominance information being horizontally sub-sampled with respect to the luminance information, the luminance and chrominance information optionally representing residual luminance and chrominance after deduction at an encoder of a prediction for that region or for part of that image region; and performing entropy decoding of block coefficients optionally followed by -10-dequantisation and/or inverse spatial transform for each luminance transform block and for each chrominance transform block.
11. A method according to Claim 10, wherein the binary splitting of chrominance information is conducted in a root block preceding quadtree splitting.
12. A method according to Claim 10 or Claim 11, in which a rectangular chrominance block corresponding to the same image region as a square luminance block is split using a binary splitting of the root and where its quad-split branches are derived from the block splitting structure used in that square luminance block.
13. A method according to any one of Claim 1010 Claim 12, in which a rectangular chrominance block corresponding to the same image region as a square luminance block is split according to the block splitting structure employed in a selected half of that square luminance block.
14. A method according to Claim 12, wherein the binary splitting of received chrominance information is conducted in leaves of the quadtree splitting.
15. A method according to any one of Claim 1010 Claim 14, wherein the manner of chrominance splitting is signalled to a decoder.
16. A method according to Claim 19, wherein the manner of chrominance splitting is signalled to a decoder by reference to the manner of luminance splitting.
17. A method according to any one of Claim 10 to Claim 14, wherein the decoder decodes the manner of chrominance splitting.
18. A method according to Claim 17, wherein the manner of chrominance splitting is decoded by reference to the manner of luminance splitting.
19. A method according to any one of Claim 1010 Claim 18, where in entropy decoding the coefficient scanning orders for chrominance differ between uniformly sampled chrominance (for example 4:4:4 or 4:2:0) and non-uniformly sampled chrominance (for example 4:2:2).
20. A method according to any one of Claim 1010 Claim 19, where in binary entropy decoding the contexts differ between blocks representing uniformly sampled chrominance and blocks representing non-uniformly sampled chrominance.
21. A video encoder configured to implement a method according to any one of Claim ito Claim 20.
22. A video decoder configured to implement a method according to any one of Claim 10 to Claim 20.
23. A non-transitory computer program adapted to cause programmable apparatus to implement a method according to any one of Claim Ito Claim 20.