HK1228145B - Perceptual color transformations for wide color gamut video coding - Google Patents

Description

Perceptual color transform for wide color gamut video coding

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Patent Application No. 62/156,124, filed May 1, 2015, which is hereby incorporated by reference in its entirety.

Technical Field

The present disclosure relates generally to video coding.

Background Art

There is growing interest in distributing video or other visual content with high dynamic range (HDR) and wide color gamut (WCG) due to its ability to provide a more enhanced visual experience compared to traditional standard dynamic range (SDR) and standard color gamut (SCG) content. The brightness of SDR content typically ranges from, for example, 0.1 to 100 nits, a range significantly lower than what the human eye can see in real life. However, footage shot or remastered in HDR and WCG contains more information, which in turn requires larger file sizes with increased bit rates.

Summary of the Invention

Various embodiments are directed to providing improved content (e.g., picture) quality at a fixed bit rate by using a constant luma encoding pipeline, wherein the picture content can be converted to a near-perceptually uniform luma/chroma (Y/U'V') color space. Furthermore, the present invention discloses various chromaticity transforms used in the encoding pipeline (using a dedicated chromaticity transform distinct from and in addition to the photometric transform).

According to one embodiment of the present disclosure, a computer-implemented method includes converting an additive color model signal into a uniform color space signal having a chromaticity component and a luminance component. The computer-implemented method further includes transforming the chromaticity component and the luminance component by applying a chromaticity ratio transform to the chromaticity component and applying a luminance ratio transform to the luminance component. Furthermore, the computer-implemented method includes quantizing the transformed chromaticity component and the luminance component, and encoding the quantized chromaticity component and the luminance component.

According to another embodiment of the present disclosure, a non-transitory computer-readable storage medium has computer executable program code included thereon, which is configured to cause a processor to: decode quantized color and luminance components of an input bit stream carrying an image signal, the image signal being represented by a first color space model having luminance and color components; dequantize the quantized color and luminance components; apply a first inverse transform to the dequantized color components; apply a second inverse transform to the dequantized luminance components; and convert the image signal represented by the first color space model into an image signal represented by a second color space model, which second color space model is an additive color space model.

According to another embodiment of the present disclosure, a system includes a converter for converting an image signal represented by a first color space model into an image signal represented by a second color space model having luminance and color components. The system further includes a first converter for applying the first transform to the color components; an optical transfer function encoder for applying the second transform to the luminance components; a quantizer for quantizing the transformed color and luminance components; and an encoder for encoding the quantized color and luminance components and outputting a bitstream.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in detail according to one or more embodiments with reference to the accompanying drawings, which are for illustration purposes only and depict only typical or example embodiments.

1 is a flow diagram illustrating example operations that can be performed to achieve a constant luma encoding pipeline with dedicated chromaticity transforms, in accordance with various embodiments.

2A is a schematic diagram representation of a video processing pipeline in which the constant luma encoding pipeline with dedicated chromaticity transform of FIG. 1 may be implemented according to various embodiments.

2B is a schematic representation of a constant luminance encoding pipeline with the dedicated chromaticity transform of FIG. 1 that may be used with the video processing pipeline of FIG. 2A .

FIG3 is a schematic representation of a conventional non-constant brightness pipeline.

4 is a conceptual representation of a chromaticity transform that may be used with the constant luma encoding pipeline of FIG. 1 with a dedicated chromaticity transform.

5 is an example representation of the subdivision of the color space into a grid for applying a chromaticity transform that may be applied to the constant luminance encoding pipeline with the dedicated chromaticity transform of FIG. 1 .

FIG. 6 is an example representation of the formation of a conventional mesh when a transformation is applied to each vertex of the mesh.

FIG. 7A is an example representation of the stretching step of a direct transform utilized in transforming chromaticity values.

FIG. 7B is an example representation of vertices connected to boundary points of a polygon.

FIG. 7C is an example representation of equidistant points of a color space triangle mapped to a unit square.

8A is an example representation of a non-optimized least-squares chromaticity transform that may be applied in a constant-luminance encoding pipeline with the dedicated chromaticity transform of FIG. 1 .

8B is an example representation of an optimized least-squares chromaticity transform that may be applied to a constant-luminance encoding pipeline with the dedicated chromaticity transform of FIG. 1 .

FIG. 9 illustrates example computing components that may be used to implement various features of the embodiments described herein.

The foregoing diagrams are not intended to be exhaustive and do not limit the disclosure to the precise forms disclosed herein.

DETAILED DESCRIPTION

Compared to conventional non-constant luma video encoding pipelines, various embodiments disclosed herein provide a constant luma encoding pipeline with dedicated chroma transforms. As mentioned above, footage in high dynamic range (HDR) and wide color gamut (WCG) contains more information than footage in standard dynamic range (SDR), which requires more storage space or an increased distribution bitrate. Because this bitrate is too high for a particular distribution channel, higher compression efficiency is required.

FIG1 illustrates example operations for achieving more efficient color (standard color gamut (SCG) and WCG) video encoding using a constant luma encoding pipeline with dedicated chroma transforms, according to various embodiments. FIG1 is illustrated in conjunction with FIG2B , which is a schematic diagram of such an encoding pipeline. The encoding pipeline 201 can be thought of as having a pre-processing portion 209 and an encoding portion (embodied by encoder 210).

In operation 100, an additive color model signal is converted to a uniform color space signal having a chromaticity component or channel and a luminance component or channel (implemented by converter 202 in FIG. 2B ). The red, green, blue (RGB) color model is an example of an additive color model, in which red, green, and blue light are added together in various ways to produce a variety of other colors. The RGB color model can be used for image perception, representation, and display in electronic systems (e.g., monitors, TVs, etc.). Input devices (e.g., cameras, image scanners, video game consoles, digital cameras, etc.) can input RGB signals to output devices (e.g., TVs, monitors, projectors, or other displays). Uniform color spaces can be referred to as CIE color spaces or YUV color spaces used in color image pipelines. Uniform color spaces can encode images or videos that take human perception into account and allow chromaticity components to mask transmission errors and compression artifacts (rather than using a "direct" RGB representation of video or images). The result of the transform is a signal having a luminance component or channel, Y (representing brightness), and chrominance components or channels, U and V (representing color).

As noted above, it is noteworthy that the conversion from the RGB color space to the YUV color space can be performed "directly" from RGB to YUV components or channels. However, according to other embodiments, an intermediate conversion (not shown) can be performed to convert the RGB components or channels into an XYZ color space, where the XYZ components can then be converted into the aforementioned YUV components.

In operation 102, the chromaticity and luma components are transformed by applying a chromaticity-specific transform to the chroma components and a luma-specific transform to the luma component (chroma and luma transformations are performed by transformers 206 and 204, respectively, in FIG2B ). As shown in FIG2B , the U and V channels are transformed into U′ and V′ channels, where the chromaticity transform maintains perceptual uniformity according to various embodiments (which is not possible using conventional transform methods utilized in conventional encoding pipelines). Notably, the luma component or channel is processed separately (e.g., using an optoelectronic transfer function (OETF) (performed by OETF encoder 204, FIG2B )), resulting in a gamma-compressed or encoded luma component Y′. The separate processing of the luma and chroma components further helps to make the compression more efficient.

In operation 104, the luma Y' and chroma components U' and V' are quantized (performed by quantizer 208 in FIG. 2B ), resulting in a signal having components or channels of DY', DU', and DV'. Quantization may involve a lossy compression technique that reduces the number of colors used to represent the image by compressing a series of values (in this case, luma and chroma component values) into a single quantum value.

In operation 106, the chromaticity components DU' and DV' are encoded (by encoder 210 in Figure 2B). The same is true for the quantized luminance component DY'. The encoder 210 can have a memory unit with computer code that is configured to cause a processor to apply a video codec, such as the High Efficiency Video Coding (HEVC) codec, which is the successor to Advanced Video Coding (AVC) also known as H.264. It should be noted that HEVC was originally designed for SDR and SCG content, and in traditional applications, HEVC is not optimal for encoding HDR or WCG video sequences in terms of compression efficiency. However, the above-mentioned chromaticity transform disclosed in the present disclosure converts the chromaticity components (U, V) into different values (U', V'), which is more efficient for compression purposes (better utilization of codewords at the (e.g., HEVC) codec level). In addition, as previously discussed, in order to maintain perceptual adaptation to the CIE or YUV color space, the transformed chromaticity components are made as perceptually uniform as possible.

2A illustrates an example encoder and decoder architecture system 200 in which various embodiments may be implemented. System 200 may include an encoder 210, a decoder 218, and (a plurality of) networks or distribution channels 216 in communication with both encoder 218 and decoder 210. Encoder 210 may include memory 212 and a processor 214, while decoder 218 may also include memory 220 and a processor 222. Decoder 218 may be a set-top box, a tuner, or the like. In operation, processor 214 of encoder 210 may execute computer instructions stored in memory 212, such as computer instructions for encoding a video sequence. That is, video content (which may be in HDR or WCG format) is fed into encoder 210 and then compressed in preparation for storage and/or transmission. For example, processor 214 of encoder 210 may encode a video sequence by performing a process using the HEVC codec described above, and transmit the encoded video sequence to decoder 218 via network 216. The decoder 218 (embodied as some type of receiving device, such as a set-top box, a tuner, etc.) may receive the encoded HDR or WCG video sequence for decoding and presentation to a display 224. That is, the processor 222 of the decoder 218 may execute computer instructions stored on the memory 220, such as computer instructions for decoding the received HEVC encoded video sequence.

As described above, the system 200 also includes a pre-processing portion 209 and a post-processing portion 223, respectively, wherein one or more algorithms are executed for the (vis-à-vis) converter 202, the OETF encoder 204, the chromaticity converter 206, and the quantizer 208 according to various embodiments. These components can be separate hardware and/or software components or functional parts of the same hardware/software component. Although the corresponding components of the post-processing portion 223 are not shown, it should be understood that it includes similar hardware and/or software for dequantization and inverse color transformation of the encoded video sequence. That is, the input bitstream can be decoded by the decoder 218. By decoding the input bitstream, the decoder 218 can convert the input bitstream into a series of transform coefficients, spatial prediction modes, and motion compensation vectors. It should be noted that the encoding and decoding aspects of the encoding pipeline do not need to include a modified (e.g., HEVC) codec.

As described above, conventional encoding pipelines are based on non-constant luminance encoding. FIG3 illustrates an example conventional encoding pipeline 301 to provide a juxtaposition between the prior art and the various embodiments disclosed herein. Conventional encoding pipeline 301 takes as input an image in RGB color space. Because each frame of the input video sequence must be processed separately, each step in the reference pipeline is applied to each frame in the sequence. The input (RGB) image is first converted to double precision in the range [0, 1]. Therefore, a nonlinear transfer function (TF) is applied by gamma encoder 302, which redistributes the linear luminance of the R, G, and B channels to other coding levels and produces nonlinear RGB components (denoted by R', G', and B'). It can be appreciated that small changes in luminance at lower intensities are better perceived than at higher intensities. Furthermore, at least a portion of the subsequent processes in conventional encoding pipeline 301 will be lossy, and the number of bits available for storage is limited. TF preserves valuable information at lower intensities by using a larger portion of the bit space for these lower values. If certain noticeable encoding errors are introduced into the image in RGB, the viewer will perceive these errors more at lower intensities than at higher intensities. If these errors are instead in the R'G'B' domain, once the image is converted back to RGB, these errors will also be noticeable at all intensities. Therefore, lower intensity values are stretched, while higher intensity values are compressed together.

According to the BT.2020 specification, the converter 304 converts the R'G'B' components into the Y'C'bC'r color space. The Y'C'bC'r color space represents the luminance component Y', the blue difference C'b, and the red difference C'r color components. The quantizer 306 functions to quantize the Y', C'r, and C'b components represented by CY', DC'r, and DC'b. The encoder 308 (e.g., using the AVC/H.264 codec) encodes the quantized components and outputs an encoded bitstream, which can be stored and/or transmitted as described above.

Compared to encoding pipeline 201, the luminance component Y' represents the light intensity that is nonlinearly encoded based on the gamma-corrected RGB primary colors. The encoding errors introduced in the luminance may leak into the chrominance and produce observable differences not only in the brightness of the light, but also in the colors presented by the display - thus, resulting in non-constant brightness aspects of the traditional encoding pipeline (e.g., encoding pipeline 301). Although this non-constant brightness encoding pipeline is designed for encoding SDR signals (e.g., RGB up to 100 nits in the BT.709 color space), it can also be repurposed for encoding HDR and WCG signals (e.g., RGB up to 10,000 nits in the BT.2020 color space). However, this traditional encoding pipeline still has design flaws when it continues to apply non-constant encoding of luminance. In HDR content, the encoding of these errors becomes more obvious (e.g., encoding distortions within the color range become more obvious in HDR content than in SDR content).

Returning to FIG. 2B , relative to the conventional encoding pipeline 301, the luma component Y is explicitly represented (distinguished from the chroma components U and V) in the constant luma encoding pipeline 201. Therefore, any quantization and encoding errors introduced into the luma component Y' only have an effect on luma Y. Specifically, leakage of compression errors into the chroma will not occur.

Specifically, instead of converting the image from RGB color space to Y'C'bC'r color space (as in the conventional encoding pipeline 301), the RGB signal is converted to, for example, XYZ color space (Equation 1), and then converted to YUV color space (Equation 2). To return to RGB color space (during decoding in the post-processing portion 223 of the system 200), the inverse matrix of Equation 1 can be used (Equation 3).

and

and

In addition to the constant luminance of encoding pipeline 201, a dedicated chromaticity transform is provided to improve chromaticity compression and increase overall compression efficiency. The dedicated chromaticity transform (applied by transformer 206 in FIG. 2B at operation 102 in FIG. 1 ) reduces coding errors in the color channels at a fixed bit rate by broadening the codeword distribution used for compression. It can be understood that the dedicated chromaticity transform only affects the chromaticity components U and V. The chromaticity transform takes these two components and returns the channel (U', V') = T(U, V), where T represents the transform described below.

It should be noted that when quantizing an image to reduce the bit depth, a chromaticity transform can be applied to the image. In addition, before compressing the image using a codec such as HEVC, a transform can be used to enhance the picture quality at a fixed bit rate. Traditional transforms do not have the same quality enhancement effect on video sequences at the same bit rate.

The overall goal of the chromaticity transformation is shown in FIG4 , which represents a conceptual color representation (as a triangle with the primary colors red, green, and blue in the corners of the triangle and all possible color combinations between these corners) transformed into a conceptual square (the conceptual square reduces the number of waste codewords).

As an illustration, the chromaticity coordinates of all colors in the color space lie within the CIE diagram of the YUV color space (triangle 402). The corners of triangle 402 show the three primary color coordinates RGB of a color space (e.g., BT.2020). All chromaticity values defined in this color space lie within triangle 402. The three RGB primary colors of a color space define the color gamut, or the range of all possible colors, of the color space. The closer a point is to the edge of the diagram, the more saturated its color. Therefore, the larger the triangle of the color space, the more saturated the three primary colors, and the larger the color gamut. It should be noted that footage captured and stored with BT.2020 primary colors has the largest color gamut and is referred to as WCG (as opposed to, for example, BT.709, which is an example of an SCG color space).

Dynamic range is defined as the ratio between the maximum and minimum light intensities. The higher the dynamic range, the greater the difference between the maximum and minimum light intensities. SDR footage typically has a brightness range of 0.1 to 100 nits (cd/ ^m² ). However, in real life, light sources can be well above 200 cd/ ^m² , and the night sky can be well below 0.1 cd/ ^m² . For HDR content, the minimum can be as low as 0.005 cd/ ^m² or as high as 10,000 cd/ ^m² .

Because an RGB image is used as input to the encoding pipeline 201, all chromaticity coordinate components of the pixels of the RGB image are located within the U-V chromaticity space of the triangle 402 defined by the R, G, and B vertices. However, encoding in the U-V chromaticity space allows the representation of many value pairs outside the triangle 402 (e.g., invalid chromaticity values). To reduce the waste of these codewords, various embodiments of the present disclosure map all (U, V) pairs into a rectangle (e.g., the unit square 404). To benefit from the perceptual uniformity of the U-V space used for compression as described above, no strong local distortion is introduced into the mapping from the triangle 402 to the unit square 404. The (u, v) pair can be defined as the chromaticity value of the pixel, where u∈U and v∈V. The (u, v) value of each pixel in the image will lie within this triangle or on its boundary.

As previously mentioned, one goal of the chromaticity transform disclosed herein is to provide a codec (e.g., HEVC) with a larger codeword range. The quantization step takes an image in double precision format in the range [0, 1] and quantizes it to 12-bit integer values in the range [0, 4095], mapping the minimum input value (0) to 0 and the maximum input value (1) to 4095. Thus, the U and V components can each utilize the range [0, 1]. The chromaticity value (u, v) of each image pixel can be any value inside the unit square 404, which is defined by the points (0, 0), (0, 1), (1, 0), and (1, 1). This is not the case based on how the YUV color space fits into a Cartesian coordinate system. Notably, the triangle 402 only occupies a relatively small portion of the unit square.

Therefore, the required transformation T is such that every possible value defined by the three primary colors (e.g., BT.2020) within triangle 402 is mapped to a point in unit square 404, thereby making the best possible use of its area—that is, transforming the triangle into a square. The transformed chromaticity value of each pixel (u', v') and certain constraints of this transformation are discussed below.

As described above, in the post-processing step 223 in FIG2 , an inverse transform T ⁻¹ must be performed, which means that when T ⁻¹ is applied, each transformed value (u′, v′) must be mapped back to the original value (u, v). In other words, the property expressed in Equation 4 below must be preserved, meaning that T is reversible and the chromaticity transform is a lossless step in the encoding pipeline 201.

It should be understood that T will cause large global distortions, completely changing the shape of the triangle. Once the chromaticity transform is integrated into the encoding pipeline 201, the U' and V' channels are altered because encoding and decoding are lossy and introduce errors. This means that (u _rec , v _rec ) = (u, v) + ε, where ε is a small error vector in any direction. The YUV color space is nearly perceptually uniform, so the perceived difference between (u _rec , v _rec ) and (u, v) should only depend on the length of ε, not the direction. This means that T should introduce minimal local distortion, meaning that for all chromaticity values close to a certain value (u, v), their new distance to the transformed value (u', v') may increase or decrease, but should all be approximately equal.

According to one embodiment, as will be discussed below, the chromaticity transform can be based on a direct transform. According to another embodiment, also described below, the chromaticity transform can be based on a least squares transform. It should be understood that other transforms can still be used according to other embodiments.

FIG5 illustrates an example of a method for subdividing triangle 402 in FIG4 in preparation for chromaticity value transformation according to various embodiments. Triangle 402 can be subdivided into a mesh consisting of smaller triangles formed by three vertices. To subdivide the triangle, the midpoints between the three corners of the triangle are marked and connected, thus forming four smaller triangles. A second subdivision can be performed for each smaller triangle, resulting in a total of 16 triangles. This subdivision process can be repeated, with each new subdivision quadrupling the number of triangles. Therefore, n subdivisions will produce ⁴ⁿ triangles, where n∈N and n≥1. Triangles 502 and 504 are examples of main triangles that have been subdivided three and four times, respectively. Triangle 502 has 64 smaller triangles after three subdivisions, while triangle 504 has 256 smaller triangles due to four subdivisions. The result is a list of triangles, where each triangle can be stored by remembering the three vertices in the mesh. Tr(a, b, c) represents the triangle defined by vertices a, b, and c.

Once n is chosen, each of the 4 ⁿ smaller triangles is mapped to the unit square 404. This results in a certain degree of distortion. For each triangle Tr(a, b, c), the corresponding mapped triangle Tr(a', b', c') is stored. The mapped triangles should not overlap each other.

When mapping a single chromaticity value (u, v), each (u, v) value lies within the main YUV color space triangle 402 (the main triangle is the larger, unsubdivided triangle), and thus also lies within one of the smaller triangles generated after subdividing triangle 402 (triangle 502). Once this triangle Tr(a, b, c) is found, the position p of the point (u, v) can be expressed by using the barycentric coordinates λ ₁ , λ ₂ and λ ₃ for Tr(a, b, c), where

Therefore, the correct position p' in the mapped triangle Tr(a', b', c') can be obtained by the following formula:

p′＝λ ₁ α′+λ ₂ b′+λ ₃ c′

Equation 4.3

This process is performed separately for each point. The coordinates of the center of gravity can be calculated using the following formula:

where the subscript of a point defines its u or v component, and

One property of barycentric coordinate mapping is that it is reversible, which means that the reverse mapping of a point (u', v') at its position p' places it at its original position p. As mentioned above, the transformation should be reversible to allow proper decoding of the image signal.

Finding a triangle when the chromaticity point (u, v) is included should be completed in a "reasonable" time, for example, during a live broadcast, decoding should be fast enough to prevent any lag/delay. According to one embodiment, the three barycentric coordinates of each triangle can be calculated and tested whether Equation 4.2 is valid. If so, the chromaticity point is inside the triangle. Testing each of the triangles resulting from the above (multiple) subdivisions separately is inefficient. Therefore, various embodiments rely on performing the following constant time test:

Consider a linear transformation that transforms a given chromaticity value (u, v),

Where R, G, and B represent the three primary colors (the three corners of triangle 402). If this transformation is applied to each vertex on the grid, the main triangle will occupy the lower left half of the unit square 404. As shown in Figure 6, the smaller triangles will be aligned in such a way that their two shorter sides can be horizontal or vertical (in this example, n is 3). If two adjacent triangles are brought together to form a square 600, this will form a regular grid.

To find the corresponding triangle for a point (u, v), equation 4.3 is applied to (u, v) instead of the mesh itself. For a specific subdivision value n, the square of the regular grid in which the transformed point (u, v) _{trans lies} is trivial. Since all triangles are indexed and each square contains two triangles, the remaining step is to find which of the triangles contains the point (u, v). This can be done by calculating the barycentric coordinates of (u, v) for both triangles and then testing whether equation 4.2 is valid.

During post-processing 223 ( FIG. 2A ), the location of each mapped chromaticity value (u′, v′) must lie within the original grid (e.g., FIG. 5 ). According to one embodiment, a lookup table is utilized. That is, for each possible value in the transformed grid, the lookup table specifies to which location in the original grid the value is mapped. The individual triangles are then mapped onto the unit square while adhering to the properties of the transform T.

As described above, one type of transformation used in the chromaticity value transformation may be a direct transformation. When a direct transformation is used, the mesh is considered to be a collection of vertices, where each vertex will be copied and mapped to some new position inside the unit square. This type of transformation involves selecting four vertices on the boundary line of the main triangle and stretching the main triangle by pulling these vertices to the four corners of the square. It is reasonable to select three corners of the triangle, that is, the primary colors R, G and B are pulled as three vertices. A fourth vertex can also be selected on the boundary of the triangle. According to one aspect of the present disclosure, a vertex P located in the middle of the line connecting points/vertices R and B can be selected. Since n≥1, it can be seen that P exists.

FIG7 illustrates an example of this conceptual extension, in which four vertices R, G, B and P are shown on the main triangle 702, and how these four vertices are mapped are shown. R can be pulled to the upper right corner of the unit square 704, G is pulled to the upper left corner of the unit square 704, B is pulled to the lower left corner of the unit square 704, and P is pulled to the lower right corner of the unit square 704. In addition, the remaining vertices are marked along the boundary of the main triangle (emphasized with bold points). All these m vertices together form the boundary of the convex polygon P. Because these m vertices are located on a straight line between any two of the four vertices R, G, B and P, they are mapped to the corresponding boundaries of the unit square 704, equidistant from each other. For example, one of the vertices between G and B (abbreviated as v∈GB) is mapped to the left edge of the unit square, and the three vertices between B and P are mapped to the lower edge of the unit square. In addition, the m vertices on the four edges of the unit square form a convex polygon P'. The remaining vertices are now mapped from P to P’ using mean value coordinates (MVC).

MVC can be expressed as a generalization of barycentric coordinates. Barycentric coordinates (mentioned above) are used for triangles, which are essentially polygons with three vertices. However, MVC is also applicable to polygons with any number of vertices. Therefore, the mapping of a vertex v inside polygon P to a new position inside polygon P' using _MVCλi for P is given by

where p _i and f(p _i ) represent the boundary vertices of polygons P and P'. Figure 7B shows the vertices v connected to the boundary points of the polygons (along with the corner definitions for MVC), which are sorted counterclockwise. The MVC λ _i of vertex v is given by

in

and α _i is the angle

A property of the mapping illustrated in Equation 4.7 is that it is also (among other things, for a T transform) reversible, so mapping a vertex from T' back to T will place the vertex back to its original position. Returning to Figure 7A, there is shown the generation of triangles based on mapping the interior vertices of the main triangle 502 using MVC. As one of ordinary skill in the art will appreciate, the smaller triangles closer to the lower right vertex P are more distorted than the triangles around vertex G. Because the human eye is more sensitive to small changes in green hue than to blue and red hues, it is desirable to have as little distortion as possible around vertex G. This is why the fourth vertex is chosen to be along this side of the main triangle 502. Furthermore, this is the longest side of the triangle, so choosing a vertex on another edge would likely result in greater overall distortion.

In summary, to implement a direct transform for the chromaticity conversion process, a mesh including the vertices and edges that form the smaller triangles is created within a main color space triangle (e.g., triangle 702). The main triangle is stretched by selecting four vertices and pulling these vertices towards the corners of a unit square (e.g., unit square 704). MVC can be used to map the internal vertices. The barycentric coordinates of the chromaticity values (u,v) for the smaller triangle Tr(a,b,c) map (u,v) to its corresponding triangle Tr(a,'b,'c'). Figure 7C shows an example of mapping equidistant points to a unit square using the direct transform described herein. Figure 7C shows how equidistant points (of the main triangle 702) are mapped to the unit square 704, and shows the possible distortion patterns that may occur. Distributing the vertices in an optimal manner produces a chromaticity that is as perceptually uniform as possible.

According to another embodiment, the chromaticity transform utilized in transforming the chromaticity values may be based on a least squares transform. Figure 8A shows an example transformation of chromaticity values in a color space triangle 802 to a unit square 804 (this is not an optimal example).

As with the direct transformation, the same four vertices (R, G, B, and P) are selected on the main triangle 802 and a mesh is generated. For this transformation, three sets of constraints are applied to the deformation of the mesh. Some constraints stretch the main triangle 802 as much as possible, while others try to preserve the shape of the smaller triangles. It's understandable that not all constraints can be perfectly satisfied, so least-squares transformations involve trade-offs.

These constraints can be expressed in terms of energies E ₁ , E ₂ and E ₃ , where the overall sum of the energies should be minimized. The smaller the energy E _i , the more the i -th set of constraints will be satisfied. The example of FIG8 utilizes a refinement value n = 3 and parameters α = β = γ = 1 (discussed below).

The first constraint can be called an anchor constraint, which specifies that the four vertices should be mapped to the four corners of the same unit square as in the direct transformation. This can be expressed mathematically as follows:

The first energy term _E1 can be expressed as:

The second constraint may be referred to as a boundary constraint, which specifies that vertices on the boundary of the main triangle 802 (highlighted with bold dots) should be placed somewhere on the boundary of the unit square 804, as according to a direct transformation. For example, the vertex v located between points B and G should be mapped to the left edge of the unit square 804 so that its u-coordinate, v _u, is equal to zero. For all boundary vertices, these constraints are given by:

And the energy _E2 is given by:

The first two sets of constraints (anchoring and bounding) try to stretch the main triangle in the same way as the direct transformation. However, with respect to the internal vertices of the main triangle, the third set of constraints (involving uniformity constraints) aims to make the smaller triangles (after the main triangle has been subdivided) maintain their shape as much as possible to prevent deformation. Each triangle Tr(a, b, c) has a circumcenter crm, which is a special point that is equidistant from the vertices a, b and c. Therefore, these three vertices lie on a circle around crm. The distance from the vertices of the mapped triangle Tr(a', b', c') to its new circumcenter should also be minimized. Therefore, these constraints should lead to similar triangles. For all triangles in the set of triangles T, the total energy E ₃ is equal to the sum of the squared distances from each vertex to the circumcenter of the triangle to which it is mapped.

The circumcenter crm can be expressed using the barycentric coordinates λ ₁ , λ ₂ , and λ ₃ by calculating:

Crm＝λ ₁ α′-λ ₁ b′-λ ₂ C′,

Equation 4.15

in

where represents the Euclidean distance between two vertices. To ensure that the sum of these three coordinates is equal to 1, they are normalized by dividing each of the three coordinates by the sum of the three values. The uniformity constraint does not require preserving the shape of the triangles in the mesh, but only requires keeping these triangles "rigid" (e.g., not easily deformed). A triangle can deform very badly and still meet the uniformity constraint because there are an infinite number of triangles with the same circumcenter. However, combined with the other two constraints, the triangle will remain rigid and maintain close to its original shape.

Ideally, the grid should be deformed so that it utilizes the entire area of the unit square (in order to provide a codec with a larger range of codewords). To achieve this, boundary and anchor constraints are defined, but at the same time, local deformations are minimized (e.g., by defining a uniformity constraint). This leads to a trade-off between the three energy terms E ₁ , E ₂ , and E ₃ derived in equations 4.11, 4.13, and 4.14 above. To obtain optimal results, the total energy sum E is minimized according to:

E＝αE ₁ +βE ₂ +γE ₃

Equation 4.17

Additional parameters α, β, and γ are introduced when solving Equation 4 to specify how much weight the energy term should have. Initially, all three parameters are equal to 1. For example, if it is desired to tighten the boundary constraints in E ₂ and want the boundary vertices to be closer to the edge of the unit square, the value of β can be increased. It should be recognized that all three sets of constraints are linear, which makes it easier to solve Equation 4.17 than if these constraints were nonlinear. Due to this property, Equation 4.17 can be reformulated in matrix notation, and the final problem to be solved is given by the system of linear equations:

This can be solved in a least squares sense. Here, x is a solution vector containing the values of the mapped vertices. For k vertices, x must contain the u and v coordinates of each vertex, so ^x∈R2k . This solution makes the distribution of vertices as perceptually uniform as possible. The mapped vertices v' are kept consistent in x in the following way:

There are a total of l individual constraints, with A∈Rl ^×2k and ^b∈Rl . Each value in x holds an individual constraint. Since each vertex is subject to multiple constraints l≥2k, the system of equations is overdetermined.

As considered herein, the resulting least squares transformed shape is shown in unit square 804. The distorted triangles cover most of the unit square, but not all of it. A trade-off can be considered when comparing this to a direct transformation with a least squares transform. By using a direct transformation, the integrity of the unit square can be affected, but the triangles in the mesh are severely distorted. In a least squares transform, the entire unit square is not necessarily utilized, but there is less distortion of the mapped triangles.

As described for Equation 4.17, the resulting transformation can be modified depending on which constraint the user wishes to optimize by changing the parameters α, β, and γ. If these values are not modified, the least squares transformation is considered non-optimal. For example, if the user wishes to cover a larger area of the unit square, the uniformity constraint can be abandoned. To do so, the values of α and β can be increased. An example of this transformation (a so-called optimized least squares transformation) is illustrated as unit square 806, where the parameters α = β = 3 and γ = 1 are set. When the non-optimized least squares transformation (shown as unit square 804) is compared to the optimized version (shown as unit square 806), it can be seen how the grid changes its shape.

Video or image content in WCG, such as P3 or BT.2020, has a wider color gamut and therefore requires more bits for storage and transmission. According to various embodiments, the system and method provide improved image quality at a fixed given bit rate, or provide the ability to select a desired image quality by selecting a certain image quality and reducing the bit rate. Optimization is achieved by utilizing a process called chromaticity conversion, which involves converting two chromaticity channels, U and V. If the YUV color space is used for encoding, the codewords provided to the codec are fewer than the codec can actually accept. If the codeword distribution of U and V is expanded, certain coding errors will become less prominent in the reconstructed signal, and due to the chromaticity conversion disclosed herein, the picture quality of the video and image content is improved. However, despite the expansion of the codeword distribution (which translates into providing the codec with more information), the compression is more efficient, thereby canceling the impact of the increased bit rate (for example, at the same bit rate, a 1/2dB or higher improvement can be achieved for color) resulting in color preservation and better overall color quality of the content.

Figure 9 is an example of computing components that can be used to implement various features of the systems and methods disclosed herein, such as the features and functionality of one or more aspects of the pre-processing portion 209 and the post-processing portion 223 of the video encoding pipeline 200 mentioned above.

As used herein, the term component can describe a given functional unit that can be executed according to one or more embodiments of the present invention. As used herein, a component can be implemented using any form of hardware, software, or a combination of the two. For example, one or more processors, controllers, application specific integrated circuits (ASICs), PLAs, PALs, complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), logic components, software programs, or other mechanisms that can be implemented to constitute a component. In an implementation, the various elements described herein can be implemented as discrete components or can be partially shared or fully shared within one or more components. That is, since the various characteristics and functions described herein are obvious to those of ordinary skill in the art after reading this specification, various features and functionality can be implemented in any given application and can be implemented in one or more separate or shared components in various combinations and arrangements. Although the various characteristics or elements of a function can be described or stated separately as independent components, it will be understood by those of ordinary skill in the art that these characteristics or functions can be shared in one or more general software and hardware elements, and this description does not require or imply the use of separate hardware or software components to implement such characteristics or functions.

Where a component or components of an application are implemented in whole or in part using software, in one embodiment, these software elements can be implemented to operate using a computing or processing component that is capable of performing the functions described above. FIG9 illustrates an example computing component. Various embodiments are described based on this example computing component 900. After reading this specification, it will be apparent to one of ordinary skill in the relevant art how to implement the application using other computing components or architectures.

Referring now to FIG9 , computing component 900 may represent computing or processing capabilities such as may be found within self-adjusting displays, desktops, laptops, notebook computers, and tablet computers; portable computing devices (tablet computers, personal digital assistants (PDAs), smartphones, mobile phones, palmtops, etc.); smart terminals or other devices with displays; servers; or any other type of specialized or general-purpose computing device that may be needed or suitable for a given application or environment. Computing component 900 may also represent computing capabilities embedded in or otherwise available to a given device. For example, computing components may be found in other electronic devices (e.g., navigation systems, portable computing devices, and other electronic devices that may include some form of processing capabilities).

The computing component 900 may include, for example, one or more processors, controllers, control components, or other processing devices (e.g., processor 904). The processor 904 may be implemented using a general-purpose or special-purpose processing engine, such as a microprocessor, controller, or other control logic. In the example shown, the processor 904 is connected to a bus 902, however, any communication medium may be used to facilitate interaction with other components of the computing component 900 or to communicate with the outside world.

The computing component 900 may also include one or more memory components, referred to herein simply as memory 908. For example, preferably random access memory (RAM) or other dynamic memory may be used to store information and instructions executed by the processor 904. The memory 908 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by the processor 904. The computing component 900 may similarly include a read-only memory ("ROM") or other static storage device coupled to the bus 902 for storing static information and instructions for the processor 904.

The computing component 900 may also include one or more forms of information storage mechanisms 910, which may include, for example, a media drive 912 and a storage unit interface 920. The media drive 912 may include a drive or other mechanism for supporting fixed or removable storage media 914. For example, a hard drive, a solid-state drive, a tape drive, an optical drive, a compact disc (CD) or digital video disc (DVD) drive (R or RW), or other removable or fixed media drive may be provided. Thus, the storage media 914 may include, for example, a hard drive, an integrated circuit assembly, a magnetic tape, a film, an optical disc, a CD or DVD, or other fixed or removable media that is written to or read by the media drive 912. As these examples illustrate, the storage media 914 may include computer-usable storage media having computer software or data stored therein.

In alternative embodiments, the information storage mechanism 910 may include other similar means for downloading computer programs or other instructions or data into the computing component 900. These means may include, for example, fixed or removable storage units 922 and interfaces 920. Examples of these storage units 922 and interfaces 920 may include program cartridges and cartridge interfaces, removable memory (e.g., flash memory or other removable memory components) and memory slots, PCMCIA slots and cards, and other fixed or removable storage units 922 and interfaces 920 that enable software and data to be transferred from the storage unit 922 to the computing component 900.

The computing component 900 may also include a communication interface 924. The communication interface 924 may be used to transfer software and data between the computing component 900 and external devices. Examples of the communication interface 924 may include a modem and soft modem, a network interface (e.g., Ethernet, a network interface card, Wimedia, IEEE 802.XX or other interface), a communication port (e.g., a USB port, an IR port, an RS232 port interface or other port), or other communication interfaces. The software and data transmitted through the communication interface 924 are typically carried by signals, which may be electrical signals, electromagnetic signals (including light), or other signals that can be transformed by a given communication structure 924. These signals can be provided to the communication interface 924 via a channel 928. This channel 928 may carry signals and may be implemented using a wired or wireless communication medium. Some examples of channels may include a telephone line, a cellular link, an RF link, an optical link, a network interface, a local area network or a wide area network, or other wired or wireless communication channels.

As used herein, the terms "computer program medium" and "computer-usable medium" are generally used to refer to either transient or non-transitory media, such as memory 908, storage unit interface 920, media 914, and channel 928. These or various other forms of computer program media or computer-usable media may be involved in delivering one or more sequences of one or more instructions to a processing device for execution. The instructions included on the media are generally referred to as "computer program code" or "computer program product" (which may be grouped in the form of a computer program or otherwise grouped). When the instructions are executed, the instructions may enable computing component 900 to perform the features or functions of the present application discussed herein.

Although discussed above in terms of various exemplary embodiments and implementations, it is understood that the various features, aspects, and functions described in one or more individual embodiments are not limited to application only in the specific embodiment for which they are described, but instead may be applied alone or in various combinations to one or more other embodiments of the present application, regardless of whether such embodiments are described and regardless of whether such features are implemented as part of the described embodiments. Therefore, the breadth and scope of the present application should not be limited to any of the above-described exemplary embodiments.

The terms and phrases used herein, and their variations, unless expressly stated otherwise, should be considered open ended and not restrictive. For example, the term "including" should be understood as "including, but not limited to" or similar expressions; the term "example" is used to provide an illustrative example of the term in question, rather than an exhaustive or limiting list thereof; the term "a" or "an" should be understood as "at least one," "one or more," or similar expressions; adjectives such as "conventional," "usual," "normal," "standard," "known," and terms of similar meaning should not be interpreted as limiting the described term to a given time period or to limiting the term to be available at a given time, but should be understood to include conventional, common, normal, and standard technologies that are available or known now or at any time in the future. Similarly, the technologies referred to herein are obvious or known to those of ordinary skill in the art, including those that are obvious or known to those of ordinary skill in the art now or at any time in the future.

The presence of expanded words or phrases such as "one or more," "at least," "but not limited to," or other similar phrases in certain examples should not imply that a narrower context is intended or required in instances where such expanded phrases are not present. The use of the term "component" does not imply that the aspects or functionality described or claimed as part of a component are configured in a common package. In fact, any or all aspects of a component (whether control logic or other components) can be combined in a single package or maintained separately and further distributed to multiple groups or packages or across multiple locations.

In addition, the various embodiments described herein are described using exemplary block diagrams, flow charts, and other illustrations. After reading this document, it will be apparent to those skilled in the art that the illustrated embodiments and their various alternatives can be implemented without being limited to the illustrated examples. For example, the block diagrams and their corresponding descriptions should not be construed as requiring a specific architecture or configuration.

Claims (9)

1. A method for perceptual color transformation in wide color gamut video coding, comprising: The additive color model signal is converted into a uniform color space signal with chromaticity and luminance components; The chromaticity components are transformed by applying a chromaticity ratio transformation to the chromaticity components; The luminance component is transformed separately from the chromaticity component by applying a luminance ratio transformation to the luminance component. The chromaticity and brightness components of the quantization transformation; Encode the quantified chromaticity and luminance components; The uniform color space signal includes a YUV color space signal, and the transformation of the chromaticity component includes subdividing the primary triangle representation of the chromaticity component in the YUV color space into a mesh comprising multiple secondary triangles, wherein each vertex of the primary triangle representation includes one of the primary colors of the additive color model; and The middle vertex is defined between two of the primary colors in the additive color model, and each of the vertices represented by the main triangle and the defined middle vertex are mapped to a rectangular representation of the chromaticity component in the YUV color space. 2. The method according to claim 1, wherein the additive color model signal includes a red-green-blue color model signal, i.e., an RGB color model signal. 3. The method of claim 1, wherein the rectangle represents a unit square. 4. The method of claim 1, wherein the respective vertices of each of the plurality of secondary triangles are mapped to the rectangular representation of the chromaticity component in the YUV color space using average coordinates. 5. The method of claim 4, wherein the value of each chromaticity component in the YUV color space is mapped to the rectangular representation of the chromaticity component using the centroid coordinates corresponding to the value of each chromaticity component. 6. The method of claim 1, further comprising mapping each of the vertices represented by the main triangle and the defined intermediate vertex to the quadrilateral representation of the chromaticity component in the YUV color space. 7. The method of claim 6, further comprising applying at least one of a plurality of constraints affecting each of the vertices of the primary triangle representation, each of the vertices of each of the plurality of secondary triangles, and the mapping of the defined intermediate vertices to the quadrilateral polygon representation. 8. The method of claim 7, further comprising adjusting a feature of at least one of the plurality of constraints to modify the chromaticity ratio transformation. 9. The method according to claim 1, wherein the brightness ratio transformation includes a photoelectron transfer function.