HK1217393B - Adaptive reshaping for layered coding of enhanced dynamic range signals - Google Patents

Description

Adaptive Shaping for Layered Coding of Signals with Enhanced Dynamic Range

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/836,044, filed June 17, 2013; U.S. Provisional Patent Application No. 61/951,914, filed March 12, 2014; and U.S. Provisional Patent Application No. 62/002,631, filed May 23, 2014, the entire contents of each of which are hereby incorporated by reference.

This application is also related to International Application No. PCT/US2014/031716, filed on March 25, 2014, the entire contents of which are incorporated herein by reference.

Technical Field

The present invention relates generally to video images. More particularly, embodiments of the present invention relate to adaptive reshaping of images with high or enhanced dynamic range for layered encoding and decoding.

Background Art

As used herein, the term "dynamic range" (DR) can relate to the ability of the human psychovisual system (HVS) to perceive a range of intensities (e.g., illuminance, brightness) in an image, e.g., from the darkest darks (black) to the brightest brights (white). In this sense, DR relates to "scene-referred" intensity. DR can also relate to the ability of a display device to adequately or approximately render a range of intensities of a particular breadth. In this sense, DR relates to "display-referred" intensity. Unless a particular meaning is expressly designated as having particular importance at any point in the description herein, it should be inferred that the term can be used in either sense (e.g., interchangeably).

As used herein, the term high dynamic range (HDR) relates to a DR breadth of some 14-15 orders of magnitude across the human visual system (HVS). For example, well-adapted humans with essentially normal vision (e.g., in a statistical, biometric, or ophthalmological sense) have an intensity range spanning approximately 15 orders of magnitude. Adapted humans can perceive dim light sources as small as just a few photons. However, these same humans can perceive the nearly painful glare of the midday sun in the desert, sea, or snow (or even glance at the sun, but briefly to prevent damage). This span is however available to "adapted" humans (e.g., those whose HVS has a time period for reset and adjustment).

In contrast, the DR over which humans can simultaneously perceive a wide range of intensities is somewhat truncated compared to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may be used individually or interchangeably to relate to the DR that can be simultaneously perceived by an HVS. As used herein, EDR may relate to a DR that spans 5 to 6 orders of magnitude. Thus, while it may be somewhat narrow compared to real-world reference HDR, EDR represents a wide DR breadth.

In practice, an image includes one or more color components (e.g., luma, Y, and chroma, Cb, and Cr), where each color component is represented with n bits of precision per pixel (e.g., n=8). Although luminance dynamic range and bit depth are not equivalent entities, they are generally related. Images where n≤8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images where n>8 can be considered enhanced dynamic range images. EDR and HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

A video signal can be characterized by multiple parameters such as bit depth, color space, color gamut, and resolution. Modern televisions and video playback devices (e.g., Blu-ray players) support a variety of resolutions, including standard definition (e.g., 720×480i) and high definition (HD) (e.g., 1920×1080p). Ultra High Definition (UHD) is a next-generation resolution format with a resolution of at least 3,840×2,160 (known as 4K UHD) and options up to 7,680×4320 (known as 8K UHD). Ultra High Definition may also be referred to as Ultra HD, UHDTV, or Ultra High Vision. As used herein, UHD means any resolution higher than HD resolution.

To support backward compatibility with legacy 8-bit playback devices and new HDR or UHD encoding and display technologies, a variety of formats can be used to deliver UHD and HDR (or EDR) video data from upstream devices to downstream devices. Given an EDR stream, some decoders can use a set of 8-bit layers to reconstruct an HD SDR or EDR version of the content. Advanced decoders can use a second set of layers encoded at a higher bit depth than traditional 8 bits to reconstruct a UHD EDR version of the content to present it on a more capable display. As the inventors herein have appreciated, improved techniques for encoding and distribution of EDR video are desirable.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art simply because they are included in this section. Similarly, problems identified with respect to one or more approaches should not be assumed to have been recognized in any prior art based on this section, unless otherwise indicated.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements, and in which:

FIG1A depicts an example framework for EDR layered encoding according to an embodiment of the present invention;

FIG1B depicts an example framework for EDR layered decoding according to an embodiment of the present invention;

FIG. 2 depicts an example EDR signal shaping function based on a power function, wherein the function parameter α is determined according to an embodiment of the present invention.

3 depicts an example process for determining an optimal exponent of a forward shaping function for an EDR input, according to an embodiment of the present invention;

FIG4 depicts an example process for determining a forward mapping for an EDR codeword according to an embodiment of the present invention;

FIG5 depicts an example of an intermediate mapping of input EDR codewords (v _c ) to block-based scaling factors (k(v _c )) according to an embodiment of the present invention;

FIG6 depicts an example mapping of input EDR codewords to final output shaping symbols according to an embodiment of the present invention;

FIG7 depicts an example of a reverse mapping calculated according to an embodiment of the present invention;

8A and 8B depict examples of chroma range scaling according to an embodiment of the present invention; and

FIG9 depicts an example of an encoding and decoding pipeline according to an embodiment of the invention.

DETAILED DESCRIPTION

This document describes an adaptive shaping technique for layered coding of video images with enhanced dynamic range (EDR). In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be practiced without these specific details. In other instances, well-known structures and devices are not described in detail in order to avoid unnecessarily enclosing, obscuring, or obfuscating the present invention.

Overview

The example embodiments described herein relate to adaptive reshaping of video images with high or enhanced dynamic range for efficient layered coding. An encoder receives an input enhanced dynamic range (EDR) image to be encoded in a layered representation. The input image can be gamma-encoded or perceptually coded using one or more bit depth formats not supported by available video encoders. The input image is remapped to one or more quantization layers to produce output codewords suitable for compression using the available video encoder.

In one embodiment, the remapping is based on a power function using a single function parameter. A technique is presented for determining the optimal function parameter based on computing a block-based complexity measure for each block in the input EDR image and then evaluating the amount of quantization-induced distortion in the quantized image.

In another embodiment, a block-based complexity metric (such as standard deviation) and a block-based linear quantization model (wherein a separate optimal quantizer scaler is determined for each image block) are used to generate the optimal mapping. The separate optimal scalers are combined to determine the envelope slope for each input codeword, and the optimal forward mapping function between the input codeword and the output codeword is determined based on the envelope slope. The reverse mapping function can be sent to the decoder as a lookup table, or it can be approximated using a piecewise polynomial approximation.

In another embodiment, given a reverse mapping lookup table, a piecewise polynomial approximation technique is used to approximate the inverse mapping function.

In the decoder, the encoded bitstream layers are decoded to produce decoded video layers, which are recombined to produce a single decoded signal. The decoded signal is then inversely mapped to produce an estimate of the original EDR signal sent from the encoder to the decoder, given the parameters received that define the encoder's shaping or mapping function.

In another embodiment, the chrominance color components of the input video signal may be translated so that the coordinates of the desired white point are shifted approximately to the center of the translated chrominance range.

An example framework for video signal shaping and hierarchical decomposition

Layered encoding and decoding

Existing displays and playback devices, such as HDTVs, set-top boxes, or Blu-ray players, typically support signals with resolutions up to 1080p HD (e.g., 1920×1080 at 60 frames per second). For consumer applications, such signals are now typically compressed in a luma-chroma color format (e.g., YCbCr or YUV 4:2:0 color format), where the chroma components typically have a lower resolution than the luma component, using a bit depth of 8 bits per pixel per color component. Due to the 8-bit depth and the corresponding low dynamic range, such signals are often referred to as signals with standard dynamic range (SDR). As new television standards, such as ultra-high definition (UHD), are being developed, it may be desirable to encode signals with enhanced resolution and/or enhanced dynamic range.

Video images are often gamma-encoded to compensate for the properties of the human visual system. For example, ITU-R Rec.2020 defines recommended gamma encoding for UHDTV signals. For EDR images, perceptual quantization (PQ) can provide a better alternative to traditional gamma encoding. The human visual system responds to increasing light levels in a very nonlinear way. The ability of humans to see a stimulus is affected by the brightness of the stimulus, the size of the stimulus, the spatial frequencies that make up the stimulus, and the brightness level to which the eye adapts at the specific moment a person views the stimulus. The perceptual quantizer function maps linear input grayscale levels to output grayscale levels that better match the contrast sensitivity threshold in the human visual system. An example of a PQ mapping function is described in PCT application serial number PCT/US2012/068212, filed December 6, 2012, by J.S. Miller et al., entitled “Perceptual luminance nonlinearity-based image data exchange across different display capabilities,” which is incorporated herein by reference in its entirety (hereinafter referred to as the '212 application). In this application, given a fixed stimulus size, for each luminance level (i.e., stimulus level), the minimum visible contrast step at that luminance level is selected based on the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model). Compared to a conventional gamma curve, which represents the response curve of a physical cathode ray tube (CRT) device and, coincidentally, may have a very rough resemblance to the way the human visual system responds, the PQ curve as defined in the '212 application uses a relatively simple function model to simulate the true visual response of the human visual system.

In U.S. Provisional Application Serial No. 61/805,388, filed on March 26, 2013, entitled “Encoding perceptually-quantized video content in multi-layer VDR coding,” which is hereby incorporated by reference in its entirety and will from now on be referred to as the '388 application (also filed as PCT/US2014/031716 on March 25, 2014), the inventors described image shaping techniques for efficient encoding and transmission of PQ-coded EDR image data using a two-layer encoder. The present application expands upon the '388 application by describing novel mapping or shaping techniques that can be applied to encoding EDR data using both a single-layer encoder and a multi-layer encoder.

1A depicts an example framework for EDR layered coding according to an embodiment of the present invention. The input signal (102) comprises a sequence of video frames having EDR pixel values that may have been gamma or PQ encoded. For a total of L coding layers, the system comprises at least one base layer (BL) video encoder (120-0) and may comprise one or more enhancement layer (EL) video encoders (120-1, 120-L-1), up to L-1. For example, for L=2, the system comprises a dual-layer encoder. The video encoders (120) may all be the same or different, thereby implementing any one of known or future coding formats for video compression, such as: MPEG-2, MPEG-4, part 2, H.264 (or AVC), H.265 (or HEVC), etc. In addition, a video encoder in one layer may support a different bit depth than a video encoder in another layer. For example, without loss of generality, an embodiment may comprise the following configuration:

• A single-layer HEVC encoder that supports a bit depth of at least 10 bits, but preferably 12 bits or more.

Dual-layer encoders, where both encoders can encode using the same format (e.g., H.264) and support the same or different bit depths (e.g., 8-bit and 10-bit)

A dual-layer encoder, where the two encoders can encode using different coding formats, and each can support a different bit depth (e.g., 8 bits and 10 bits or more)

• A multi-layered encoder, wherein at least one encoder is an 8-bit MPEG-2 encoder and at least one other encoder is a HEVC or H.264 encoder.

The video encoder (120) may be implemented entirely by a single processor or by one or more processors.

According to an embodiment, a signal shaping module (110) quantizes an input EDR signal (denoted as v) into a signal s (112) that better conforms to characteristics (such as a maximum supported bit depth) of a video encoder (120). As used herein, the terms shaping, quantization, and (forward) mapping denote equivalent functions of mapping an input signal from a first dynamic range to an output signal of a second dynamic range that is typically lower than the first dynamic range, and may be used interchangeably.

Let _B1 denote the bit depth used by the layer l video encoder (120-1, l = 0, 1, 2, ..., L-1), then each layer can support up to _N1 = ^2B1 input codewords for a total of _NT = _N0 + _N1 + ... + _NL-1 codewords. For example, for L = 1 (single layer) and _B0 = 10, there are ²¹⁰ = 1024 quantization codewords. For L = 2 (dual layer) and _B0 = _B1 = 8, there are ²⁸ + ²⁸ = 512 quantization codewords. When L = 2 and _B10 = 10, _B1 = 8, there are ²¹⁰ + ²⁸ = 1280 quantization codewords. Thus, the system can accommodate any combination of video coding standards, each operating at its own bit depth.

Let s = f(v) denote the signal shaping/quantization function (110). Examples of such functions will be described in more detail later. Parameters identifying the shaping function may be included in a metadata signal (119). In some embodiments, the metadata (119) may be encoded by a metadata encoder (125), and the encoded metadata (127) may be signaled to a decoder (such as the decoder depicted in FIG1B ) for appropriate inverse quantization and decoding. In another embodiment, the signal shaping (110) may include a family of signal shaping functions such that a separate shaping function is used for one or more layers, or one or more chroma components within a layer. For example, in an embodiment, the signal shaping function for the base layer (l = 0) may be a linear function, while the signal shaping function for the first enhancement layer (l = 1) may include a nonlinear function or a piecewise linear function.

Layer decomposition

In an embodiment, let the pixel values of the quantized signal s (112) be divided into L segments defined by segment boundaries { _pi , i=0, 1, ..., L}, where _p0 generally represents the minimum possible value for s (e.g., _p0 =0), and

For example, for L = 1, p ₀ = 0 and p ₁ = N ₀ . This module will encode all codewords into the base layer.

For L=2, p ₀ =0, p ₁ =N ₀ , p ₂ =N ₀ +N ₁ . In an embodiment, pixels with codewords between {p ₀ ,p ₁ } will be encoded in layer 0, and pixels with codewords between {p ₁ ,p ₂ } will be encoded in layer 1. In general, given L layers, for each layer l, the s _l pixels at that layer are encoded as:

s _l =Clip3(s, p _l , p _l+1 -1)-p _l , l＝0,1,2,…,L-1, (2)

Here, d=Clip3(s,a,b) represents a clipping function, where if a≤s≤b, then d=s, if s<a, then d=a, and if s>b, then d=b.

After the layer decomposition (115), in an embodiment, each layer s _l (117-l, l = 0, 1, ..., L-1) can be independently encoded by a video encoder (120) to produce a compressed bitstream (122). As discussed in the '388 application, in some embodiments, the system depicted in FIG1A can be modified to also allow inter-layer prediction. In such an embodiment, a predictor can be used to estimate the pixel values of layer l based on the pixel values of layer l-1; then, instead of encoding the pixel values of layer l directly, the residual between the actual value and the predicted value is simply encoded and transmitted.

In some embodiments, the encoded bitstream (122), encoded metadata (127), and other data (e.g., audio data) may be multiplexed into a single bitstream and sent to a decoder as a single multiplexed bitstream (not shown).

FIG1B depicts an example framework for EDR layered decoding according to an embodiment of the present invention. As depicted in FIG1B , after demultiplexing of the received bitstream, which may combine audio, video, and auxiliary data (not shown), each of the received encoded bitstreams (122) is fed to a video decoder array (130). The decoder (130) corresponds to the encoder (120) and produces one or more of the decoded video signals (132). Using a signal inverse shaping and layer synthesizer (140) unit, the received layered signals are combined and inversely shaped to produce a signal (142) that represents an estimate of the original EDR signal (102). In an embodiment, the output EDR signal (142) may be generated as follows:

where f ^-1 () represents the inverse (or a close approximation of the inverse) of the signal shaping function (110) and represents the reconstructed layer signal (132), which represents a very close approximation of the original s _l signal (117). As depicted in FIG. 1B , there is no inter-layer prediction between the received layers; however, as is known in the art of video coding, the system can be easily extended to decoders in which the received residual signal and inter-layer prediction are used to generate the signal.

EDR signal shaping using power function

As described in the '388 application, for a PQ coded signal, in an embodiment, the signal shaping function (110) may be expressed as:

Where v _L and v _H represent the minimum and maximum values in the color channel under consideration of the input EDR signal (102), and c _L and c _H represent the corresponding minimum and maximum output values. For example, in an embodiment, as defined in equation (1), c _L = 0 and c _H = p _L -1. The value of α is constant, but can be modified and varied based on each frame, each scene, or other suitable criteria. Figure 2 depicts an example of the power shaping function of equation (4) for α < 1. In an embodiment, if the input (112) is PQ encoded, then α > 1, otherwise, if the input (112) is gamma encoded, then α < 1.

In the '388 application, a method is described for determining an optimal value for α using a block complexity metric ( _MEDR ) based on the standard deviation of pixels within a block or the difference between the minimum and maximum pixel values within a block. In addition to these metrics, other complexity metrics can be applied based on the spatial or frequency characteristics of the block. For example, _MEDR can correspond to the variance of the pixels in the block, the DC value of the block, or another function of its DCT coefficients or pixel values.

Consider a video scene consisting of F video frames, each of which is divided into N blocks (e.g., each block is 8×8 pixels). The image blocks may overlap, or in a preferred embodiment, do not overlap. FIG3 depicts the process of computing the optimal α based on a generalized block complexity measure _MEDR (j,n) (e.g., without loss of generality, the standard deviation of the pixel values within a block).

As depicted in FIG3 , after step ( 305 ) in which M _EDR ( j, n ) is calculated for each block in all frames in the scene, in step ( 310 ), a set Φ _j is constructed that includes all image blocks that meet a certain criterion (e.g., M _EDR ( j, n ) > T, where T is a pre _- specified threshold, such as 0).

Steps (315), (320) and (325) include a loop (327) of calculations for various values of α _j within a predetermined range (e.g., MIN_α ≤ _{α j} ≤ MAX_α). For example, initially α _j may be set equal to 1 and may then be increased or decreased depending on how the original EDR video data is encoded. For example, it would be increased for PQ encoded data and decreased for gamma encoded data. In step (320), the input EDR data is quantized using the given α _j and equation (4), and a new metric M _LD (j, n, α _j ) may be calculated for each quantized block. In some embodiments, the complexity metric M _LD may be the same as the complexity metric M _EDR . In some other embodiments, the two complexity metrics may be different. The more the input EDR data is quantized, the more the characteristics of the quantized signal (112) will change. Ideally, quantization (110) should distort the input as little as possible. In step (325), a measure of the distortion due to quantization may be applied to identify whether the selected α _j is optimal. For example, in an embodiment, if

Then α _j can be selected as the optimal one, where T _σ is another predetermined threshold (eg, T _σ =0).

After all blocks in the scene have been quantized, the overall optimal alpha value is selected in step (330). For example, in an embodiment, for α>1, the overall optimal α is selected as the smallest of all the optimal _αj values. Similarly, for α<1, the overall optimal α is selected as the largest of all the optimal _αj values.

In some embodiments, to adjust for the quantization effects due to lossy compression by the video encoder (120), the overall optimal alpha can be further adjusted (e.g., α=α+Δα, where Δ is negative when α>1 and positive when α<1). The same parameter optimization process can also be easily extended to other linear or nonlinear quantization and shaping functions characterized by more than one function parameter.

Table 1 provides in pseudocode an example algorithm for shaping gamma-encoded values based on the process depicted in FIG. 3 , where a desired α<1, according to an embodiment.

Table 1 — Method for calculating optimal α for gamma-coded EDR signals

In the decoder, the inverse shaping operation (140) can be expressed as:

In some embodiments, the power function of equation (4) can be expressed as a piecewise linear polynomial, a piecewise 2-d or 3-d order polynomial, or a piecewise B-spline. In such an implementation, it is recommended that smoothness and monotonicity constraints between segments should be enforced to avoid quantization-related artifacts. Search methods similar to those described earlier or in the '388 application can then be applied.

Block-adaptive shaping function

Consider again a video scene comprising F video frames, each frame divided into N blocks (e.g., each block being 8×8 pixels). The image blocks may overlap, or in a preferred embodiment, do not overlap. FIG4 depicts an example data flow for mapping an input EDR codeword (102) to a shaped output value (112), according to an embodiment.

As previously mentioned, a block-based complexity measure ( _MEDR ) may be defined. In an embodiment, in step (405), without loss of generality, a complexity measure is considered that is calculated based on the standard deviation (std) of the pixels in the block. Note that checking whether the standard deviation of block n in frame j (j=1, 2, ..., F) is zero (e.g., _MEDR (j,n)= _stdjn =0) is equivalent to checking whether the difference between the maximum value in the block (e.g., B(j,n)) and the minimum value in the block (e.g., A(j,n)) is zero.

Assuming that the shaping function (110) is constructed by a piecewise linear line, for the input _vi∈ [A(j,n) B(j,n)], the local quantizer can be expressed as:

where k(j,n) is a scaling factor that adjusts the slope of the quantizer at the nth block in the jth frame.

In step (410), let Φ denote the set of all blocks whose block metrics satisfy a certain criterion. For example, let Φ denote the set of all blocks with non-zero standard deviation before quantization, or

φ={(j,n)|B(j,n)-A(j,n)>0}, (7)

In an embodiment, given a threshold _Tth (where, without loss of generality, _Tth ≥ 1), a minimum pixel value and a maximum pixel value of a given block, in step (415), the optimal k(j,n) can be derived as follows:

Given the data {A(j,n),B(j,n),k(j,n)}, the triplet value reveals that within the segment [A(j,n),B(j,n)], the quantizer should have a slope of at least k(j,n). Assuming that a specific EDR code (e.g., v _c ) can belong to multiple [A(j,n),B(j,n)] segments, for the EDR codeword v _c , it is necessary to determine the maximum slope for each codeword to satisfy all blocks.

Let θ(v _c ) denote the set of all segments in all blocks covering codeword v _c , or

θ(v _c )={(j,n)|A(j,n)≤v _c ≤B(j,n), (j,n)∈φ}. (9)

Then, in step (420), the required slope at codeword v _c can be determined as the envelope of all optimal slopes within those blocks belonging to the set θ(v _c ), or

In step (425), let the sum of all such envelope slopes be expressed as:

Then, for each v _c codeword, without loss of generality, in step (430), the cumulative slope function can be defined as follows:

To ensure that all codewords are mapped within the bounds of [c _L ,c _H ], the following equation can be used to calculate the mapping from v _c codewords to s _i values:

Given equation (13), a forward mapping lookup table (e.g., ) can be used to calculate the mapping between the input v _c values and the output s _i values. In an embodiment, the table can be stored in the data or sent to the decoder as part of the image metadata (119) so that the decoder can reconstruct the inverse mapping process.

In an example embodiment, Table 2 summarizes the mapping process depicted in FIG. 4 using pseudocode.

Table 2

In some embodiments, an alternative function may be used to calculate the cumulative slope function K(v _c ) in equation (12). For example, the k(v _c ) values may be filtered or weighted before being _summed as in the following equation:

Here, _wi represents filter coefficients or predetermined weights of a filter having (2u+1) filter taps (eg, u=2 and the filter coefficients correspond to those of a low-pass filter).

FIG5 depicts an example plot of _vc codeword versus envelope slope k( _vc ) value for a test sequence of frames given a [ _cL , _cH ] range of [0, 255].

Given the data depicted in Figure 5, Figure 6 depicts an example of v _c pair mapping.

Reverse plastic surgery

In the decoder, given the value of equation (13), the inverse quantizer or shaping function (140) can be determined as follows:

For each decoded codeword

but

In other words, for a given codeword in the quantized domain, the corresponding estimated EDR codeword is constructed by first grouping all pixels with quantized values, finding the corresponding EDR codeword, and then averaging all collected EDR codewords. From equation (16), a backward lookup table can be constructed and stored in the data or sent to the decoder, for example, as part of the metadata (119).

Given the data depicted in FIG6 , FIG7 depicts an example of mapping or inverse reshaping ( 140 ).

In an embodiment, the mapping defined by the relationship can be sent to the decoder using metadata (119, 127). Such an approach may be too expensive in terms of bit rate overhead. For example, for 8-bit data, the lookup table may include 255 entries to be sent each time there is a scene change. In other embodiments, the inverse mapping can be converted into a piecewise polynomial approximation. Such polynomials can typically include first-order polynomials and second-order polynomials, although higher-order polynomials or B-splines can also be used. The number of polynomials that approximate the LUT for a certain layer l (l = 0, 1, ..., L-1) can vary depending on the available bandwidth and processing complexity. In an embodiment, the base layer uses up to 8 segments, while the enhancement layer uses a single segment.

Table 3 depicts an example algorithm for approximating a decoder LUT table using a second order polynomial, according to an embodiment.

Table 3 - LUT approximation using a 2d-order polynomial

As depicted in Table 3, in an embodiment, the inputs to the approximation process include: the original lookup table (e.g., calculated using Equation (16)), the acceptable error tolerance between the values in the LUT and those produced by the polynomial approximation, the number of available codewords, and their first codeword values (see Equation (1)). The output may include the endpoints (also known as pivot points) of each polynomial and the polynomial coefficients.

Starting from the first pivot point, the algorithm attempts to fit the largest possible range of available codewords using a quadratic polynomial without loss of generality. Any known polynomial fitting algorithm can be used, such as mean square error polynomial fitting, etc.

When the calculated maximum error exceeds the input tolerance, the parameters of the optimal polynomial are stored and a search for a new polynomial begins until the entire LUT table is mapped.

In some embodiments, the number of polynomials that can be used to approximate the LUT can be constrained to a fixed value, say eight. In this case, a higher error tolerance can be incorporated into the algorithm.

The method of Table 3 can also be easily modified to accommodate other approximation functions, such as higher order polynomials, B-splines, or a combination of approximation functions.

Encoding in perceptually uniform color spaces

Video signals are typically presented in the familiar RGB color space; however, most video compression standards (such as MPEG-2, H.264 (AVC), H.265 (HEVC), etc.) have been optimized to operate in opposing color spaces (such as YCbCr or YUV). These color spaces are sufficient for encoding and transmitting 8-10-bit standard dynamic range (SDR) video, but they may not be the most efficient when encoding and transmitting EDR video from the perspective of the number of bits per pixel required. For example, Lu'v' and Log(L)u'v' color spaces have also been proposed in the past.

As the inventors have appreciated, encoding of signals in a perceptually uniform space may benefit from additional processing of the u' and v' chrominance data prior to processing by the video codec. For example, in an embodiment, such processing may be performed on the input signal (102) as part of a signal shaping process (110) in the encoder.

White point conversion

In an embodiment, the conversion from linear XYZ to Luma u'v' color space may include the following steps:

a) Define the coordinates of the white point (e.g., D65)

b) Solve for Luma = f(Y), and

c) Solve for u’ and v’ from X, Y and Z

As used herein, the function f(Y) represents any luma-related function, such as L (or L'), log(L), etc. In a preferred embodiment, f(Y) may represent a perceptual quantization (PQ) mapping function as described in the '212 application.

In an embodiment, the white point may be defined as D65 (6500K) with u' and v' coordinates:

Du=d65u=0.1978300066428;

Dv=d65v=0.4683199949388;

In an embodiment, u' and v' may be derived as follows:

If (X+15Y+3Z)≠0, then

And if (X+15Y+3Z)=0, then

u'＝Du (17c)

v'＝Dv (17d)

Reverse operations include:

a) Define the coordinates of the white point (e.g., D65)

b) Solve for Y = f ^-1 (Luma)

c) Solve for X and Z from u’ and v’

For example, in an embodiment using a perceptual quantization function according to the '212 application, a corresponding inverse PQ mapping may be applied to generate Y pixel values.

In an embodiment, X and Z may be derived as follows:

If v'≠0, then

And if v' = 0, then X = Z = Y. (18c)

FIG8A depicts a conventional mapping of a white point (805) (e.g., D65) in the u'v' chrominance space. As depicted in FIG8A , the range of u' and v' chrominance values is approximately (0, 0.623) for u' and (0, 0.587) for v'. As depicted in FIG8A , the D65 white point is not centered in the u'v' signal representation. This can result in color shifts after subsampling and upsampling the chrominance components to convert between the original 4:4:4 color format and the 4:2:0 or 4:2:2 color formats commonly used in video encoding. To mitigate such color shifts, it is proposed to apply a transformation function to the chrominance values. In one embodiment, the transformation function shifts the white point approximately to the center of the transformed u'v'; however, in cases where it may be desirable to see a reduction in chrominance errors, the white point can be transformed to any other color value. For example, if the transformed values _u′t and _v′t are in the range (0, 1), the following mapping can be applied:

u′ _t =(u′-Du)a ₁ +b ₁ , (19a)

v′ _t =(v′-Dv)a ₂ +b ₂ , (19b)

Where Du and Dv represent the original u' and v' coordinates of the selected white point, (b ₁ , b ₂ ) determine the coordinates of the desired location of the white point in the converted color space, and a _i (i=1,2) are constants calculated based on the desired conversion point and the minimum and maximum values of u' and v'. In one embodiment, the conversion parameters (e.g., a ₁ and a ₂ ) can be fixed for the entire video sequence. In another embodiment, the conversion parameters can be calculated on a per-scene or per-frame basis to take advantage of variations in the chromaticity range of the incoming content.

FIG8B depicts the mapping of the white point (805) in the transformed chromaticity space according to an example embodiment. In FIG8B , the original u' and v' chromaticity values are transformed so that the selected point (e.g., D65) is approximately at the center (0.5, 0, 5) of the transformed chromaticity space. For example, if the transformed values _u't and _v't are in (0, 1), then for _b1 = _b2 = 0.5, in one embodiment, the following mapping may be applied:

u′ _t = (u′-Du)1.175+0.5, (20a)

v′ _t = (v′-Dv)1.105+0.5, (20b)

Where Du and Dv represent the u' and v' coordinates of the chosen white point. This conversion causes chromaticity errors to appear as undersaturation rather than hue shifts. Those skilled in the art will appreciate that nonlinear functions can also be applied to the u' and v' chromaticity values to achieve the same conversion. Such nonlinear functions can assign greater precision to near-neutral colors to further reduce the visibility of color errors caused by encoding and quantization.

Reduce chromatic entropy

If the u' and v' pixel components are multiplied by a function of the luminosity, the visibility of the chrominance details can be further improved. For example, in an embodiment, the converted chrominance values can be derived as follows:

u′ _t =g(Luma)(u′-Du)a ₁ +b ₁ , (21a)

v′ _t =g(Luma)(v′-Dv)a ₂ +b ₂ , (21b)

Wherein, g(Luma) represents a function of the luminance channel. In an embodiment, g(Luma)=Luma.

In the decoder, the incoming signal can be represented as Luma _{u't v't} . In many applications, the signal must be converted back to XYZ, RGB, or some other color space before further processing. In an example embodiment, the color conversion process from Luma _{u't v't} to XYZ may include the following steps:

a) Undo brightness encoding

Y＝f ^-1 (Luma)

b) Unscale the range of u' _t and v' _t values to restore u' and v'

c) Use equation (18) to restore X and Z

In some embodiments, the Luma, _u't , and _v't components of the incoming signal may be normalized to the (0, 1) range before any color transformation. In some embodiments, equations (17)-(21) may be implemented using a combination of lookup tables, multiplications, and additions. For example, in an embodiment, let

Y＝f ^-1 (Luma),

B＝3u′,

C = 20v', and

D＝1/(4v’)

Denote the output of the three lookup tables with Luma, u', and v' as their inputs. Then, from equation (18), the X and Z values can be calculated using four multiplications and two additions as follows:

Z＝(Y*D)*(12-B-C),

and

X＝(Y*D)*(3*B).

For example, in an embodiment, for a 10-bit encoded signal, each LUT may have 1024 entries, with each entry being at a sufficiently high precision (eg, 32 bits) for the target application.

FIG9 depicts an example of an encoding and decoding pipeline according to an embodiment of the present invention. The input signal (902) may be in RGB 4:4:4 or any other suitable color format. In step (910), the signal (902) is converted to a perceptual Luma u'v' format, for example by using a perceptual quantization (PQ) mapping for luma values and equation (17) for chroma values. In step (915), a transform, such as that described in equations (19)-(21), is applied to convert the original u'v' chroma values to transformed chroma values _u't and _v't such that _{the white point is approximately placed at the center of the transformed chroma space. The color transformed and transformed Luma u't v't 4} :4:4 signal (e.g., 917) may be color subsampled (not shown) to a 4:2:0 or 4:2:2 format before being encoded by a video encoder (920). The video encoder (920) may include the signal shaping (110) and layer decomposition (115) processes described above. At the receiver, a video decoder (930) produces a decoded signal (932). The video decoder (930) may include an inverse signal shaping and layer synthesizer (140). After optional color upsampling (e.g., from 4:2:0 to 4:4:4), an inverse chroma conversion step (935) may convert the Luma _{u't v't} signal (932) back into a Luma u'v' signal (937) by reversing the conversion operation in (915). Finally, the Luma u'v' signal (937) may be converted to an output signal (942) in RGB or other appropriate color space for display or further processing.

Encoding in perceptually quantized IPT color space

White point conversion can also be applied to other color spaces, such as the IPT color space or the IPT-PQ color space, which appear to be perceptually quantized color spaces ideally suited for encoding video signals with enhanced or high dynamic range. The IPT-PQ color space was first described in PCT application PCT/US2014/016304, filed February 13, 2014, by R. Atkins et al., entitled “Display management for high dynamic range video,” which is incorporated herein by reference in its entirety.

The IPT color space, described in "Development and testing of a color space (ipt) with improved hue uniformity" by F. Ebner and M.D. Fairchild, in Proc. ^6th Color Imaging Conference: Color Science, Systems, and Applications, IS&T, Scottsdale, Arizona, Nov. 1998, pp. 8-13 (hereinafter referred to as the Ebner paper), as incorporated herein by reference in its entirety, is a model of color differences between cones in the human visual system. In this sense, it is like the YCbCr or CIE-Lab color spaces; however, some scientific studies have shown that it mimics human visual processing better than these spaces. Like CIE-Lab, IPT is a normalized space to some reference luminance. In embodiments, the normalization may be based on the maximum luminance of the target display.

As used herein, the term "PQ" refers to perceptual quantization. The human visual system responds to increasing light levels in a highly nonlinear manner. A person's ability to see a stimulus is affected by the brightness of the stimulus, the size of the stimulus, the spatial frequencies that comprise the stimulus, and the brightness level to which the eye adapts at the particular moment a person views the stimulus. In a preferred embodiment, a perceptual quantizer function maps linear input grayscale levels to output grayscale levels that better match the contrast sensitivity threshold in the human visual system. An example of a PQ mapping function is described in the '212 application, in which, given a fixed stimulus size, for each brightness level (i.e., stimulus level), the minimum visible contrast step size at that brightness level is selected based on the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model). Compared to the traditional gamma curve, which represents the response curve of a physical cathode ray tube (CRT) device and, coincidentally, may bear a very rough resemblance to the way the human visual system responds, the PQ curve as defined in the '212 application uses a relatively simple functional model to simulate the true visual response of the human visual system.

Table 1 describes the calculation of the perceptual curve EOTF for converting digital video code values to absolute linear luminance levels at one point on the display. Also included is the inverse OETF calculation for converting absolute linear luminance to digital code values.

Table 1

Example Equation Definition

D = Perceptual curve digital code value, SDI-legal unsigned integer, 10 or 12 bits

b = the number of bits per component in the digital signal representation, 10 or 12

V = normalized perceptual curve signal value, 0≤V≤1

Y = normalized brightness value, 0≤Y≤1

L = absolute luminance value, 0≤L≤10,000cd/ ^m2

Example EOTF decoding equation:

Example OETF encoding equation:

Example constants:

Notes:

1. The operator INT returns a value of 0 for a decimal part in the range of 0 to 0.4999… and a value of +1 for a decimal part in the range of 0.5 to 0.9999…, that is, it rounds up a decimal greater than 0.5.

2. All constants are defined as multiples of 12-bit rational numbers to avoid rounding problems.

3. The R, G or B signal component will be calculated in the same manner as the Y signal component above.

Converting a signal to the IPT-PQ color space may include the following steps:

a) Convert the signal from the input color space (e.g., RGB or YCbCr) to XYZ

b) Convert the signal from XYZ to IPT-PQ as follows:

a. Apply 3×3XYZ to the LMS matrix to convert the signal from XYZ to LMS

b. Convert each color component of the LMS signal into a perceptually quantized LMS signal (L'M'S' or LMS-PQ) (e.g., by applying equation (t2))

c. Apply 3×3 LMS to the IPT matrix to convert the LMS-PQ signal to the IPT-PQ color space

Examples of 3×3XYZ to LMS and L'M'S' (or LMS-PQ) to IPT conversion metrics can be found in the Ebner paper. Assuming that the chrominance components (e.g., P' and T') of the IPT-PQ signal are in the range (-0.5, 0.5), a bias α (e.g., α = 0.5) can be added to make the range of the chrominance components basically in the range (0, 1), for example:

P′＝P′+a (22a)

T′＝T′+a (22b)

The inverse color operation may include the following steps:

a) Subtract any bias added to the chroma components

b) Applying the 3×3 I’P’T’ to the LMS transfer matrix to convert from IPT-PQ to LMS-PQ

c) Apply the inverse PQ function to convert from LMS-PQ to LMS (e.g., by using equation (t1))

d) applying 3×3 LMS to the XYZ transform to convert from LMS to XRZ, and

e) Convert from XYZ to the selected device-dependent color space (e.g., RGB or YCbCr).

In practice, the color conversion step during encoding and/or decoding may be performed using a pre-computed 1-D lookup table (LUT).

Reduce chromatic entropy

As previously mentioned, if the P' and T' pixel components are multiplied by a function of luminosity (e.g., I'), the visibility of chrominance details can be further improved. For example, in an embodiment, the converted chrominance values can be derived as follows:

P′ _t =g(I′)(P′-a)+a, (23a)

T′ _t =g(I′)(T′-a)+a, (23b)

Wherein, g(I') represents a linear or nonlinear function of the luminance channel (I'). In an embodiment, g(I') = I'.

Example Computer System Implementation

Embodiments of the present invention may be implemented using a computer system, a system configured in electronic circuit systems and components, an integrated circuit (IC) device (such as a microcontroller), a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete-time or digital signal processor (DSP), an application-specific IC (ASIC), and/or an apparatus comprising one or more of such systems, devices, or components. The computer and/or IC may execute, control, or perform instructions related to adaptive shaping techniques for layered coding of video images with enhanced dynamic range (EDR), such as those described herein. The computer and/or IC may calculate any of the various parameters or values associated with the adaptive shaping process described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

Certain implementations of the present invention include a computer processor executing software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. can implement methods related to adaptive shaping techniques for layered coding of video images with enhanced dynamic range (EDR) by executing software instructions in a program memory accessible to the processor, as described above. The present invention can also be provided in the form of a program product. The program product can include any medium that carries a set of computer-readable signals that, when executed by a data processor, includes instructions that cause the data processor to perform the methods of the present invention. Program products according to the present invention can take any of a variety of forms. The program product can include, for example, physical media such as magnetic data storage media (including floppy disks, hard drives), optical data storage media (including CD ROMs, DVDs), electronic data storage media (including ROMs, flash RAM), etc. The computer-readable signals on the program product can optionally be compressed or encrypted.

Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, discussion of the component (including discussion of "means") should be interpreted to include any component that is an equivalent to the component and performs the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structure and perform the functions in the illustrated example embodiments of the invention.

Equivalents, extensions, replacements, and others

Thus described are example embodiments relating to adaptive reshaping techniques for layered coding of video images with enhanced dynamic range (EDR). In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary between different implementations. Thus, the sole and exclusive indicator of what is the invention, and what the applicants intend the invention to be, is the set of claims in the specific form in which this application issues, and in which such claims issue, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Accordingly, no limitation, element, property, feature, advantage, or attribute that is not expressly recited in a claim shall limit the scope of such claim in any way. The specification and drawings are, therefore, to be regarded in an illustrative, rather than a restrictive, sense.

Claims (21)

1. A method for encoding a sequence of input augmented dynamic range (EDR) images comprising image patches, the method comprising: Receive the sequence of the input enhanced dynamic range (EDR) images; For at least one input image patch in the sequence of input EDR images, one or more block complexity measures are computed, the block complexity measure representing the change in pixel value; Construct a first set of image patches, the first set including image patches whose calculated block complexity metric satisfies a predetermined criterion; For each block in the first set of image blocks, an optimal slope is determined according to a slope generation function for a linear quantization model, the optimal slope representing the minimum slope of the linear quantization model for the block; For each input codeword in the sequence of the input EDR images: A second set of image blocks is constructed, the second set including blocks belonging to the first set of image blocks, wherein the input codeword is within the minimum and maximum pixel values of the image blocks; and The envelope slope of the input codeword is generated, and the envelope slope is calculated using the largest optimal slope among the optimal slopes of the blocks in the second set of image blocks; Calculate the sum of the envelope slopes of all codewords; and For each input codeword: A cumulative slope is generated, wherein the cumulative slope is equal to the sum of the envelope slopes up to and including the input codeword; and A mapping function is generated between the input codeword and the output codeword, which is calculated from the cumulative slope of the codeword and the sum of the envelope slopes of all codewords. 2. The method according to claim 1, further comprising: The mapping function is applied to the input EDR image to produce a reshaped image; Decompose the reshaped image into one or more layers; and One or more video encoders are used to encode the one or more layers. 3. The method of claim 2, wherein, given integer pixels s and L layers, the decomposition step includes mapping pixel s to pixel values s _{and L} : s _l =Clip3(s, p _l , p _l+1 -1)-p _l , Where l = 0, 1, ..., L-1, represents one of the L layers, Clip3() is a clipping function that clips the integer pixel s between the values of p _l and p _l+1 -1, and p _l represents the minimum pixel value of the integer sequence at layer l. 4. The method according to claim 3, wherein _p0 = 0, and For i = 1, ..., L, Where N_{_j} represents the number of codewords available in the video encoder at level j. 5. The method of claim 1, wherein the linear quantization model comprises a function represented by the following formula: Where _vL and _νH represent the minimum and maximum codeword values in the EDR input sequence, _cL and _cH represent the minimum and maximum output codeword values, k(j,n) represents the optimal slope of the nth block in the jth frame of the EDR input sequence, _vi represents the input codeword, and _si represents the corresponding output codeword. 6. The method of claim 5, wherein the optimal slope k(j, n) is generated by calculating the following formula: Where T_{_th} is a predetermined threshold, B(j,n) represents the maximum pixel value in block n of frame j, and A(j,n) represents the minimum pixel value in block n of frame j. 7. The method of claim 1, wherein calculating the block complexity metric of an image block comprises calculating the difference between the maximum and minimum pixel values in the image block, and wherein the first set of image blocks comprises all image blocks whose block complexity metric is greater than 0. 8. The method of claim 1, wherein generating the envelope slope k( _vc ) of the codeword _vc comprises calculating: Where k(j,n) represents the optimal slope of block n in frame j of the sequence of the input EDR image, and θ( _vc ) represents the second set of the image blocks. 9. The method according to claim 1, wherein calculating the cumulative slope K( _{v_c} ) of the codeword _{v_c} comprises calculating: in, k(i) represents the envelope slope of the i-th input EDR codeword. 10. The method of claim 1, wherein generating the mapping function between the input codeword and the output codeword comprises calculating: in, K(v _c ) represents the cumulative slope of codeword v _c , k represents the sum of the envelope slopes of all codewords, and c _L and c _H represent the minimum and maximum output code values, respectively. 11. A method for generating an inverse mapping between received mapping codewords and output EDR codewords in an EDR coding system, wherein the received mapping codewords are generated according to the method of claim 1, the method comprising: For the received mapped codeword, identify all EDR codewords that have been mapped to the received mapped codeword using a mapping function; and A reverse mapping is generated by mapping the received mapping codewords to the output EDR codewords, which are calculated based on the average of all identified EDR codewords mapped to the received mapping codewords. 12. A method for encoding a sequence of input augmented dynamic range (EDR) images comprising image patches, the method comprising: Receive the sequence of the input enhanced dynamic range (EDR) images; For one or more image patches in at least one input image in the sequence of input EDR images, a first block complexity metric is calculated, the first block complexity metric representing the change in pixel value; Construct a first set of image patches, the first set including image patches whose calculated block complexity metric satisfies a first predetermined criterion; Select a candidate set of function parameter values for the mapping function used to map the input codeword to the quantized output codeword; Mapped code values are generated by applying a mapping function of a candidate set having the function parameter values to input EDR codewords in blocks belonging to a first set of the image blocks; and Determining whether the selected candidate set is optimal from the generated mapping code values, wherein the steps for determining whether the selected candidate set is optimal include: For one or more blocks belonging to the first set, the resulting mapping code value is used to calculate a second block complexity metric, which represents the change in pixel value; Construct a second set of image patches, the second set including a second set of image patches whose metrics satisfy a second predetermined criterion; and If the number of blocks in the second set is the same as the number of blocks in the first set, then the selected candidate set is determined to be optimal. If the selected candidate set is determined to be unoptimal, the selection, generation, and determination are repeated using different candidate sets of function parameter values; and The optimal candidate set of determined function parameter values is used to generate a mapping function that maps the input codeword to the quantized output codeword. 13. The method of claim 12 further includes, after all blocks in the scene have been quantized, selecting the overall optimal candidate set of function values from the determined optimal candidate set. 14. The method of claim 12, wherein calculating the first block complexity metric for an image block of pixels includes calculating the standard deviation of pixel values in the block, and wherein the first predetermined metric includes comparing whether the first block complexity metric is greater than zero. 15. The method of claim 12, wherein the candidate set of function parameter values comprises the exponent α of the quantization function calculated as follows: Where _vL and _νH represent the minimum and maximum codeword values in the EDR input sequence, _cL and _cH represent the minimum and maximum output codeword values, _vi represents the input codeword, and _si represents the corresponding output codeword. 16. The method of claim 1 or 12, wherein the output codeword generated by the method of claim 1 or 12 is decomposed into a plurality of layers, the plurality of layers including a first layer and a second layer, and wherein a first video encoder receives the first layer and encodes the first layer using a first bit depth, and a second video encoder receives the second layer and encodes the second layer using a second bit depth, wherein the second bit depth is different from the first bit depth. 17. The method of claim 16, wherein the first bit depth is 8 bits, and the second bit depth is 10 bits, 12 bits, or 14 bits. 18. The method of claim 1 or 12, wherein the output codeword generated by the method of claim 1 or 12 is decomposed into a plurality of layers, the plurality of layers including a first layer and a second layer, and wherein a first video encoder receives the first layer and encodes the first layer using a first encoding format, and a second video encoder receives the second layer and encodes the second layer using a second encoding format, wherein the second encoding format is different from the first encoding format. 19. The method according to claim 18, wherein the first encoding format is MPEG-2 encoding format, and the second encoding format is AVC or HEVC encoding format. 20. An apparatus comprising a processor and configured to perform the method described in any one of claims 1-19. 21. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing the method according to any one of claims 1-19.