HK1235587B - Method and apparatus for backward-compatible coding and decoding for video signals - Google Patents

Description

Method and apparatus for backward compatible encoding and decoding of video signals

This application is a divisional application of the Chinese invention patent application with application number 201380069054.7, application date December 4, 2013, and titled "Backward compatible coding for ultra-high definition video signals with enhanced dynamic range".

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No. 61/748,411, filed January 2, 2013; U.S. Provisional Application No. 61/821,173, filed May 8, 2013; and U.S. Provisional Patent Application No. 61/882,773, filed September 26, 2013, all of which are incorporated herein by reference in their entireties.

Technical Field

The present invention relates generally to images and, more particularly, to backward-compatible encoding and decoding of high-definition signals with enhanced dynamic range.

Background Art

Audio and video compression is a key component in the development, storage, distribution and consumption of multimedia content. The choice of compression method involves a trade-off between coding efficiency, coding complexity and delay. As the ratio of processing power to computational cost increases, more complex compression techniques that allow for more efficient compression can be developed. As an example, in video compression, the Moving Picture Experts Group (MPEG) from the International Organization for Standardization (ISO) has continuously improved the original MPEG-1 video standard by issuing MPEG-2, MPEG-4 (Part 2) and H.264/AVC (or MPEG-4, Part 10) coding standards.

Despite the compression efficiency and achievements of H.264, a new generation of video compression technology called High Efficiency Video Coding (HEVC) is now under development. HEVC is expected to provide improved compression capabilities over the existing H.264 (also known as AVC) standard. A draft of HEVC is available in "High Efficiency Video Coding (HEVC) Text Specification Draft 8" by B. Bross, W.-J. Han, G.J. Sullivan, J.-R. Ohm, and T. Wiegand, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC) document JCTVC-J1003, July 2012, the entire contents of which are incorporated herein by reference. The existing H.264 standard is published as "Advanced Video Coding for generic audio-visual services", ITU T Rec. H.264 and ISO/IEC 14496-10, the entire contents of which are incorporated herein by reference.

A video signal can be characterized by multiple parameters, such as bit depth, color space, color gamut, and resolution. Modern televisions and video playback devices (e.g., Blu-ray players) support a variety of resolutions, including standard definition (e.g., 720×480i) and high definition (HD) (e.g., 1920×1080p). Ultra high definition (UHD) is a next-generation resolution format with at least 3,840×2,160 resolution (known as 4K UHD) and options up to 7680×4320 (known as 8K UHD). Ultra high definition may also be referred to as Ultra HD, UHDTV, or ultra-high vision. As used herein, UHD means any resolution higher than HD resolution.

Another aspect of the characteristics of a video signal is its dynamic range. Dynamic range (DR) is the range of intensities (e.g., brightness, luma) in an image (e.g., from the darkest dark to the brightest bright). As used herein, the term "dynamic range" (DR) can be related to the ability of the human psychovisual system (HVS) to perceive the range of intensities (e.g., brightness, luma) in an image (e.g., from the darkest dark to the brightest bright). In this sense, DR is related to the intensity "of a reference scene". DR can also be related to the ability of a display device to fully or nearly present an intensity range with a particular breadth. In this sense, DR is related to the intensity "of a reference display". Unless a particular meaning is explicitly designated as having particular importance anywhere in the description herein, it should be inferred that the term can be used interchangeably in either sense, for example.

As used herein, the term high dynamic range (HDR) is associated with a DR breadth of 14-15 orders of magnitude across the human visual system (HVS). For example, a substantially normal, adaptable person (e.g., in one or more of a statistical, biometric, or ophthalmological sense) has an intensity range spanning approximately 15 orders of magnitude. Adaptable people can perceive dim light sources as few as a few photons. However, these same people can perceive the almost painfully bright intensity of the midday sun in the desert, at sea, or in the snow (or even look toward the sun, but briefly to prevent damage). However, this span is available for "adaptable" people (e.g., those whose HVS has a time period in which to reset and adjust).

In contrast, the DR over which humans can simultaneously perceive a wide breadth of intensity range may be somewhat truncated relative to HDR. As used herein, the terms "enhanced dynamic range" (EDR), "visual dynamic range," or "variable dynamic range" (VDR) may be individually or interchangeably related to the DR that can be simultaneously perceived by an HVS. As used herein, EDR may be related to DR that spans 5-6 orders of magnitude. Thus, while perhaps somewhat narrow relative to true scene-referenced HDR, EDR represents a broad breadth of DR. As used herein, the term "simultaneous dynamic range" may be related to EDR.

To support backward compatibility with legacy playback devices and new HDR or UHD display technologies, multiple layers can be used to deliver UHD and HDR (or EDR) video data from upstream devices to downstream devices. Given such a multi-layer stream, legacy decoders can use the base layer to reconstruct the HD SDR version of the content. Advanced decoders can use both the base layer and the enhancement layer to reconstruct the UHD EDR version of the content to present it on a more capable display. As the inventors herein have appreciated, improved UHD EDR video coding techniques are desirable.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply because it is included in this section. Similarly, the problems identified for one or more approaches should not be assumed to have been recognized in any prior art based on this section, unless otherwise indicated.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references refer to similar elements and in which:

FIG1 depicts an example implementation of a UHD EDR encoding system according to an embodiment of the present invention;

FIG2 depicts an example implementation of a UHD EDR decoding system according to an embodiment of the present invention;

3 depicts a variation of the system depicted in FIG. 1 in which the base layer comprises an interlaced signal, according to an embodiment of the present invention;

4 depicts a variation of the decoding system of FIG. 2 in which the base layer comprises an interlaced video signal, according to an embodiment of the present invention;

FIG5 depicts an example implementation of a nonlinear quantizer for a residual signal in an enhancement layer according to an embodiment of the present invention;

FIG6A depicts an adaptive prequantization process for residual pixels according to an embodiment of the present invention; and

6B depicts an adaptive process for setting a lower or upper input bound for a nonlinear quantizer of a residual signal according to an embodiment of the present invention.

DETAILED DESCRIPTION

This paper describes backward-compatible encoding of ultra-high-definition signals with enhanced dynamic range. Given an input video signal that can be represented by two signals: one with ultra-high-definition (UHD) resolution and high or enhanced dynamic range (EDR), and the other with UHD (or lower) resolution and standard dynamic range (SDR), the two signals are encoded in a backward-compatible layered stream, which allows older decoders to extract the HD standard dynamic range (SDR) signal and newer decoders to extract the UHD EDR signal.

In the following description, for illustrative purposes, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and apparatus are not described in detail to avoid unnecessarily obscuring the present invention.

Overview

The example embodiments described herein relate to backward-compatible encoding and decoding of ultra-high-definition signals with enhanced dynamic range. Given an input video signal represented by two signals: one with ultra-high-definition (UHD) resolution and high or enhanced dynamic range (EDR), and the other with UHD (or lower) resolution and standard dynamic range (SDR), the two signals are encoded in a backward-compatible layered stream, which allows legacy decoders to extract the HD standard dynamic range (SDR) signal and allows modern decoders to extract the UHD EDR signal. In response to a base layer HD SDR signal, a prediction signal is generated using separate luma prediction models and chroma prediction models. In the luma predictor, the luma pixel values of the prediction signal are calculated based only on the luma pixel values of the base layer, while in the chroma predictor, the chroma pixel values of the prediction signal are calculated based on both the luma pixel values and the chroma pixel values of the base layer. A residual signal is calculated based on the input UHD EDR signal and the prediction signal. The base layer signal and the residual signal are encoded separately to form an encoded bitstream.

In another embodiment, a receiver demultiplexes a received layered bitstream to generate an encoded base layer (BL) stream at HD resolution and standard dynamic range (SDR) and an encoded enhancement layer (EDR) stream at UHD resolution and enhanced dynamic range (EDR). The encoded BL stream is decoded using a BL decoder to generate a decoded BL signal at HD resolution and standard dynamic range. In response to the decoded BL signal, a predicted EDR signal is generated, wherein pixel values of a luma component of the prediction signal are predicted based solely on luma pixel values of the decoded BL signal, and pixel values of at least one chroma component of the prediction signal are predicted based on both luma and chroma values of the decoded BL signal. The encoded EL stream is decoded using an EL decoder to generate a decoded residual signal. In response to the decoded residual signal and the prediction signal, an output UHD EDR signal may also be generated.

In another embodiment, the residual signal in the enhancement layer is adaptively pre-processed before being quantized with a non-linear quantizer. In one embodiment, residual pixel values are pre-quantized to zero if the standard deviation of pixels surrounding the residual pixel value is below a threshold.

In another embodiment, the input range of the non-linear quantizer is limited based on a measure of pixel connectivity of residual pixels having very large or very small pixel values.

In another embodiment, the parameters of the non-linear quantizer are set based on the extreme values of the residual pixels over a sequence of consecutive frames in the scene.

Encoder for Ultra HD EDR signals

Existing display and playback devices, such as HDTVs, set-top boxes, or Blu-ray players, typically support signals with resolutions up to 1080p HD (e.g., 1920×1080 at 60 frames per second). For consumer applications, such signals are now typically compressed using a bit depth of 8 bits per pixel per color component in a luminance-chrominance color format, where the chrominance components typically have a lower resolution than the luminance component (e.g., YCbCr or YUV 4:2:0 color formats). Because of the 8-bit depth and the corresponding low dynamic range, such signals are often referred to as signals with standard dynamic range (SDR).

As new television standards, such as Ultra High Definition (UHD), are being developed, it may be desirable to encode signals with enhanced resolution and/or enhanced dynamic range in a format that can be processed by both older HDTV decoders and newer UHD decoders.

FIG1 depicts an embodiment of an example implementation of a system that supports backward-compatible encoding of UHD signals with enhanced dynamic range (EDR). The encoder includes a base layer (BL) encoder (130) and an enhancement layer (EL) encoder (160). In an embodiment, the BL encoder 130 is a legacy encoder, such as an MPEG-2 or H.264 encoder, and the EL encoder 160 is a new standard encoder, such as an HEVC encoder. To support legacy BL decoders, the BL encoder 130 is typically an 8-bit encoder; however, the EL encoder 160 can support input streams with higher bit depths (such as 10 bits) as specified by the H.264 and HEVC standards. However, the system is applicable to any combination of known or future encoders, regardless of whether they are standard-based or proprietary.

As depicted in FIG1 , an input signal, such as a movie or television broadcast, can be represented by two signals: a UHD EDR input (102) and a UHD SDR input (104). For example, the UHD EDR signal (102) can be a 4K (e.g., 3840×2160) resolution signal captured by an HDR camera and color graded for an EDR display. The same signal can also be color graded on a 4K SDR display to produce a corresponding 4K SDR signal 104. Alternatively, the SDR signal 104 can be produced by applying any of the tone mapping or display management techniques known in the art to the EDR signal. Without loss of generality, both input signals can typically be represented in an RGB color space using a 16-bit or equivalent (e.g., floating point) bit depth representation. As used herein, the term N-bit signal refers to an image or video signal having one or more color components (e.g., RGB or YCbCr), where each pixel in any of these color components (e.g., Y) is represented by an N-bit pixel value. Given an N-bit representation, each such pixel can take a value between 0 and ^2N - 1. For example, in an 8-bit representation, each pixel can take a value between 0 and 255 for each color component.

In an embodiment, the UHD SDR signal 104 may be downsampled to an HD SDR signal (e.g., 1080p), which is then color-converted to a color format suitable for encoding using an older 8-bit encoder (e.g., an 8-bit YCbCr 4:2:0 color format). Such conversion may include color conversion (e.g., RGB to YCbCr conversion 115-C) and chroma subsampling (e.g., 4:4:4 to 4:2:0 conversion 120-C). Thus, the HD SDR signal 128 represents a backward-compatible signal representation of the original UHD EDR signal 102. The signal 128 may be encoded using a BL encoder 130 to produce a backward-compatible encoded bitstream 132. The BL encoder 130 may compress or encode the HD SDR signal 128 using any of known or future video compression algorithms (e.g., MPEG-2, MPEG-4 Part 2, H.264, HEVC, VP8, etc.).

Given a UHD EDR signal 102, downsampling (110-A) and color conversion processes (115-B and 120-B) can convert the UHD EDR signal 102 into a reference predicted HD EDR signal 124. In a preferred embodiment, the downsampling and color conversion processes (110-A, 115-B, and 120-B) (e.g., the selected filters and color space) in this stage should be the same as or as close as possible to the downsampling and color conversion processes (110-B, 115-C, and 120-C) used to generate the HD SDR signal 128 in the base layer.

After UHD EDR conversion to HD EDR, the output of the HD EDR signal 124 is separated into luma (Y 124 -Y) and chroma (CbCr 124 -C) components, which are applied to determine prediction coefficients for the luma predictor 145 and the chroma predictor 140 .

Given an HD SDR signal 128, the BL encoder 130 generates not only an encoded BL bitstream 132, but also a BL signal 126 representing the HD SDR signal 128 as it will be decoded by a corresponding BL decoder. In some embodiments, the signal 126 may be generated by a separate BL decoder (not shown) following the BL encoder 130. In some other embodiments, the signal 126 may be generated from a feedback loop used to perform motion compensation in the BL encoder 130. As depicted in FIG1 , the output of the HD EDR signal 126 may also be separated into its luma (Y 126-Y) and chroma (CbCr 126-C) components, which are applied to a luma predictor 145 and a chroma predictor 140 to predict an HD EDR signal 147.

In an embodiment, the luma predictor 145 may include a polynomial predictor that predicts the luma component of the HD EDR signal 147 based on the luma pixel values of the base layer HD SDR signal 126-Y. In such a predictor, the luma pixel component may be predicted without considering any pixel values in any other color components of the signal. For example, let g _i denote the luma pixel value (126-Y) of the BL HDSDR signal. Then, without loss of generality, the cubic polynomial predictor may be expressed as:

Where a _k , b _k and c _k are predictor coefficients. In an embodiment, the predictor coefficients may be determined using any minimum error technique known in the art, such as minimizing the mean square error (e.g., ) between the predicted value and the luma pixel value (124-Y)(s _i ) in the reference HD EDR signal.

In an embodiment, the chroma predictor 140 may also be a polynomial predictor similar to the polynomial predictor described above; however, in a preferred embodiment, the chroma predictor 140 comprises a multi-color channel, multiple regression (MMR) predictor, such as the predictor described in PCT application Ser. No. PCT/US2012/033605, filed Apr. 13, 2012, by G.M. Su et al., “Multiple Color Channel Multiple Regression Predictor,” published as WO 2012/142471, the entire contents of which are incorporated herein by reference. The MMR predictor uses information from both luma and chroma pixel values in the HD EDR reference signal 124 and the base layer HDSDR signal 126 to predict the chroma components of the HD EDR signal. The prediction coefficients in the MMR model may also be determined using a mean square error minimization technique by minimizing the MSE between the predicted chroma values and the luma and chroma pixel values of the reference HD EDR signal 124.

Because both the HD SDR signal 126 and the reference HD HDR signal 124 are in the YCbCr 4:2:0 format (where the spatial resolution of the luma component is twice that of each chroma component), the luma components of both signals are downsampled (135-A and 135-B) before being applied to the chroma predictor 140. In a preferred embodiment, the filters used in the luma downsampling 135-A and 135-B are the same as the chroma downsampling filters used in the 4:4:4 to 4:2:0 processing (120). The luma and chroma prediction coefficients can be updated at various time intervals of interest, such as per scene, per group of pictures, or per frame. The prediction filter coefficients can be communicated to the decoder in various ways, such as by embedding their values in the bitstream as ancillary data or metadata.

Given the predicted HD EDR signal 147, the upsampler 150 generates a UHD EDR signal 152, which is used to generate a residual signal 167. Because the UHD EDR signal is in a preferred encoding format (e.g., YCbCr 4:2:0), additional color conversion (115-A) and chroma downsampling (120A) steps may be required to convert the original UHD EDR signal 102 in the original format (e.g., RGB) to the preferred encoding format UHD EDR signal 122. Signals 122 and 152 are subtracted to create the EL residual signal 167.

In an embodiment, the color transform (115-A) and chroma subsampling process (120-A) are the same as or as close as possible to the color transform (115-B and 115-C) and chroma subsampling process (120B and 120-C) used to generate the BL encoded signal 128 and the prediction signal 124.

In an embodiment, the EL signal 167 may be processed using a nonlinear quantizer (NLQ) 155 before being encoded by the EL encoder 160. An example of a suitable nonlinear quantizer can be found in PCT patent application number PCT/US2012/034747, filed on April 24, 2012 (published as WO/2012/148883), entitled "Non-linear VDR Residual Quantizer," which is incorporated herein by reference in its entirety. The output of the NLQ 155 may be compressed using the EL encoder 160 to produce an encoded EL bitstream 162 that may be sent to a suitable decoder. In addition, in some embodiments, the residual (167) may also be spatially downsampled using a downsampling module (not shown). Such downsampling (e.g., downsampling by a factor of 2 or 4 in two dimensions) improves coding efficiency, particularly at very low bit rates. Downsampling may be performed before or after the nonlinear quantizer (155).

The EL encoder 160 may be any suitable encoder, such as those described in MPEG-2, MPEG-4, H.264, HEVC specifications, etc. In an embodiment, the BL encoded bitstream 132, the EL encoded bitstream 162, and metadata related to the encoding process (e.g., predictor parameters or lookup tables) may be multiplexed into a single bitstream (not shown).

As depicted in FIG1 , in a preferred embodiment, downsampling (110-A or 110-B) is preferably applied before color format conversion (115-B and 120-B or 115-C and 120-C); however, in some embodiments, downsampling can be performed after color conversion. For example, in one embodiment, the input to 110-A can be received directly from the UHD EDR YCbCr signal 122, thereby eliminating the need to perform color conversion processing 115-B and 120-B to generate the HD EDR reference signal 124. Similarly, downsampling 110-B can be performed after the color conversion step 120-C.

In some embodiments, the baseline HD SDR signal 128 may already be in the correct resolution and color format for use by the encoder 100. In such a case, the downsampling (110-B) and color conversion steps (115-C and 120-C) may be bypassed.

In some embodiments, the UHD EDR signal 120 may be used with less than or greater than 16 bits of precision; however, its precision is expected to be greater than 8 bits (e.g., 10 bits or 12 bits). Similarly, the UHD SDR signal 104 may already be used with less than 16 bits of precision (e.g., 8 bits or 10 bits).

Decoder for Ultra HD EDR signals

FIG2 illustrates an embodiment of an exemplary implementation of a system that supports backward-compatible decoding of UHD signals with enhanced dynamic range (EDR). In response to an encoded signal transmitted by an encoder (e.g., 100), a decoder 200 receives and demultiplexes an encoded bitstream comprising at least two encoded substreams: an encoded BL stream 132 and an encoded EL stream 162.

The encoded BL stream 132 includes an HD SDR signal (217) that can be decoded using a BL decoder 215. In an embodiment, the BL decoder 215 matches the BL encoder 130. For example, to be backward compatible with existing broadcast and Blu-ray standards, the BL decoder 215 can follow one or more of the MPEG-2 or H.264 encoding specifications. After BL decoding 215, the HD SDR decoder can apply an additional color transform (270) to the decoded HD SDR signal 217 to convert the incoming signal from a color format suitable for compression (e.g., YCbCr 4:2:0) to a color format suitable for display (e.g., RGB 4:4:4). A receiver with enhanced resolution and/or EDR display capabilities can combine information from both the BL and EL bitstreams (132 and 162) to produce a UHD signal (e.g., 232) with enhanced dynamic range as depicted in FIG.

After BL decoding 215, the decoded signal 217 is divided into its luma (217-Y) and chroma (217-C) components. The luma component (217-Y) is processed by a luma predictor 240 to produce a luma estimate for an HD EDR signal 255. The luma and chroma components are also processed by a chroma predictor 250 to produce a chroma estimate for the HD EDR signal 255. In an embodiment, before the luma signal 217-Y is processed by the chroma predictor, it is subsampled by a downsampler 245 so that it matches the resolution of the chroma components. The luma and chroma predictors (240 and 250) match the luma and chroma predictors (145 and 140) in encoder 100. Thus, the luma predictor 240 can be a polynomial predictor, while the chroma predictor can be an MMR predictor. In an embodiment, the characteristics and filter parameters of these predictors can be determined using metadata embedded in the received encoded bitstream. After the luma and chroma prediction steps ( 240 and 250 ), the predicted HD EDR signal 255 is upsampled ( 260 ) to produce a UHD EDR signal 265 .

Given the encoded bitstream 162, the EL decoder 210 decodes it to produce a UHD EDR residual signal 212. The EL decoder 210 matches the EL encoder 160. If the encoder 100 applies the nonlinear quantizer 155 to the residual 167, the nonlinear quantization process is reversed by applying a nonlinear dequantizer (NLDQ) 220 to produce a dequantized residual 222. If the encoder (100) applies spatial downsampling to the residual (167), a spatial upsampler (not shown) before or after the NLDQ (220) can upsample the decoded residual (e.g., 212 or 222) to its appropriate spatial resolution. By adding (225) the residual 222 to the estimate 265 of UHD EDR, the decoder 200 can produce a UHD EDR signal 227 that matches the resolution and color format (e.g., 4:2:0 YCbCr) of the UHD EDR signal 122 sent by the encoder. Depending on the target application, a set of color transforms (230) may transform the UHD EDR signal 232 into a format suitable for display or other processing. In an embodiment, given a YCbCr 4:2:0 signal 227, the color transform 230 may include a 4:2:0 to 4:4:4 chroma upsampling step followed by a YCbCr to RGB color conversion step.

Encoding and decoding of mixed progressive and interlaced formats

Despite the increased adoption of progressive video signals (e.g., 720p or 1080p), broadcasts of interlaced video signals (e.g., 1080i) remain quite common in video broadcasting. In another embodiment, FIG3 depicts another example of a UHD EDR encoding system (300) that supports layer encoding using a mix of progressive and interlaced formats. In the example, the BL signal (332) is encoded in an interlaced format (e.g., 1080i or 2160i), while the EL signal (162) is encoded in a progressive format (e.g., 2160p).

The encoding system (300) shares most of the functionality of the encoding system (100), and therefore, in this section, only the key differences between the two systems will be discussed. As depicted in Figure 3, in base layer processing, the SDR signal (104) is color converted to a color format suitable for encoding using the BL encoder (130) (e.g., 4:2:0 YCbCr). In an example embodiment, the output (332) of the BL encoder (130) may include an interlaced SDR signal. The interlacer (320-A) may apply any interlacing and downsampling technique known in the art to convert the progressive input (128) to an interlaced signal (e.g., 1080i) of the desired encoding resolution for the base layer signal (332).

Compared to system (100), in the enhancement layer, the processing components (110-A), (115-B), and (120-B) of system (100) can all be replaced by an interlacer (320-B). Interlacer (320-B) can apply any interlacing and downsampling techniques known in the art to convert the progressive input (122) into an interlaced signal (124) that matches the resolution of the interlaced signal (126). In a preferred embodiment, the downsampling and interlacing functions of (320-A) and (320-B) should be identical to each other or as close as possible to reduce color artifacts and improve overall image encoding quality.

The luma and chroma predictors (145 and 140) in system (300) remain the same as the luma and chroma predictors in system (100); however, they now operate on separate fields of their inputs since signals (124) and (126) are now interlaced signals.

The de-interlacer (350) also has a dual function; it de-interlaces the predicted HD EDR signal (347) and upsamples it to match the resolution of the UHD EDR signal (122), thereby producing a predicted UHD EDR signal (152) having the same resolution and format as the signal (122). The processing of the residual (167) in the system (300) remains the same as that described for the system (100).

In some embodiments, the SDR signal (104) may already be in an interlaced format, and the interlacer (320-A) may be replaced with a downsampler. If the input signal (104) is already interlaced and of an appropriate resolution, the interlacer (320-A) may be eliminated.

In an embodiment, the input signals (102) and (104) may both be HD resolution signals (e.g., 1080p). Then, the output of the system (300) may include an encoded interlaced HD (e.g., 1080i) base layer signal (332) and an encoded progressive HD (e.g., 1080p) residual (162).

In an embodiment, the BL signal (332) and the residual (162) may both be of the same resolution but in a mixed format. For example, the BL signal (332) may be encoded at 2160i, while the EL signal (162) may be encoded at 2160p.

FIG4 depicts an embodiment of an example implementation of a decoder system (400) for decoding a signal produced by a mixed format encoder (300). The system (400) is nearly identical to the decoder system (200) except for the following differences: a) the decoded BL signal (417) is now an interlaced video signal, (b) the luma and chroma predictors (240 and 250) operate on fields of the interlaced signals (417) and (247), and c) the predicted HD EDR signal (455) is an interlaced signal.

The deinterlacer (460) functionally matches the deinterlacer (350) in the system (300); thus, it deinterlaces and upsamples the interlaced HD EDR signal (455) so that its output (the UHD EDR signal (465)) has the same resolution and format as the decoded error residual signal (222).

As previously noted, the system (300) may also include a spatial downsampling module (not shown) in the EL path, before or after the nonlinear quantizer (155). In such a case, in the decoder (400), a spatial upsampler before or after the NLDQ (220) may be used to restore the decoded residual (212) to its proper spatial resolution.

Luminance range driven adaptive upsampling

As depicted in FIG1 , after the luma and chroma prediction steps (140, 145), the predicted HD EDR signal (147) is upsampled (150) by a factor of 2 to produce a predicted UHD EDR signal 152. Similar processing is also performed in the decoder (200), where, after the luma and chroma prediction steps (240, 250), the predicted HD EDR signal (255) is upsampled (260) by a factor of 2 to produce a predicted UHD EDR signal (265). The upsamplers (150) and (260) may include any upsampling technique known in the art; however, improved image quality may be achieved by utilizing a luma range driven adaptive upsampling technique as described in this section.

It has been observed that the prediction error (167) between the original EDR signal (122) and its predicted value (152) can vary depending on the luma value in the corresponding SDR signal (104). That is, the residual (167) in bright or highlighted areas of the image exhibits different types of characteristics than the residual in dark or mid-tone areas. In an embodiment, the luma range of the SDR input can be divided into two or more luma sub-ranges. An adaptive upsampling filtering method can apply different upsampling filters to different pixels of the EDR predicted image, where each filter is selected based on the luma sub-range of the corresponding pixel in the SDR image. Thresholds identifying each of these luma sub-ranges and the identity of the filter used and/or the filter coefficients themselves can be transmitted from the encoder (100) to the decoder (200) via metadata or other auxiliary data so that both the encoder and decoder can apply the same upsampling filter to improve image quality.

Let t denote the luma pixel value of the HD EDR signal (147), which is predicted based on the luma value of the output of the BL encoder (130), i.e., the SDR signal s _ij (126-Y). Let th(i) (i=0, N) denote a set of thresholds for dividing the luma range of a pixel (0≤s _ij ≤1) into N luma ranges of interest (N≥1) (e.g., black, midtones, and highlights for N=3). Let H _i denote a set of filter coefficients for the i-th (i=1, N) upsampling filter for the i-th luma range of interest in step (150) or (260), and let t denote a function of s _ij or its local neighbors, then in an embodiment, upsampling filtering (e.g., 150 or 260) may be performed according to the following algorithm 1 expressed in pseudo code:

Algorithm 1 - Luminance Range Driven Upsampling

In some embodiments, _Hi may represent the filter coefficients of a 2-D non-separable filter. In some other embodiments, _Hi may represent the coefficients of a 2-D separable upsampling filter, including but not limited to coefficients for horizontal and vertical upsampling filters. The filter coefficients _Hi may be pre-computed and stored in memory, or they may be adaptively computed based on some image quality criterion. For example, in an embodiment, the filter coefficients _Hi may be computed to minimize the mean square error between the output of the up-scaling filter (the predicted UHD EDR signal (152)) and the input UHD EDR signal (122).

In some embodiments, it may represent a single pixel value of interest (e.g., _sij or _sij-1 ), while in some other embodiments it may represent a local average or some other function (e.g., median, minimum, or maximum) of one or more pixels surrounding _sij .

In embodiments, the th(i) threshold may be determined based on image statistics of the input signal (e.g., the average of blacks, midtones, or highlights). These statistics may be calculated per pixel region, per frame, or per scene (e.g., a set of sequential pictures with similar luminance characteristics). In some embodiments, th(i) may be determined iteratively as part of the filter design process. For example, considering the case where filter coefficients _Hi are calculated based on some optimization criterion (e.g., minimizing the mean square error (MSE) of the signal (167)), then, in embodiments, Algorithm 2 describes in pseudocode an example method for determining a new threshold (th*) given two boundary thresholds (t_low and t_high) and a threshold search step size (step):

Algorithm 2 - Threshold determination for two brightness sub-ranges (N=2)

In the above description, t_low and t_high represent boundary values of interest for which a threshold may be searched. For example, in an embodiment, t_low = min(s _ij ) = 0 and t_high = max(s _ij ) = 1 (where 1 represents the normalized maximum possible value) covers the entire range of possible luminance values; however, in other embodiments, the range of boundary values may be much smaller. For example, the threshold value calculated for the input frame at time t may take into account a threshold value calculated earlier (say, at time t-1), thereby searching only within a smaller range (e.g., th(i)-C, th(i)+C, where C is a constant) centered around the previous threshold value.

Given Algorithm 2, in some embodiments, a similar approach can be used to subdivide the luminance range of a picture frame into additional partitions of the luminance range using additional thresholds. In an example embodiment, the following algorithm (Algorithm 3) can be used to subdivide a given luminance range (A, B) into two or three luminance sub-ranges.

Algorithm 3 - Threshold determination for three brightness sub-ranges (N=3)

The thresholds calculated by Algorithms 2 and 3 may be applied to Algorithm 1 in both the encoder (100) and the decoder (200). In an embodiment, the calculated thresholds may be sent from the encoder (100) to the decoder (200) using metadata.

As previously mentioned, deinterlacers (350) and (460) can combine both deinterlacing and upsampling functions. Those skilled in the art of image processing will appreciate that the luma range driven approach discussed herein for the improved design of upsamplers (150) and (126) can also be applied to the design of the upsamplers in deinterlacers (350) and (460).

Adaptive residual processing

As depicted in Figures 1 and 3, in an enhancement layer (EL), a residual signal (167) may be processed by a nonlinear quantizer (NLQ) (155) before being compressed by an EL encoder (160) to produce an EL stream (162). Without loss of generality, Figure 5 depicts an example input-output relationship for the NLQ (155) according to an embodiment of the present invention.

As depicted in FIG5 , let (−X _max , X _max ) denote the range of pixel values for the residual pixel x (167) to be encoded in the frame or frame region of interest. Let Level denote the number of available codewords on each side of the quantizer (e.g., Level = 128 for x ≥ 0), then, given a positive threshold T, let

Then, given an input residual x, after clipping x to be within the range (-X _max , X _max ), the quantization operation of FIG5 can be expressed as:

Wherein, Q(x) represents the quantized output, SL represents the slope of Q(x) within (T, X _max ), and M represents an offset value representing the output codeword when the residual x = 0. The threshold T is a relatively small value, and in some embodiments, T = 0.

The parameters T, M, X _max and SL can be defined separately for each color component of the residual signal x and can be transmitted to the receiver using metadata. In some embodiments, one or more of the NLQ quantization parameters can also be defined for an entire frame, one or more partitions or sub-regions of a frame, or a group of frames (e.g., a scene).

Given such a quantizer, at the receiver (e.g., (200)), the dequantization process (e.g., NLDQ (220)) can be expressed as:

in

R _cmp represents the received (decoded) residual (or EL signal (212)), and represents the dequantized output (222), which may also be limited to, for example, a range.

Experimental results show that appropriate preprocessing of the residual data (167) combined with adaptive setting of the parameters of the NLQ (155) can lead to more efficient encoding of the EL stream, resulting in reduced coding artifacts and better overall image quality. In this section, three residual preprocessing algorithms are described next.

Residual prequantization using standard deviation metric

Improper quantization and encoding of the residual signal (167), particularly when encoding the EL stream at a relatively low bitrate (e.g., 0.5 Mbits/s), can result in blocking artifacts in the decoded signal (232). In an embodiment, these artifacts can be reduced by adaptively pre-quantizing certain residual values that are perceived as being in relatively "smooth" regions. An example of such a process according to an embodiment of the present invention is depicted in FIG6A , where, without limitation, measuring the smoothness of a rectangular region of pixels surrounding each residual pixel is based on calculating the standard deviation of the pixels in that region.

Let r _fi denote the i-th residual pixel of the f-th frame. Let the pixel be at the center of a W _σ ×W _σ pixel region (e.g., W _σ = 15) denoted as n _fi . Then, in step (602), the standard deviation σ _fi of the pixel can be determined as:

in

Given a threshold T _σ , in step (606), if σ _fi < T _σ , the residual pixel r _fi can be set to a predetermined value (e.g., zero). The threshold T _σ can be fixed, or in a preferred embodiment, can be adaptively determined based on the residual frame characteristics and the overall bit rate requirements. For example, let P _f denote the total number of pixels in the f-th frame. Let σ _fi denote the standard deviation value calculated in step (602). In step (604), T _σ can be determined as follows:

(a) Sort σ _fi in descending order to produce a sorted list;

(b) Then, _Tσ is the k* _Pf value in the sorted list, where k is defined in the range 0.0 to 1.0. For example, for k=0.25, given a 1920×1080 frame, _Tσ corresponds to the value of the 518,400th standard deviation value in the sorted list.

Alternative methods of calculating smoothness may also include calculating the mean or variance of _Wσ × _Wσ pixels, or calculating a metric based on an edge map of the area surrounding each pixel, or using any other smoothness detection and determination algorithm known in the art.

Residual tail boundary adjustment

Let denote the maximum positive residual value in frame f, and let denote the absolute value of the minimum negative residual value in frame f. Then,

and

As depicted in FIG5 , the input boundaries of the quantizer (e.g., ) can be determined in terms of and ; however, experimental results show that the residual values have a bell-shaped distribution, and there are typically very few pixels close to or in each frame. As previously noted, for the quantizer depicted in FIG5 , the quantization step size is proportional to . For a fixed number of codewords (e.g., the value of Level), the distortion due to quantization is proportional to the value of X _max ; therefore, smaller values of X _max are preferred. In an embodiment, instead of determining X _max in terms of or , a new, smaller range [Th _f- Th _f+ ] is determined. Before applying NLQ (155), the residual pixel values are restricted (or clipped) to lie within the new range [Th _f- Th _f+ ]; where, for frame f, Th _f+ represents the boundary of the positive residual and Th _f- represents the boundary of the negative residual. That is,

r _fi =clip3 (r _fi , Th _f- , Th _f+ ),

The clip3() function indicates that residual pixel values greater than Th _f+ are clipped to Th _f+ , and residual pixel values less than Th _f− are clipped to Th _f− .

While a smaller input range for the NLQ process results in smaller errors due to quantization, unrestricted clipping of the residual signal may result in noticeable artifacts, so the selection of the new range needs to be modified based on the characteristics of the residual signal. In an embodiment, the two thresholds are adaptively determined based on the connectivity (or sparsity) of the observed residual pixel values. That is, isolated residual pixels with very large values can be clipped with minimal impact on the overall quality; however, the pixel values of connected residual pixels should be appropriately encoded. An example implementation of such a boundary determination process according to an embodiment of the present invention is depicted in Figure 6B as process (650).

Process (650) calculates a threshold value Th that satisfies the following condition: residual pixel values equal to or greater than Th are considered to be sparsely connected, so they can be clipped. Process (650) can be used to calculate either _Thf- or Thf ₊ boundaries based on the input residual value. For example, to determine Thf ₊ = Th, the process only considers positive residual pixel values, such as in the range (0,):

To determine Th _f −=Th, the process only considers the absolute values of negative residual pixel values, for example, in the range (0,):

In step (610), the process begins by setting an initial value to a threshold value Th. Therefore, given the original bounds of r _fi (e.g., Th_L=0 and or), in an example embodiment, the initial threshold value can be set to the middle of a known range, such as:

Th＝(Th_H+Th_L)/2.

Given a threshold Th, in step (612), a binary map _Mf is generated, wherein the elements of the binary map are calculated as:

m _fi =(R _fi ≥ Th)

M _f (i)＝m _fi .

Given M _f , in step (614), the connectivity of each binary pixel can be determined. For example, in MATLAB, the function bwconncomp can be used to calculate the neighbor connectivity (e.g., a neighborhood of 4 pixels or 8 pixels connected). Let NC _f (i) denote the number of neighbors of each pixel in the binary image M _f . In step (618), the threshold Th can be adjusted so that if the connectivity of the pixels exceeds a predetermined connectivity threshold T _∝ (e.g., T _∝ = 5 pixels), then these pixels are not cropped. For example, if the maximum pixel connectivity over all pixels exceeds a predetermined connectivity threshold T _∝ , then the threshold Th can be increased, otherwise, the threshold Th can be decreased. For example, using a binary search,

To reduce computational complexity, in an embodiment, the process may include a convergence test step (620). For example, the convergence step (620) may calculate the difference between the previous (or old) threshold and the new threshold. If the difference is greater than a predetermined convergence threshold, the process continues again from step (612) with the new threshold. Otherwise, it terminates and outputs the final boundary to be used (e.g., Th _f+ = Th).

Scenario-based nonlinear quantization

As previously discussed, in some embodiments, the nonlinear quantizer (155) may be expressed in terms of the following parameters: _Xmax , offset (e.g., M), and Level (see also the discussion regarding FIG5). In some embodiments, it may be beneficial to determine these parameters in terms of residual pixel characteristics in a sequence of frames (e.g., a scene).

Given the sum for a sequence of F frames, let

Then, the parameters of the nonlinear quantizer can be set for the entire scene as:

Level=max(2 ^EL_bitdepth -1)-Offset, Offset},

and

X _MAX = (1+Δ)max{X ^- ,X ⁺ },

Where EL_bitdepth represents the bit depth of the EL encoder (160) (e.g., EL_bitdepth=8), and Δ represents a small positive value (e.g., Δ=0.1). In an embodiment, for chroma components, the number of quantization levels can be determined using the following formula:

In another embodiment, the sum value may also be replaced by the corresponding Th _f+ and Th _f- values calculated as before.

Example Computer System Implementation

Embodiments of the present invention may be implemented using a computer system, a system configured using electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller), a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete-time or digital signal processor (DSP), an application-specific IC (ASIC), and/or an apparatus comprising one or more of such systems, devices, or components. The computer and/or IC may execute, control, or run instructions related to encoding UHD EDR signals such as those described herein. The computer and/or IC may calculate any of the various parameters or values related to encoding UHD EDR signals as described herein. The encoding and decoding embodiments may be implemented using hardware, software, firmware, and various combinations thereof.

Certain implementations of the present invention include a computer processor that runs software that causes the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to encoding UHD EDR signals as described above by running software instructions in a program memory accessible to these processors. The present invention can also be provided in the form of a program product. The program product may include any medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention may be in any of a variety of forms. The program product may include, for example, physical media such as magnetic data storage media (including floppy disks, hard drives), optical data storage media (including CD ROMs, DVDs), electronic data storage media (including ROMs, flash RAMs, etc.). The computer-readable signals on the program product may optionally be compressed or encrypted.

Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, discussion of that component (including discussion of "means") should be interpreted as an equivalent of that component, including any component that performs the function of the described component (e.g., functionally equivalent), including components that perform the functions in the illustrated example embodiments of the invention that are not structurally equivalent to the disclosed structure.

Equivalence, extension, substitution, and others

Thus described are example embodiments relating to backward-compatible encoding and decoding of UHD EDR signals. In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and what the applicants intend for the invention, is the set of claims from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Accordingly, any limitation, element, property, feature, advantage, or attribute that is not expressly recited in a claim shall not limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims (12)

1. A method for decoding a layered stream using a decoder including a processor, the method comprising: The encoded bitstream is received, the encoded bitstream including an enhancement layer EL stream with a first spatial resolution and a first dynamic range, and a base layer BL stream with a second spatial resolution and a second dynamic range, wherein the first dynamic range is higher than the second dynamic range. The encoded BL stream is decoded using a BL decoder to generate the first decoded BL signal; In response to the decoded BL signal, a prediction signal having the first dynamic range is generated, wherein the luminance pixel value of the prediction signal is predicted based only on the luminance pixel value of the decoded BL signal, and the chrominance pixel value of at least one chrominance component of the prediction signal is predicted based on both the luminance pixel value and the chrominance pixel value of the decoded BL signal. The encoded EL stream is decoded using an EL decoder to produce the first decoded residual signal; The residual signal from the first decoding is processed by a nonlinear dequantizer to generate the residual signal from the second decoding; and In response to the residual signal and the prediction signal of the second decoding, an output signal with a first spatial resolution and a first dynamic range is generated. 2. The method of claim 1, wherein the first dynamic range is a high or enhanced dynamic range, and the second dynamic range is a standard dynamic range. 3. The method according to claim 1, wherein the first spatial resolution is the same as the second spatial resolution. 4. The method of claim 1, wherein the first spatial resolution is higher than the second spatial resolution, and generating the output signal further comprises: The predicted signal is upsampled to generate an upsampled predicted signal with the first spatial resolution; and In response to the second decoded residual signal and the upsampled prediction signal, an output signal with a first spatial resolution and a first dynamic range is generated. 5. The method of claim 4, wherein the first spatial resolution is an ultra-high definition (UHD) spatial resolution and the second spatial resolution is a high definition (HD) spatial resolution. 6. The method of claim 4, wherein generating the upsampled prediction signal further comprises: Receive luminance thresholds from the encoder to divide the luminance range into luminance sub-ranges; Receive extended filter information for each of the lumen sub-ranges from the encoder; For one or more pixels in the predicted signal, the luminance threshold and selection criteria are applied to determine a luminance subrange within the luminance subrange for one or more corresponding pixels in the decoded BL signal; and An extended filter corresponding to the said luminance subrange is applied to the one or more pixels in the predicted signal to produce the corresponding pixel values of the upsampled predicted signal. 7. The method of claim 1, wherein the decoded BL signal comprises an interlaced signal, and generating the output signal further comprises: The predicted signal is upsampled and deinterlaced to produce a line-by-line upsampled predicted signal with the first spatial resolution; and In response to the second decoded residual signal and the line-by-line upsampled prediction signal, an output signal with a first spatial resolution and a first dynamic range is generated. 8. The method of claim 1, wherein generating the second decoded residual signal comprises calculating: in R _cmp represents the residual of the first decoding, represents the residual of the second decoding, and SL, M, and T represent the dequantization parameters transmitted from the encoder to the decoder in the encoded bitstream. 9. The method of claim 8, wherein the value of the residual of the second decoding is further limited to a minimum residual value and a maximum residual value. 10. An apparatus for decoding a layered stream, comprising a processor and configured to perform the method according to any one of claims 1 to 9. 11. A non-transitory computer-readable storage medium storing computer-executable instructions for performing the method according to any one of claims 1 to 9 using one or more processors. 12. An apparatus for decoding a layered stream, comprising means for performing the method according to any one of claims 1 to 9.