HK1227202B - Methods of encoding and decoding image data - Google Patents

Description

Method for encoding image data and method for decoding image data

This application is a divisional application of the invention patent application with the application date of September 6, 2011, application number "201180049970.5" (international stage application number is PCT/US2011/050484), and invention name "Image processing method and device using local color gamut definition".

References to Related Applications

This application claims priority to U.S. Provisional Application No. 61/394,294, filed October 18, 2010, the entire contents of which are incorporated herein by reference.

Technical Field

The present invention relates to image processing, image data compression, image display and image reproduction. Embodiments of the present invention provide methods and apparatus for processing image data for transmission to and display on a downstream device.

Background Art

Image data (including video data and still image data) can have any of a variety of different formats. Some example image data formats are: RGB, YLU, GIF, TIFF, JPEG/JIF, PNG, BMP, PDF, RAW, FITS, MPEG, MP4, high dynamic range (HDR) formats such as BEF, HDRi, JPEGXR, JPEGHDR, RGBE, ScRGB, and a variety of other formats. Image data formats can have capabilities that differ significantly in various aspects, such as the range of different colors that can be specified (i.e., gamut), the range of brightness levels that can be specified (i.e., dynamic range), the number of discrete colors within a gamut that can be specified, the number of discrete brightness levels that can be specified, etc. Some image data formats include multiple versions with different capabilities.

Colors can be specified in many different color spaces. Some examples include RGB, HSV, CIE LUV, YCbCr, YIQ, xvYCC, HSL, XYZ, CMYK, CIE LAB, IPT, and others. Different image data formats can specify colors in different color spaces.

Displays can include any of a wide variety of basic display technologies. Display technologies range from digital cinema displays to televisions and may include: liquid crystal display (LCD) panel displays, where the LCD panel is backlit by different types of backlighting systems; light emitting diode (LED) displays; organic LED displays (OLED displays); plasma displays; cathode ray tube (CRT) displays; laser projectors; digital micromirror device (DMD) displays; electroluminescent displays, and the like. Within any of these general technologies, there can be a wide variety of different configurations and arrangements for the light emitting and/or filtering elements. As a result, different displays can have significantly different capabilities in various aspects, such as different ranges of colors that can be displayed (i.e., color gamuts), ranges of brightness values that can be displayed (i.e., dynamic ranges of displays), and the like.

A wide color gamut display may be able to reproduce colors outside the gamut displayable on a conventional display. However, the range and number of discrete colors that can be specified by the image data provided to the display are constrained by various factors, such as the capabilities of the image data format, the image data bandwidth, and image data compression.

There is a need for methods and apparatus for processing and displaying image data having different color gamuts. There is a need for methods and apparatus for processing and displaying image data so as to reproduce different colors on a display with high accuracy. In particular, there is a need for methods and apparatus that can efficiently deliver image data to a display or other downstream device and that can operate within the constraints of different image data formats, available image data bandwidth, and image data compression.

Summary of the Invention

Various aspects of the invention and example embodiments of the invention are described below and illustrated in the accompanying drawings.

According to one aspect of the present invention, a method for encoding image data for transmission to a downstream device is provided, the method comprising: receiving image data, the image data having a format and a content gamut; determining one or more gamut characteristics of the image data; determining a local gamut definition for the image data based at least in part on the gamut characteristics; and applying a gamut transform to the image data, wherein color coordinates specified by the image data are mapped to corresponding color coordinates of the local gamut definition; and wherein the gamut of the image data having the format is the range of colors that can be represented by the format; the content gamut of the image data is the range of colors appearing in the image data; and the local gamut definition limits fewer color coordinate values than the gamut of the image data having the format.

According to another aspect of the present invention, there is provided a method for decoding image data for reproduction on a display, the image data having a format, the method comprising: extracting metadata from the image data, the metadata representing a local color gamut definition; decompressing the image data; and applying a color gamut transform to the image data, wherein the color gamut transform maps color coordinate values of the local color gamut definition to color coordinate values of the image data format; and wherein the color gamut of the image data having the format is a range of color coordinate values that can be represented by the format, the local color gamut definition includes a range of color coordinate values appearing in the image data, and the local color gamut definition limits fewer color coordinate values than the color gamut of the image data having the format.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate non-limiting embodiments of the invention.

1A and 1B are color gamut diagrams in the CIE u'v' color space.

2A and 2C are flowcharts of a method of encoding image data according to example embodiments.

2B and 2D are flowcharts of a method of decoding image data according to example embodiments.

3A , 3B, and 3C are flowcharts of a method of selecting a local color gamut definition according to example embodiments, which may be used in the method of FIG. 2A .

FIG. 4 shows an area of a frame of image data divided into sub-frame blocks.

FIG5 illustrates a predefined color gamut definition for a region of a frame of image data.

6A and 6B are block diagrams respectively illustrating an encoding apparatus and a decoding apparatus according to example embodiments.

DETAILED DESCRIPTION

In the following description, specific details are set forth to provide a more thorough understanding for those skilled in the art. However, well-known elements may not be shown or described in detail to avoid unnecessarily obscuring the present disclosure. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Colors may be represented in a three-dimensional or three-component color space. The content color gamut may be defined as the range of color space occupied by local groups of pixels in the image data. The color gamut of an image data format may be defined as the range of color space that may be specified by the image data format. For example, an image data format that specifies colors in a three-component color space using four bits per component allows up to (2 ⁴ ) ³ or 4096 color or component combinations to be specified within the color gamut of the image data format. Unless a large sample of image data (e.g., a long sequence of frames) is considered, it is unlikely that local groups of pixels in the image data will require all possible colors that may be specified by the image data format. Furthermore, it will be appreciated that certain color component combinations tend to appear more frequently in image data than other color component combinations.

Certain embodiments of the methods and apparatus described herein may be applied to situations where one or more frames, or regions of frames, of image data do not utilize the full color gamut that may be specified by the image data format. In such situations, the image data may be encoded using a local color gamut definition. As described below, the use of a local color gamut definition may facilitate image data compression, increased color depth specification for image data, and processing of the image data by downstream devices in a video delivery pipeline.

Figures 1A and 1B illustrate, for example, situations where an area of the content gamut (the range of colors occupied by a local group of pixels in the image data), bounded by content gamut boundary 22, occupies a portion of the area of the entire gamut of the respective image data formats of the image data. In Figure 1A , the image data can be assumed to be in a Rec. 709 format having a gamut bounded by gamut boundary 20. In Figure 1B , the image data can be assumed to be in a wide gamut format (i.e., having a gamut wider than the Rec. 709 format) having a gamut bounded by gamut boundary 26. For each of these situations, the image data can be encoded using a local gamut definition that can be determined at least in part based on content gamut boundary 22. For reference purposes, a spectrum locus 30 (a locus representing points of pure spectral or monochromatic colors) and CIE standard illuminant D65 (a white point representing average daylight) are also plotted on Figures 1A and 1B. It should be understood that for ease of illustration, Figures 1A and 1B illustrate the gamut boundaries in two dimensions. As noted above, colors can be represented in a three-dimensional or three-component color space. Color information specified in a three-dimensional color space can be transformed into a color space in which chromaticity information is specified along two axes (such as shown in Figures 1A and 1B) and luminance information is specified along a third axis (not shown in Figures 1A and 1B).

In the case shown in Figure 1A, a local color gamut can be defined having a local color gamut boundary 24 that is consistent with the content color gamut boundary 22. In other embodiments, the local color gamut boundary 24 can surround or include the content color gamut boundary 22 and/or be close to the content color gamut boundary 22. The origin of the color space occupied by the local color gamut can be defined at a point within the local color gamut. During encoding of the image data, color coordinates specified in the image data can be mapped to corresponding color coordinates in the local color gamut. Because the local color gamut includes a portion of the area of the color gamut of the image data format, a reduced number of bits can be used to represent the color coordinates within the local color gamut. Therefore, in such cases, the local color gamut definition can be used to reduce the amount of data used to represent the image.

Bandwidth limitations for wireless or wired data transfer may limit the amount of image data that can be transmitted to a display. Therefore, it may be desirable to apply a local color gamut definition, such as that described above with reference to FIG. 1A , to reduce the amount of data required to represent the colors in an image while preserving the original color depth of the image.

In other embodiments, a local color gamut definition may be employed to reduce the size of the steps between colors that can be specified within a partition of the color gamut of the image data format (i.e., reduce the quantization levels between colors), thereby increasing the precision with which colors can be specified. The number of colors within a particular color gamut that can be represented in an image at one time, or the color depth, is limited by the number of bits used to represent the color of each pixel (i.e., the "bit depth"). For example, assume that the content color gamut of the image data to be encoded is bounded by the content color gamut boundary 22 of FIG. 1A , and that a 16-bit depth color scheme (using 5 bits to represent 32 levels of red, 6 bits to represent 64 levels of green, and 5 bits to represent 32 bits of blue, for a total of 32×64×32 or 65,536 different colors) is being used to represent all available colors within the Rec. 709 format shown as bounded by the color gamut boundary 20. In some image frames or frame regions, all 65,536 levels of color that can be specified by such a color scheme may not be used; therefore, to reduce the quantization levels between colors, a new local color gamut definition may be applied in which all 65,536 levels of color are redefined as colors within the local color gamut. The use of a local color gamut definition may increase the color depth within the local color gamut (thereby reducing the quantization levels between colors) at the expense of the depth of colors outside the local color gamut. For example, if the local color gamut is defined within the Rec.709 format as having a local color gamut boundary 24 that coincides with the content color gamut boundary 22, a new 16-bit depth color scheme may be defined to represent all available colors within the local color gamut, thereby increasing the number of colors that can be specified within the local color gamut to 65,536 (i.e., 32×64×32).

In certain situations where a display is capable of reproducing colors at a greater color depth than can be specified by the image data format and/or where finer control over the quantization levels between colors is desired, it may be desirable to increase the color depth using a local color gamut definition. For example, as shown in FIG1B , if the bit depth is constant between different image data formats, the color specification accuracy of image data in a wide color gamut format (having a color gamut bounded by a color gamut boundary 26) will be lower than the color specification accuracy of image data in a narrower color gamut format such as the Rec.709 format (having a color gamut bounded by a color gamut boundary 20). The difference between the quantization levels of colors in the wider color gamut is necessarily larger to allow colors to be specified in the increased color gamut area. In order to increase the accuracy with which colors can be specified, the image data can be provided at a higher bit depth, but this may not be practical in most cases.

In the example of FIG1B , the image data is in a wide color gamut format, and some colors of the content color gamut (defined by color gamut boundary 22) are outside the Rec. 709 color gamut. The content color gamut of the image data occupies a portion of the area of the color gamut of the wide color gamut format shown as being bounded by color gamut boundary 26. To increase the accuracy with which colors in the image data can be represented, a local color gamut having a local color gamut boundary 24 that coincides with or includes content color gamut boundary 22 can be defined for the image data. A new color scheme can be defined to represent colors within this local color gamut. Color coordinates specified in the image data can be mapped to corresponding color coordinates in the local color gamut.

An example illustrating how color depth can be increased by using a local color gamut definition is as follows: Assume that the full color gamut of an image data format is described by N combinations of color component codewords, and that the local content color gamut is smaller than the full color gamut of the image data format (e.g., the local content color gamut requires a subset of the N possible combinations). The local content color gamut boundary can be determined, and the N codewords can be reallocated to occupy the color space up to the local content color gamut boundary. Thus, the N codewords can be used to specify colors in a smaller partition of the full color gamut of the image data format at a higher color resolution (color depth) than before.

The new color scheme used for the local color gamut definition may have the same bit depth as the image data format. In other cases, the new color scheme used for the local color gamut definition may have a reduced bit depth than the image data format, while still allowing colors within the local color gamut to be specified at a desired color depth.

In some embodiments, the new color scheme defined for the local color gamut can be selected and used by a specialized color grader during post-production processing of the image data to specify colors with greater precision than would otherwise be permitted by the existing color scheme of the image data format. In other embodiments, the new color scheme defined for the local color gamut can be determined or retrieved by a downstream device and used to enhance the color specification of processed image data to which color gamut compression (e.g., a process that compresses wide color gamut image data into the constraints of a more commonly used color gamut (e.g., the Rec. 709 color gamut) has been applied.

In the case where the image data is encoded using a local color gamut definition, information about the local color gamut definition may be transmitted to a downstream device so that the downstream device can decode the image data. Such information may be encoded as metadata in the image data, or may be transmitted as metadata in front of a frame of image data (such as at the beginning of a movie, broadcast, scene, etc.). A downstream device (e.g., a set-top box or display processor) may decode the image data for display by extracting the local color gamut information from the image data and applying such information to convert color coordinate values defined using the local color gamut definition into color coordinate values for reproduction on a display. The remapped color coordinate values may have the same format and definition as ordinary color coordinate values so that the transmission, display, etc. of the image data are not affected.

Prior to display, the downstream device may optionally filter the decoded image data to reduce visual artifacts that may appear in the image data after decoding. Visual artifacts may result from encoding the image data using different local color gamut definitions between frames or frame regions.

In certain embodiments, local color gamut information may be used at various stages in a video delivery pipeline to facilitate the specification of color information and operations that depend on local color characteristics. A typical video delivery pipeline includes various stages, such as content capture, post-production editing, encoding, transmission, decoding, and display. Local color gamut information generated earlier in the pipeline by an upstream device may be passed down the pipeline (e.g., using metadata) to guide one or more downstream devices in processing the video data. Downstream devices may use the local color characteristics to, for example, enhance detail in an image, mask or suppress noise or artifacts, perform local tone mapping, and/or determine the local color gamut required for a colored backlight in a dual-modulation display. The overall computational cost of such operations may be reduced by using local color gamut information that has been pre-calculated by an upstream device and provided to downstream devices.

FIG2A illustrates a method 100 for encoding image data using a local color gamut definition, according to one embodiment. The method 100 begins by receiving content image data 102, which may include one or more frames or regions of frames of image data. The content image data 102 may include, for example, video data. At block 104, the content color gamut of the content image data 102 is characterized. The color gamut characterization of block 104 may include, for example, determining one or more of the following:

A boundary that includes all points of the content gamut or a portion of the content gamut;

An area within the boundaries that includes all points of the content gamut or a portion of the content gamut;

• Statistical information about color coordinates or color information specified for a local pixel group, such as mean, standard deviation, variance, maximum and minimum values, etc.

The color gamut characterization method is also discussed below with reference to FIG. 3A to FIG. 3C .

At block 107, the color gamut representation determined at block 104 is evaluated to determine whether conditions are suitable for applying a local color gamut transform to the image data. If conditions are determined to be suitable at block 107, a local color gamut for the content image data 102 is determined or retrieved at block 108. The local color gamut can be selected from a set of predetermined local color gamut definitions or can be generated for the content image data 102 based on the color gamut representation at block 104. Methods for determining local color gamut definitions are discussed below with reference to Figures 3A to 3C. Local color gamut definitions can be assigned to blocks or other regions of an image. In some embodiments, a frame can be divided into multiple sub-frame blocks, one or more of which can be associated with a local color gamut definition. In other embodiments, the region associated with the local color gamut can be defined based on the color gamut characteristics of the image (e.g., a local color gamut can be assigned to objects or regions within a frame having similar hues).

The local color gamut of block 108 may be defined by a color gamut boundary that is consistent with or encompasses the content color gamut. The color gamut boundary may be represented in any of a variety of ways, such as:

A set of points that is a subset of the points in an existing color gamut (eg, the Rec. 709 color gamut); and a polynomial representation.

An example of a local color gamut boundary 24 corresponding to the content color gamut defined by the content color gamut boundary 22 is depicted in FIG1A and FIG1B . The local color gamut may be in the same or a different color space than the color gamut of the image data format. The origin of the color space defining the local color gamut may be located at a point within the local color gamut, but this is not required, and the origin may be defined at some other location. In some embodiments, the origin may be located at the average of the color coordinate values of the image data.

At block 110, a local color gamut transform is applied to the image data to map color coordinates specified by the content image data 102 to corresponding color coordinates in the local color gamut. In some embodiments, a new color scheme may be defined to represent color coordinates in the local color gamut. The new color scheme may have the same or a different bit depth than the image data format; the bit depth determines the precision with which colors can be specified in the local color gamut. For example, where the content color gamut occupies a portion of the area of the color gamut of a conventional or non-wide color gamut format (e.g., the case shown in FIG. 1A ), it may not be necessary to use the full bit depth for the image data, and a reduced bit depth may be used to represent color coordinates within the local color gamut. On the other hand, where the content color gamut occupies a portion of the area of the color gamut of a wide color gamut format and at least a portion of the content color gamut is outside the gamut of the conventional or non-wide color gamut format (e.g., the case shown in FIG. 1B ), the full bit depth of the image data format may be used to represent all available colors within the local color gamut to achieve the desired level of precision.

The local color gamut transform at block 110 may be performed using, for example, the following:

Look-up table (LUT);

A function that accepts input values and converts those input values into corresponding output values in the local color gamut;

●And so on.

In certain embodiments, for the purposes of characterizing the color gamut, evaluating color gamut characteristics, determining a local color gamut transform, and/or applying a local color gamut transform (i.e., blocks 104, 107, 108, and/or 110 of method 100A), color information may be transformed from its original color space into another color space. The new color space may be more convenient for visual and perceptual analysis. The transformation to the new color space may facilitate image data processing and more efficient representation of image data. The transformation to the new color space may be nonlinear.

Statistical information about the color signals of the image data, evaluated in the original color space, can be converted into different statistical values in the new color space. When characterizing the color gamut, statistical metrics such as mean, maximum, minimum, standard deviation, and variance can be considered for local groups of pixels in the original and/or new color spaces. Boundary descriptions such as distance from the mean can also be considered.

According to some embodiments, one or more transforms may be applied at block 110 to reduce the range of the transformed color signal information while preserving the salient color characteristics. In some embodiments, by testing nonlinear transforms on various statistical metrics, an optimal transform may be selected and applied to compress the color signal information while minimizing or reducing the distortion of the salient color characteristics. According to certain embodiments, a nonlinear transform is applied followed by a generally optimal linear transform (e.g., singular value decomposition or principal component analysis) to further compress the color signal information.

At block 112, the content image data 102 is compressed. Compression may be performed using one or more techniques appropriate to the image data format, such as:

• transform coding, such as using the discrete cosine transform (DCT), as may be used for example for JPEG image data and MPEG video data;

● as can be used, for example, for predictive coding of MPEG video data;

• Lossless compression (e.g., Lempel-Ziv-Welch compression) as may be used for, for example, GIF and TIFF image data;

Wavelet-based compression;

●A combination of one or more of the above technologies;

●And so on.

In some embodiments, the residual image data (i.e., the difference between the input or previous frame image data and the predicted output or subsequent frame image data after applying the prediction technique) is encoded using standard image coding techniques (e.g., DCT, frequency-dependent or frequency-independent quantization, frequency scan ordering, fixed or variable run length coding, etc.). Prediction is performed using inter-frame or intra-frame prediction techniques. The frame can be divided into local sub-frame blocks. Each consecutive block is predicted using previously encoded image data. Due to the block-based prediction technique, the residual image data carries some statistical information of the input image and some new local statistical information. The local color domain representation technique described herein can be applied to the residual image data.

To assist downstream devices in processing and decoding the image data, metadata 116 may be embedded in the image data or transmitted along with the image data at block 114. The metadata 116 may provide information about the local color gamut definition for a particular frame or region of a frame. The metadata 116 may include, for example, an index into a library of local color gamut definitions 40 ( FIG. 5 ) available to the downstream device; coordinates 42 ( FIG. 5 ) identifying the region of the frame to which the local color gamut definition (e.g., the local color gamut definition specified by the index) is applied; information about a mapping function or a lookup table from which a lookup table may be derived (i.e., for mapping color coordinates defined using the local color gamut definition back to color coordinates in the image data for reproduction on a display); and the like. Compressed processed image data 118 including the metadata 116 is then output for transmission to the downstream device.

The local color gamut definition may not be applied in every case. For example, in method 100, for certain frames or regions of the image data, if the evaluation of the color gamut characteristics at block 107 determines that conditions are not suitable for applying a local color gamut transform to the image data, then the local color gamut definition may not be applied. In such cases, method 100 may bypass the local color gamut determination and transformation and proceed to compressing the image data at block 112. In some embodiments, metadata 116 indicating that the local color gamut definition was not applied to such image data may be transmitted to a downstream device.

For example, as shown in method 120 of FIG. 2B , processed image data 118, which has been encoded according to method 100 of FIG. 2A , can then be decoded by a display or other downstream device to prepare the data for output to the display. Method 120 begins by receiving processed image data 118. Method 120 continues to block 122 by extracting metadata 116 from processed image data 118. At block 124, the image data is decompressed. At block 126, the color information of the decompressed image data is decoded by applying an inverse local color gamut transform to the image data. The inverse transform in block 126 may be based at least in part on metadata 116, which guides the mapping of color coordinate values from the applicable local color gamut definition to color coordinate values in the image data format for reproduction on a display. For example, a downstream device implementing method 120 may have the local color gamut definition library 40 (see FIG. 5 ) available to it. The metadata 116 may include a local color gamut flag for indexing the local color gamut definition library 40 and coordinates for identifying regions of the frame to which a particular local color gamut definition is applied (see FIG. 5 ). In some embodiments, the metadata 116 may be used to configure the local color gamut definition library 40 .

In other embodiments, the transform of block 126 may map color coordinate values from the applicable local color gamut definition to color coordinate values within a different color space (eg, an RGB color space) for reproduction on a display.

At block 128 of method 120, the decoded image data is optionally filtered to reduce visual artifacts that may appear in the image data after decoding.The resulting filtered decoded image data is output as output image data 130 to a display.

Transmission metadata 116 requires some overhead to allow downstream devices to decode the image data. Despite the increased overhead, greater compression of the image data can be achieved in appropriate circumstances by encoding the image data using a local color gamut definition, such as in the case shown in FIG. 1A , where the area of the content color gamut is a fraction of the area of the color gamut of the image data format.

Figures 3A, 3B, and 3C illustrate methods 200, 240, and 280, respectively, which may be applied as specific example implementations of blocks 104, 107, and 108 of method 100 (Figure 2A), according to example embodiments. In each of methods 200, 240, and 280, a frame of content image data 102 is received at block 202 and divided into sub-frame blocks, each of which includes N×N pixels associated with a value representing color information. Figure 4, for example, illustrates a region of frame 31 divided into four sub-frame blocks 32. A frame may actually have more than the four sub-frame blocks shown. In the example shown in Figure 4, each sub-frame block 32 has 8×8 pixels. If the image data is represented in the CIE LUV color space, each of the 64 pixels of the sub-frame block may be associated with an L', u', and v' value. For image data represented in other color spaces, each pixel may be associated with other color values.

In each of methods 200, 240, and 280, color gamut characteristics of each sub-frame block 32 are determined at block 204. Based on such color gamut characteristics, one or more conditions may be evaluated at block 206. If certain conditions are met, then a local color gamut definition is retrieved at block 208.

For each of methods 200, 240, and 280, the color gamut characteristics determined based on the image data may include a value representing the average color of each sub-frame block (as determined at block 204A). In some embodiments, such a value may be, for example, a DC coefficient (i.e., DC value) of a DCT coefficient matrix (see FIG. 4 ). Determination of the DC coefficient 33 may include applying a discrete cosine transform (DCT) to the pixel values in the sub-frame block 32. The color gamut characteristics of the image data may also include a minimum and maximum color value for each sub-frame block (as determined at block 204B). When the image data is represented in the CIE LUV color space, the average value (e.g., DC coefficient 33) and the minimum and maximum values for each sub-frame block 32 may be calculated separately for each of the L', u', and v' components. In certain other embodiments, the color gamut characteristics at block 204 may be provided to the encoder via transmission from an upstream device (such as a color grading tool) or via metadata within the image data.

Methods 200, 240, and 280 differ in the conditions evaluated at block 206 and the local color gamut definition selected or generated at block 208. Method 200 of FIG. 3A evaluates whether conditions are suitable for using a local color gamut definition to facilitate data compression. Method 240 of FIG. 3B evaluates whether conditions are suitable for using a local color gamut definition to increase color depth. Method 280 of FIG. 3C includes aspects of methods 200 and 240 and evaluates whether conditions are suitable for using a local color gamut definition to facilitate data compression or increase color depth.

As described above, the method 200 of FIG3A begins by dividing the frame into sub-frame blocks 32 at block 202 and determining color gamut characteristics at block 204. After determining the color gamut characteristics at block 204, the method 200 continues to block 206B by evaluating, for each sub-frame block 32, whether (1) the absolute value of the difference between the average and the minimum value and (2) the absolute value of the difference between the average and the maximum value, respectively, are less than a predetermined threshold value (i.e., "Threshold 1"). If these differences are not less than the threshold value, the method 200 may continue by returning an indication that conditions are not suitable for use of a local color gamut definition at block 209. For example, the content color gamut may cover an area so large that use of the local color gamut definition does not reduce the image data by more than a negligible amount, after accounting for the overhead of transmitting information regarding the use of the local color gamut definition to a downstream device.

In alternative embodiments, one or more other conditions may be evaluated at block 206B of method 200 to determine whether it is appropriate to use a local color gamut definition for data compression purposes. These other conditions may include, for example:

• Evaluate the area of the content gamut as a percentage of the area of the image data format's gamut;

• Evaluate the volume of the content gamut as a percentage of the volume of the image data format's gamut (ie, considering the gamut in a three-dimensional color space);

= Evaluate the distance between the mean and the boundaries of the color gamut of the image data format;

● Evaluate the color characteristics of adjacent subframe blocks;

• Evaluate the costs and benefits of applying a local color gamut transform (e.g., costs may include visual artifacts);

● Evaluate the costs and benefits of applying a global color gamut transform;

●And so on.

If the difference between the average and the minimum/maximum values, as evaluated at block 206B of method 200, is less than the threshold, method 200 continues by retrieving a local color gamut definition for the image data at block 208D. The local color gamut definition may be selected from a set of predetermined local color gamut definitions, retrieved from a memory, extracted from a video signal, and/or generated based on the average and/or minimum and maximum values (or other color gamut characteristics) determined at block 204. The local color gamut definition may have boundaries that coincide with or encompass the content color gamut.

Method 240 of FIG3B determines a local color gamut definition that can be used to encode image data to increase its color depth. Method 240 can be applied, for example, in situations where the image data format has a wide color gamut (wider than a conventional color gamut such as the Rec. 709 format), because in such situations, it may be desirable to increase the precision with which colors can be specified in the image data. As described above with reference to FIG1B , encoding image data using a local color gamut definition can allow a greater number of colors to be specified within a region of the color gamut of the image data format without increasing the bit depth of the image data.

As shown in Figure 3B, after determining the color gamut characteristics at block 204, method 240 continues to block 206A by determining whether a point on the boundary of the content color gamut is sufficiently close to the boundary of the color gamut of the image data format. If the boundary is not sufficiently close, method 240 may continue by returning an indication that the conditions are not suitable for using a local color gamut definition at block 209 (e.g., because the content color gamut does not approach the boundary of the color gamut of the wide color gamut format, there is no need to define a special local color gamut; however, in some embodiments, another color gamut definition may be used to encode the image data, such as the color gamut definition of a traditional or non-wide color gamut format).

If the boundary of the content color gamut is sufficiently close to the boundary of the color gamut of the image data format, then the method 240 proceeds to block 206C by evaluating whether (1) the absolute value of the difference between the average and the minimum value and (2) the absolute value of the difference between the average and the maximum value are each less than a predetermined threshold value (i.e., "Threshold 2"). If these differences are not less than the threshold value, then the method 240 may continue by returning an indication that the conditions are not suitable for use of a local color gamut definition at block 209. For example, the content color gamut may cover an area so large that use of a local color gamut definition does not increase the color depth more than noticeably.

If the difference between the average and the minimum/maximum values evaluated at block 206C of method 240 is less than the threshold, method 240 continues by retrieving a local color gamut definition for the image data at block 208A. The local color gamut definition may be selected from a set of predetermined local color gamut definitions, or may be generated based on the average and/or minimum and maximum values (or other color gamut characteristics) determined at block 204. The local color gamut definition may have boundaries that coincide with or encompass the content color gamut. A new color scheme may be defined for the local color gamut to increase color depth.

In other embodiments, the local color gamut definition may be defined regardless of how close the content color gamut boundary is to the color gamut boundary of the image data format. However, certain operations (such as transforms based on local color gamut information) may be based on the relative relationship of the local content color gamut boundary to the color gamut boundary of the image data format and/or to the local content color gamut boundary of an adjacent sub-frame block.

Method 280 of FIG3C includes aspects of methods 200 and 240 to retrieve a local color gamut definition, thereby facilitating data compression or increasing color depth. In method 280, steps similar to those of methods 200 and 240 are labeled with the same reference numerals. As can be seen in FIG3C, after determining the color gamut characteristics at block 204, method 280 proceeds to block 206 (which includes blocks 206A, 206B, and 206C) by evaluating the conditions described above with reference to block 206B of FIG3A and blocks 206A and 206C of FIG3B for each subframe block 32. Based on the results of such evaluation, method 280 proceeds to one of the following steps as shown in FIG3C:

• returning an indication at block 209 that the condition is not suitable for use of the local color gamut definition;

• Retrieving a local color gamut definition at block 208A to increase color depth; and

• Retrieve the local color gamut definition at block 208D to facilitate data compression.

In some embodiments of methods 200, 240, and 280, if conditions are not suitable for using a local color gamut definition, method 240 may simply do nothing at block 209 (causing image processing to continue with the next steps without selecting and applying a local color gamut definition), rather than returning an indication at block 209.

It should be understood that the steps of determining and evaluating color gamut characteristics and selecting a local color gamut definition (as described above for methods 200, 240, and 280) may be performed for each subframe block (or other portion) of image data. These methods may be repeated for successive frames of image data.

FIG2C illustrates an example method 100A for encoding image data using a local color gamut definition according to another embodiment. The method 100A of FIG2C is similar in some respects to the method 100 of FIG2A , and similar reference numerals are used to represent similar steps, except that the reference numerals for the steps of method 100A are appended with the letter "A." The method 100A may be performed on each frame (or region of a frame) of the content image data 102.

Method 100A begins by receiving a frame or region of a content image data 102. At block 104A, the content color gamut of the content image data 102 is characterized. At block 107A, the color gamut characterization determined at block 104A is evaluated to determine whether conditions are suitable for applying a local color gamut transform to the image data. The steps performed at blocks 104A and 107A may be similar to the steps performed at blocks 104 and 107 of method 100.

If the conditions are determined to be met at block 107A, method 100A may continue by determining an average color coordinate value for the image data and, based on such average color coordinate value, applying a local color gamut transform to the remaining color coordinate values of the image data. For example, in the illustrated embodiment of method 100A, a discrete cosine transform (DCT) may be applied to the pixel values of the image data at block 106A to determine the DC coefficient 33 (i.e., DC value) of the DCT coefficient matrix of the image data (see FIG. 4 ). Based on the DC coefficient 33, a local color gamut transform may be determined at block 108A. A local color gamut transform may be applied to the image data at block 110A so that the remaining values in the DCT coefficient matrix (i.e., AC coefficients 34, see FIG. 4 ) tend to become smaller, on average, as the local color gamut transform maps the color coordinate values to new values closer to the DC coefficient 33. Consequently, when the transformed image data is quantized at block 111A, many of the AC coefficients 34 may be quantized to zero, thereby facilitating compression of the image data at block 112A. The local color gamut transform at block 110A does not affect the DC coefficient 33. In some embodiments, the local color gamut transform at block 110A may be determined based on the DC coefficient of the block (or other region) within the frame, or based on the DC coefficients of multiple neighboring blocks within the frame (e.g., such as an average of the DC coefficients of the neighboring blocks).

The transform used in method 100A for determining the DC or mean value at block 106A and applying the local color gamut transform at block 110A is not limited to a discrete cosine transform. In other embodiments, some other transform may be applied that begins by determining the DC or mean component for the image data and then determines the difference (or "residual") from the mean component for each value.

To assist downstream devices in processing and decoding the image data, metadata 116 may be embedded in the image data or transmitted along with the image data at block 114A. The metadata 116 may include an indication that a local color gamut transform was applied to the image data. In particular embodiments, the metadata 116 may provide information about the algorithm used to determine and apply the local color gamut transform (e.g., whether the local color gamut transform was selected based on the DC coefficient of a single block within a frame or based on the DC coefficients of multiple adjacent blocks within a frame, etc.). The resulting transformed, compressed processed image data 118A is output to the downstream device.

In other embodiments, blocks 104A and 107A of method 100A need not be performed. In such embodiments, a local color gamut transform may be performed on each sub-frame block (or other region) of image data based on the DC coefficient of the sub-frame block and/or the DC coefficients of surrounding or nearby sub-frame blocks. Upon receiving the sub-frame block (or other region) of image data, the method may continue by determining the DC coefficient for the sub-frame block of image data, as described, for example, at block 106 of method 100A. In other embodiments, a local color gamut transform may be performed on each sub-frame block of image data based on color gamut information generated, for example, by a color grading tool and transmitted using metadata transmitted in the image data.

In some embodiments where a local color gamut transform is performed on each sub-frame block (or other region) of image data, it may not be necessary to encode metadata 116 in the image data at block 114A of method 100A. For example, a downstream device may be configured to proceed by decoding the image data under the assumption that a local color gamut transform is applied to each sub-frame block of the image data based on a predetermined algorithm (e.g., an algorithm based on the DC coefficient of the block and/or the DC coefficients of surrounding or nearby sub-frame blocks).

FIG2D illustrates an example implementation of a method 120A that can be performed by a display or other downstream device to decode image data that has been encoded according to, for example, method 100A of FIG2C . Method 120A can be performed on each frame or region of frames of processed image data 118 to prepare the image data for output to the display. Method 120A of FIG2D is similar in some respects to method 120 of FIG2B , and similar reference numerals are used to represent similar steps, except that the reference numerals for the steps of method 120A are appended with the letter "A."

Method 120A begins by receiving a frame or region of a frame of processed image data 118. Method 120 continues to block 122A by extracting metadata 116 from the processed image data 118. At block 124A, the processed image data 118 is decompressed. If, at block 124B, the metadata 116 indicates that a local color gamut transform was applied to the image data, then, at block 125A, a DC coefficient 33 is extracted from the image data (since it is unaffected by the local transform applied at block 110A of method 100A). Based on this DC coefficient 33, an inverse local color gamut transform can be determined at block 125B. That is, based on the DC coefficient 33 (optionally in conjunction with the metadata 116, for certain embodiments), a downstream device can determine the algorithm used to determine the local color gamut transform at block 108A of method 100A. At block 126A, an inverse local color gamut transform is applied to the image data to map the color coordinate values of the applicable local color gamut definition to color coordinate values in the image data format for reproduction on a display. In other embodiments, the transform of block 126A may map the color coordinate values of the applicable local color gamut definition to color coordinate values in a different color space (e.g., an RGB color space) for reproduction on a display.

At block 128A of method 120A, the decoded image data may be filtered to reduce visual artifacts that may appear in the image data after decoding. The resulting filtered decoded image data is output as output image data 130 to a display.

6A and 6B illustrate encoding and decoding devices 140 and 160, respectively, that can be used to implement one or more methods described herein. The decoding device 160 can be used to decode image data that has been encoded by the encoding device 140. The encoding device 140 can implement, for example, one or more of the methods 100 ( FIG. 2A ), 100A ( FIG. 2C ), 200 ( FIG. 3A ), 240 ( FIG. 3B ), and 280 ( FIG. 3C ). The decoding device 160 can implement, for example, the methods 120 ( FIG. 2B ) and 120A ( FIG. 2D ).

In the illustrated embodiment of FIG. 6A , encoding device 140 includes a color gamut characterization and selection unit 142. The color gamut characterization and selection unit 142 may be configured to perform, for example, one or more of methods 200 ( FIG. 3A ), 240 ( FIG. 3B ), and 280 ( FIG. 3C ). The color gamut characterization and selection unit 142 may be configured to receive content image data 102 and determine or obtain color gamut characteristics of the content image data 102. The color gamut characterization and selection unit 142 may have access to a repository 144 of local color gamut definitions. The color gamut characterization and selection unit 142 may execute software and/or hardware functions to evaluate the color gamut characteristics of the content image data 102. This evaluation may be performed for each of a plurality of sub-frame blocks of an image data frame. Based on this evaluation, the color gamut characterization and selection unit 142 may select an appropriate local color gamut definition 146 from the repository 144 of local color gamut definitions for use in encoding the image data. As described above, the use of the local color gamut definition 146 may facilitate data compression or more accurate color reproduction.

The local color gamut definition 146 may be provided to a color gamut transform unit 148 along with the content image data 102. The color gamut transform unit 148 is configured to map color coordinates specified by the content image data 102 to corresponding color coordinates of the local color gamut definition, resulting in transformed data 150 that is provided to a compression unit 152 of the encoding device 140.

For some frames or sub-frame blocks of image data, it may be determined based on an evaluation of the color gamut characteristics that conditions are not suitable for using a local color gamut definition. In such cases, the color gamut characterization and selection unit 142 causes the content image data 102 to be passed to the compression unit 152, thereby bypassing the color gamut conversion unit 148.

Compression unit 152 is operable to compress the image data it receives using an appropriate compression technique, such as one of the compression algorithms described above. The resulting compressed image data 154 is passed to metadata writer 156, which is in communication with color gamut conversion unit 148 and has access to metadata 116 in a metadata repository. Based on the signal received from color gamut conversion unit 148, metadata writer 156 can encode metadata (e.g., metadata specifying a local color gamut definition) that will assist downstream devices in decoding the metadata. The resulting processed image data 118, including the metadata 116, is then output for transmission to downstream devices.

FIG6B illustrates a downstream device (i.e., a decoding device 160) that can receive and decode processed image data 118 for reproduction on a display. In the illustrated embodiment, decoding device 160 includes a metadata reader or extractor 162 configured to extract metadata 116 from processed image data 118. The image data is then provided to a decompression unit 164 for decompression. An inverse transform unit 168 receives uncompressed data 166 and metadata 116. Guided by metadata 116, inverse transform unit 168 decodes the image data by applying the inverse of an applicable local color gamut transform to the image data. The inverse transform maps color coordinate values defined by the local color gamut to color coordinate values in an image data format for reproduction on a display. A filter 172 filters the resulting decoded image data 170 to reduce visual artifacts after decoding and outputs the image data 130 to a display.

Some specific ways in which the method and apparatus according to the present invention may be implemented include:

● an image processing chip comprising a device according to an embodiment of the present invention;

● a set-top box, television, computer monitor, projector, and/or other display that includes a decoding device as described herein;

A source of image data, such as a DVD player (e.g., a Blu-ray player), a video player, a camera, a mobile phone, etc., including the encoding apparatus described herein;

● a physical or non-transitory computer-readable medium including firmware or other computer-readable instructions that, when executed, configure a programmable processor and/or a configurable logic device to perform a method according to the present invention;

one or more central processing units (CPUs), one or more microprocessors, one or more field programmable gate arrays (FPGAs), one or more application specific integrated circuits (ASICs), one or more graphics processing units (GPUs), or any combination thereof, or any other suitable processing unit comprising hardware and/or software capable of performing the methods according to the present invention; and

• A color grading tool including the encoding device described herein.

In addition, the technology provided by this disclosure can be configured as follows:

Solution 1. A method for encoding image data for transmission to a downstream device, comprising:

receiving image data, the image data having a content color gamut;

determining one or more color gamut characteristics of the image data;

determining a local color gamut definition for the image data based at least in part on the color gamut characteristic; and

A color gamut transform is applied to the image data, wherein color coordinates specified by the image data are mapped to corresponding color coordinates defined by the local color gamut.

Option 2. The method according to claim 1, comprising: compressing the image data after applying the color gamut transformation.

Option 3. The method according to any one of 1 or 2 includes: encoding metadata corresponding to the local color gamut definition in the image data.

Option 4. A method according to any one of 1 to 3, wherein the image data includes subframe blocks of image data.

Option 5. The method according to claim 4, wherein the color gamut characteristic includes a value representing an average color of the sub-frame block.

Option 6. The method according to claim 5, wherein a discrete cosine transform (DCT) is applied to the subframe block to provide a DCT coefficient matrix, and the color gamut characteristic includes a DC value of the DCT coefficient matrix.

Option 7. The method according to any one of Claims 4 to 6, wherein the color gamut characteristics include a minimum color value and a maximum color value of the subframe block.

Option 8. A method according to any one of Claims 4 to 7, wherein the color gamut characteristic includes an area of the content color gamut.

Option 9. A method according to any one of Claims 4 to 7, wherein the color gamut characteristic comprises a volume of the content color gamut.

Option 10. The method according to any one of 4 to 9, wherein the color gamut characteristics include color characteristics of adjacent sub-frame blocks of image data.

Option 11. The method according to any one of items 1 to 10, wherein the image data has an image data format having a color gamut wider than a Rec.709 color gamut.

Option 12. A method according to any one of 1 to 11, wherein the content color gamut is a subset of the color gamut of the image data format.

Option 13. The method of claim 12, wherein the local color gamut definition limits the range of colors within the content color gamut.

Option 14. The method according to 13, wherein the local color gamut definition has the same bit depth as the image data format.

Option 15. A method according to any one of claims 1 to 14, wherein evaluating the color gamut characteristics includes transforming the color coordinates specified by the image data into a new color space.

Option 16. A method according to any one of claims 1 to 15, wherein applying the color gamut transform includes transforming the color coordinates specified by the image data into a new color space.

Option 17. The method according to 16, wherein the color gamut transformation is nonlinear.

Embodiment 18. A method of decoding image data for rendering on a display, comprising:

extracting metadata from the image data, the metadata representing a local color gamut definition;

decompressing the image data; and

An inverse color gamut transform is applied to the image data, wherein the inverse color gamut transform maps color coordinate values defined by the local color gamut to color coordinate values in an image data format.

Option 19. The method according to 18, comprising: filtering the image data after applying the inverse color gamut transform to reduce the occurrence of visual artifacts.

Embodiment 20. A method of decoding image data for rendering on a display, comprising:

determining one or more color gamut characteristics of the image data;

determining a local color gamut definition for encoding the image data based on the color gamut characteristic;

An inverse color gamut transform is applied to the image data, wherein the inverse color gamut transform maps color coordinate values defined by the local color gamut to color coordinate values in an image data format.

Option 21. The method according to 20, wherein the color gamut characteristic includes a DC value of the image data.

Solution 22. A device for encoding image data, comprising:

The color gamut characterization and selection unit is configured to:

receiving image data;

determining a color gamut characteristic of the image data; and

selecting a local color gamut definition from a repository based on the evaluation of the color gamut characteristics; and

The color gamut conversion unit is configured to:

receiving the image data and the local color gamut definition; and

A local color gamut transform is applied by mapping color coordinates specified by the image data to corresponding color coordinates defined by the local color gamut.

Solution 23. The device according to 22 comprises: a compression unit configured to receive and compress the image data transformed by the color gamut transformation unit.

Solution 24. The device according to any one of 22 or 23, comprising:

A metadata writer is configured to encode metadata corresponding to the local color gamut definition in the image data.

Solution 25. A device for decoding image data, comprising:

a metadata extractor configured to extract metadata from the image data, the metadata representing a local color gamut definition;

a decompression unit configured to decompress the image data; and

An inverse transform unit is configured to apply an inverse color gamut transform to the image data, wherein the inverse color gamut transform maps color coordinate values defined by the local color gamut to color coordinate values in an image data format.

Option 26. The device according to option 25, comprising: a filtering unit configured to filter the inverse-transformed image data to reduce the occurrence of visual artifacts.

Solution 27. A device for decoding image data, comprising:

a local color gamut definition unit configured to: evaluate a color gamut characteristic of the image data, and determine a local color gamut definition for encoding the image data based on the color gamut characteristic;

a decompression unit configured to decompress the image data; and

An inverse transform unit is configured to apply an inverse color gamut transform to the image data, wherein the inverse color gamut transform maps color coordinate values defined by the local color gamut to color coordinate values in an image data format.

Embodiment 28. A non-transitory computer-readable medium storing computer-executable instructions, wherein when the computer-executable instructions are loaded on a processor, the processor is caused to perform the steps of any one of methods 1 to 21.

Although various exemplary aspects and embodiments have been discussed above, those skilled in the art will appreciate certain variations, permutations, additions, and sub-combinations of these exemplary aspects and embodiments. It is therefore intended that the following appended claims, as well as the claims set forth below, be interpreted as including all such variations, permutations, additions, and sub-combinations as are within their true spirit and scope.

Claims (19)

1. A method for encoding image data for transmission to a downstream device, the method comprising: Receive image data, the image data having a format and content color gamut; Determine one or more gamut characteristics of the content gamut of the image data; Based at least in part on the color gamut characteristics, determine the local color gamut definition for the image data; and A gamut transformation is applied to the image data for which the local gamut definition has been determined, wherein the color coordinates specified by the image data are mapped to the corresponding color coordinates defined by the local gamut; and Wherein, the color gamut of the image data having the aforementioned format is the range of colors that can be represented by that format; The content color gamut of the image data is the range of colors appearing in the image data; and Compared to the color gamut of the image data having the aforementioned format, the local color gamut definition defines fewer color coordinate values. 2. The method according to claim 1, comprising: compressing the image data to which the color gamut transformation has been applied. 3. The method according to claim 2, further comprising: encoding metadata corresponding to the local color gamut definition in the compressed image data. 4. The method according to claim 3, wherein the image data comprises subframe blocks of image data. 5. The method of claim 4, wherein the color gamut characteristic includes a value representing the average color of the subframe block. 6. The method of claim 4, wherein a Discrete Cosine Transform (DCT) is applied to the subframe block to provide a DCT coefficient matrix, and the gamut characteristics include DC coefficients of the DCT coefficient matrix associated with the Discrete Cosine Transform. 7. The method according to claim 4, wherein the color gamut characteristic includes the minimum color value and the maximum color value of the subframe block. 8. The method of claim 4, wherein the color gamut characteristic includes the area of the content color gamut. 9. The method of claim 4, wherein the color gamut characteristic includes the volume of the content color gamut. 10. The method of claim 4, wherein the color gamut characteristics include the color characteristics of adjacent subframe blocks of image data. 11. The method of claim 1, wherein the color gamut of the image data having the format is wider than the Rec.709 color gamut. 12. The method of claim 1, wherein the content color gamut is a subset of the color gamut of the image data having the format. 13. The method of claim 12, wherein the local color gamut definition defines the range of colors within the content color gamut. 14. The method of claim 13, wherein the local color gamut definition has the same bit depth as the image data having the format. 15. The method of claim 1, wherein determining one or more color gamut characteristics comprises transforming the color coordinates specified by the image data to a new color space. 16. The method of claim 15, wherein applying the color gamut transformation includes transforming the color coordinates specified by the image data to a new color space. 17. The method of claim 16, wherein the color gamut transformation is nonlinear. 18. The method of claim 1, wherein the content color gamut represents the range of colors appearing in the image data, occupied by local pixel groups; and the local color gamut definition includes the range of colors within the content color gamut. 19. A method for decoding image data for reproduction on a display, the image data having a format, the method comprising: Metadata is extracted from the image data, and the metadata represents a local color gamut definition; Decompress the image data from which the metadata has been extracted; and A color gamut transformation is applied to the decompressed image data, wherein the color gamut transformation maps the color coordinate values defined by the local color gamut to the color coordinate values of the image data format; and Wherein, the color gamut of the image data having the aforementioned format is the range of color coordinate values that can be represented by that format. The local color gamut definition includes the range of color coordinate values appearing in the image data, and Compared to the color gamut of the image data having the aforementioned format, the local color gamut definition defines fewer color coordinate values.