HK1226574A - Method and apparatus for image data transformation - Google Patents

Description

Method and apparatus for image data transformation

The present application is a divisional application of an invention patent application having an application number of 201280013129.5, an application date of 2012, 3 and 15, and an invention name of "method and apparatus for image data conversion".

Cross Reference to Related Applications

This application claims priority from U.S. provisional patent application 61/453,107 filed on 3/15/2011 and U.S. provisional patent application No.61/567,784 filed on 12/7/2011, both of which are incorporated herein by reference in their entirety for all purposes.

The present application also relates to international patent application No. pct/US2012/027267, filed 3/1/2012, incorporated herein by reference.

Technical Field

The present invention relates to displaying and processing images. The invention relates in particular to a method and apparatus involving tone and/or gamut mapping. The methods and apparatus as described herein may be used to provide a quality image on a target display while preserving authoring intent. The present invention may be embodied in, for example, electronic displays (such as televisions, computer monitors, media players, video-enabled cellular telephones, and other portable devices), dedicated displays (such as virtual reality displays, advertising displays, etc.), and upstream image processing devices (such as set-top boxes, access points, etc.).

Background

Patent publications in the general field of the invention include the following:

US20010050757；

US20020075136；

US20020080245；

US20070127093；

US20080094515；

US20080170031；

US20080186707；

US20090201309；

US20090267876；

US20100007599；

US201000118008；

US7158673；

US6989859；

US 5276779; and

JP2002092655。

a creator of a video work or other image (e.g., a director, colorist, etc.) may set the hue and color of the pixels in the image so that the image, when viewed, has a desired appearance consistent with the creator's creative intent. For example, an author may want certain scenes to have a darker, more oppressive feel than others. The author may want some of the features depicted in the scene to be prominent or less prominent. Adjusting the hue and color of pixels in an image may include performing color grading (or "color mediation") on the source video data. Color grading may be performed using a hardware/software system that allows a user to change video data in various ways to achieve a desired appearance.

Various display technologies are now available. For example, there are plasma displays, LCD displays that are back-illuminated by various types of light sources (such as various types of LEDs, fluorescent lamps, or high-intensity incandescent lamps), CRT-based displays, digital cinema displays, and the like. Certain displays combine display hardware with a video signal processing component that receives a video signal and drives the display hardware to display video content of the video signal.

Different displays may differ greatly in terms of features such as:

a display reproducible color gamut;

the maximum brightness that can be achieved;

a contrast ratio;

resolution ratio;

an acceptable input signal format;

color depth;

white level (white level);

black level (blacklevel);

white points;

a gray scale;

and the like.

Thus, the same image content may appear different when played back on different displays. Image content that matches the author's creative intent when displayed on some displays may deviate from the author's creative intent in one or more ways when viewed on other displays.

Some current displays may outperform prior art displays in one or more respects when certain content is authored. For example, the new displays may be able to provide images with brighter highlights, greater contrast, and/or a wider color gamut than the older displays. It may be desirable to take advantage of these improved capabilities without significantly deviating from the creative intent embodied in the content being viewed.

It may be desirable to play video content authored with a high performance display on an older display or a display with lower capabilities. It would be desirable to provide methods and apparatus for altering the appearance of video and other images displayed on different displays to preserve as much as possible the creative intent embodied in the image data.

The perception of color and brightness may be affected by ambient conditions. The viewer's perception of a video or other image presented in a cinema condition (low ambient lighting) is significantly different than the perception of the same video or other image when viewed in a condition of significant ambient light. Furthermore, characteristics of ambient light (such as color temperature) may affect the viewer's perception of the video content. It would be desirable to display videos or other images taking into account the viewing environment of the content to preserve as much as possible the creative intent embodied in the videos or other images.

There is a need for a viewing experience that provides viewers of images (including still images and/or video images) with the ability to take advantage of the display on which they view the images. There is also a need for an apparatus and method that can be used to adjust image data such that video or other image content encoded in the image data has a desired appearance when played.

Disclosure of Invention

The present invention has several aspects. These aspects include, but are not limited to, devices that incorporate gamut transformation functionality; methods for gamut transformation, methods for adapting the display of image content to take into account ambient lighting conditions; program product comprising computer readable code which, when executed by a data processor, causes the data processor to perform a method according to the invention.

One non-limiting aspect provides an apparatus comprising a pixel coordinate mapping unit configured to transform image data according to a transfer function. The transfer function is characterized by a plurality of anchor points (anchorpoints) and free parameters. The transfer function has a mid-range slope controlled by the free parameter. The transformation at the anchor point is not affected by the free parameters. Such an apparatus may be useful, for example, for transforming color graded content for display on a particular target display.

In some embodiments, the apparatus comprises or receives signals from an ambient light sensor, the circuitry connected to receive the ambient illumination signals from the ambient light sensor being configured to control one or more of the coordinates and the free parameters of one of the anchor points based at least in part on the ambient illumination signals.

In some embodiments, the image data includes a set of pixel values for pixels in the image. The set of pixel values includes a color value for each of a plurality of primary colors (e.g., values corresponding to red, green, and blue primary colors). The apparatus includes a plurality of pixel coordinate mapping units, each pixel coordinate mapping unit connected to transform a corresponding one of the color values. The parameters for the transfer functions in different coordinate mapping units may be the same or different. The transformation may perform color correction as well as gamut conversion by appropriately selecting different parameters.

Another aspect of the invention includes a method for mapping image data for display on a target display. These methods include transforming pixel values of image data into corresponding transformed pixel values according to a transfer function. The transfer function is characterized by a plurality of anchor points and free parameters. The transfer function has a mid-range slope controlled by a free parameter. The transformation of the pixel values corresponding to the anchor points is not affected by the free parameters. In some embodiments, one or more of the free parameters and the location of the mid-range anchor point are automatically changed to account for the adaptation of the ambient lighting and/or the viewer's visual system.

Another aspect includes a method for mapping image data for display on a target display by combining a global tone mapping transformation with a local multi-scale tone mapping operation.

Another aspect provides a color manipulation device. The apparatus may comprise a workstation for modifying, for example, a still image or a video image. The apparatus may be used to modify color values in image data. The color manipulation device comprises a first memory or input for source image data and a second memory or output for modified image data. The pixel coordinate mapping unit is connected to access a first memory or input and is configured to transform the source image data according to a transfer function characterized by a plurality of anchor points and free parameters. The transfer function has a mid-range slope controlled by a free parameter, wherein the transformation at the anchor point is unaffected by the free parameter, resulting in modified image data and providing the modified image data to a second memory or output. The user input is configured to accept a value of a free parameter from a user. The display is connected to display the modified image data. The user may adjust the value of the free parameter to obtain a desired appearance of the image displayed on the display.

Further aspects of the invention and features of particular embodiments of the invention are described below.

Drawings

The drawings show non-limiting embodiments of the invention.

Fig. 1 is a schematic depiction of a video distribution pipeline.

Fig. 2 shows an apparatus according to an example embodiment of the invention.

Fig. 3 illustrates an example transfer function.

FIG. 4 is a flow chart illustrating a method of determining appropriate values for parameters defining a transfer function using information about a target display and input image data.

Fig. 5 is a flowchart illustrating a method for processing image data according to an example embodiment.

Fig. 6 depicts a flow diagram of a method of combining a global tone mapping operator with a local multi-scale tone mapping operator according to an example embodiment.

Detailed Description

Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the present invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Fig. 1 schematically shows a video distribution pipeline 20. The raw video data 22 is captured and edited at an editing suite 23 to provide a raw video work 24. A colorist (e.g., a person using tools provided by the color mediation station through a suitable user interface) adjusts the hue and/or color in the original video work at the color mediation station 26 to achieve a color mediated video work 27. The color mediation station 26 includes a professional monitor 30 on which the colorist views the video work. Using the tools and controls provided by the color mediation station 26, the colorist adjusts the hue and/or color of all or a portion of the images making up the video work to achieve an overall appearance that matches the colorist's artistic intent when viewed on the display 30.

If all viewers of the color mediated video work 27 view the video work on the same display as the display 30, under the same ambient conditions as those experienced by the colorist, they will see the video work exactly as the colorist's intent (i.e., in a manner that is faithful to the colorist's artistic intent), except for the individual differences in human perception of the image. Given the wide range of displays in use, it is not practical to expect that viewers will all have the same display or even that displays on which different viewers will view a video work will have similar characteristics, such as maximum brightness, black level and color gamut.

One aspect of the present invention provides a mapping method and apparatus that may be used automatically to map tones and/or colors from image data (such as, for example, color mediated video work 27) for display on a particular target display in a manner that closely replicates the viewing experience of a colorist.

In some embodiments, these mapping methods and apparatus directly control one or more of the following:

average image brightness (change point);

-a mid-tone local contrast;

-a color saturation;

-displaying a level of input black; and

-displaying the level of input white.

These parameters affect the viewing experience.

Fig. 2 shows an apparatus 40 according to an example embodiment of the invention. In this example, device 40 has an input 42 for receiving video data 43 to be displayed on a screen 44 of target display 41 for viewing by viewer V. Video data 43 may include color-mediated video data embodying the author's intent. Apparatus 40 includes a color space converter 46 that converts pixel values of video data 43 into a color space local to the target display (native). In the illustrated example embodiment, the local color space of the target display 41 is an RGB color space, which specifies colors according to the intensities of the primary colors of the target display 31.

Color space converter 46 may include, for example, a matrix multiplier that multiplies vector 43 of pixel values in video data 43 by a 3 x 3 matrix to obtain a vector of local color space values (e.g., RGB values) for display 41. The transfer matrix may be specified taking into account the primary colors and white point of the target display 41. In some embodiments, the color space converter 46 may be configured to apply the color space transformation matrix without scaling the peak luminance. This may make the selection of parameters for subsequent image processing operations more intuitive, as explained below.

In the following example, the pixel values in the video data 43 are represented in the XYZ color space, and the color space converter 46 performs conversion from the XYZ color space to positive RGB values. The present invention is not limited to color data presented in the XYZ color space. Video data 43 may be presented in any suitable color space.

A conversion for a combination of pixel values that is out of gamut (e.g., a color that cannot be rendered using any available combination of primary colors used by the display) may result in negative RGB values. Any negative RGB values generated by the color space converter 46 may be clamped to low non-negative values. In the alternative, out-of-gamut pixel values may be mapped to in-gamut pixel values prior to conversion (e.g., according to a mapping within the color space of video data 43). This may be performed, for example, by a separate mapping unit or a component of the color space converter 46.

After being processed by color space converter 46, video data 43 includes values 48R, 48G, and 48B corresponding to the red, green, and blue (RGB) primaries of target display 41, respectively.

The values 48R, 48G, and 48B are each independently mapped to new values by the mapping unit 50. Mapping units 50R, 50G and 50B are shown. Each mapping unit maps the corresponding input values received from the color space converter 46 to transformed values. In the illustrated embodiment, the transformed values are indicated by 48R ', 48G ' and 48B ', respectively.

Each mapping unit 50 maps its input values to output values according to a transfer function 55. Advantageously, the transfer function(s) 55 may be characterized by a free parameter that adjusts the slope of the transfer function in the mid-range region and a plurality of fixed points, which may be referred to as "anchor points". This slope corresponds to mid-range contrast. Adjustment of the free parameters provides a means for controlling mid-range contrast. The transfer function may be linear or nearly linear in the mid-range region.

Fig. 3 shows an example transfer function. In fig. 3, the input values are indicated on the horizontal axis, and the output values are indicated on the vertical axis. Each axis has a logarithmic scale. The transfer function 55 is characterized by a maximum value 56A of the output values, a minimum value 56B of the output values, and a substantially linear mid-tone region 56C. The transfer functions 55R, 55G and 55B applied to the red, blue and green channel signals by the mapping units 50A, 50B and 50C may be the same or different. The mapping units 50A, 50B, and 50C may be completely independent or may share hardware and/or software components.

In an example embodiment, the transfer function 55 is given by:

wherein, C₁、C₂And C₃Is a constant, V is the input value of the color channel, V' is the output value of the color channel, and n is a parameter. The transfer function of equation (1) is an example of a parameterized sigmoidal tone curve function.

Other parameterized transfer functions may be used in the alternative. In some embodiments, the transfer function includes parameters that provide control over one or more of the following for the transfer function: the low-end slope, the high-end slope, and the "sharpness" of the attenuation (roll-off) at the top and bottom ends.

One method for creating the value of the parameter in equation (1) in a particular situation is illustrated by method 70 of FIG. 4. The method 70 uses information about the target display and information about the display used for color mediation or approval (approving) of the input video data ("color mediation display") to determine the appropriate values for the parameters of equation (1). Block 71 identifies three luminance anchor points on curve 55. The first anchor point 57A has a horizontal coordinate and a vertical coordinate equal to the black level of the color mediation display and the target display, respectively. In some embodiments, information about the color-mediated display is inferred from the input signal. For example, the black level of a color-mediated display may be inferred from the input signal by taking a small percentile (e.g., a 0.1 percentile) of the luminance channels in the input signal. The black level of the target display is the black level of the target display.

The horizontal coordinate of the second anchor point 57B is the white level of the color-mediating display and its vertical coordinate is the white point of the target display. For example, the white point of a color-mediated display may be inferred from the input signal as the maximum value of any color channel in the input signal.

The location of the intermediate anchor point 57C affects the overall brightness of the displayed image (e.g., "key" of the image). Proper selection of the mid-tone anchor point 57C helps the input image to be perceived as suitably bright on the target display.

The horizontal position of point 57C may be set in various ways; these ways include the following:

calculating the geometric mean of the input luminance;

selecting a fixed value that will be perceived as a suitable intermediate value in a color grading environment. For example, in some embodiments, this value may be set to a level such as 10.

The vertical value of point 57C may be based on a brightness level (luminancevelel) corresponding to the mid-gray of the target display. For example, 1cd/m can be generated²And 400cd/m²Middle gray scale of about 20cd/m in a display with intermediate brightness values²(20cd/m²Logarithmically between 1 and 400cd/m²Middle between). A suitable value for point 57C may therefore be about mid-gray (in this example, about 20 cd/m)²) The corresponding value. In embodiments where the color space transformer 46 is configured to apply a color space transformation matrix that is not scaled for peak luminance, the value 20 will correspond to a mid-gray of 20cd/m²。

In some embodiments, mid-tone anchor point 57C is selected such that the ratio of the coordinates of the mid-tone anchor point to the coordinates of the white anchor point is equal, within a desired factor, for both input and output of the transfer function.

In some embodiments, a different transfer function for each of the RGB coordinates may be used to provide a transformation such that the white point of the video data is transformed to match the white point of the target viewing environment and/or target display. One way to achieve this is to express the white point of the input video data in terms of chromaticity coordinates (such as, for example, CIEx, y chromaticity coordinates) and convert to scaled XYZ values given by:

Y＝1(3)

these XYZ values can then be converted to the RGB color space of the target display to yield what can be represented as (R, G, B)_wp,inWhite point of the input data. In the case where the source and target white points are the same, the normalized RGB coordinates for these two white points should be (111). The luminance anchor values may then be multiplied by the white point values to obtain the coordinates of anchor points 57A, 57B, 57C for the red, blue and green channels as follows:

(R，G，B)_min，in＝Y_min，in(R，G，B)_wp.in(5)

(R，G，B)_max，in＝Y_max，in(R，G，B)_wp.in(6)

(R，G，B)_mid，in＝Y_mid，in(R，G，B)_wp.in(7)

(R，G，B)_min，out＝Y_min，out(R，G，B)_wp.out(8)

(R，G，B)_mid，out＝Y_mid，out(R，G，B)_wp.out(9)

(R，G，B)_max，out＝Y_max，out(R，G，B)_wp.out(10)

where the subscript "in" represents the input image data and the subscript "out" represents the output data (i.e., the data passed for display); (Y)_max,in,Y_max,out) Is the coordinates of unadjusted anchor point 57B; (Y)_min,in,Y_min,out) Is the coordinates of unadjusted anchor point 57A; (Y)_mid,in,Y_mid,out) Is the coordinates of unadjusted anchor point 57C; (R, G, B)_wp,outIs the RGB coordinates of the white point of the target display.

Equations (5) through (10) provide a set of three anchor points for each color channel. For example, anchor point 57A for the red channel is composed of (R)_max,in,R_max,out) Giving out; anchor point 57B for the red channel consists of (R)_min,in,R_min,out) Giving out; anchor point 57C for the red channel consists of (R)_mid,in,R_mid,out) It is given. In case the white point of the input video data and the target display are not the same, the set of anchor points will be different, which results in different transfer functions for each color channel.

The transfer function for each color channel in the form provided by equation (1) may be obtained from the coordinates of the corresponding anchor point by performing the following calculation:

wherein x is₁、x₂And x₃Given by:

and y is₁、y₂And y₃Given by:

one feature of the above transfer function is that n remains a free parameter. This allows the mid-tone contrast to be set to any desired level. It should be noted that if the mid-tone anchor point is not centered in the input range and the output range, the log-log slope at the mid-tone anchor point will be slightly different from the value of n. However, the halftone contrast may be set by adjusting the value of n. A good starting point for the mid-tone contrast parameter n is 1. This n-value ensures that the mapped scene has a substantially similar mid-range local contrast on the target display and in the original scene.

With a transfer function as given above, the display linear luminance value for each of the red, green and blue channels can be expressed as follows:

these values may be used to drive the target display to display an image. In some embodiments, these values may be corrected for the response of the target display to linear input values (e.g., normalized values) before driving the target display using the values.

In some embodiments, the following relationship is used to calculate the normalized drive value (R) for the target display_norm,G_norm,B_norm)：

The normalized values may be scaled to a range of drive signals for the target display (e.g., to a range of 0-255 for an 8-bit target display).

Alternatively, image color may be enhanced by increasing color saturation. This can be done, for example, using the following relationship:

reference and inverse color space converter 46([ X, Y, Z)]^T＝M*[R,G,B]) The corresponding elements of the inverse transform matrix M define the values of a, b, and c in equation (20), specifically, a may be given by a ═ M (2,1), b may be given by b ═ M (2,2), and c may be given by c ═ M (2, 3). In the formulae (20), (21) and (22), S is a free parameter. A value of S greater than 1 will result in an increase in color saturation. A value of S less than 1 will cause the color saturation to decrease (e.g., will cause the color to become more desaturated).

The normalized drive values may be gamma corrected, if necessary or desired. This can be done, for example, according to the following relationship:

where γ is the display response. Gamma is about 2.2 in some target displays. In the case of re-saturating the normalized drive values, gamma correction may be performed on the re-saturated drive values (R ', G ', and B ').

In some embodiments, the image is re-saturated in color to at least approximately restore the saturation lost due to the tone compression. In the case where the tone compression is not constant over the range of tones in the image, different levels of tone compression applied to different tones results in different colors being desaturated to different degrees. Generally, the greater the amount of tone compression, the greater the amount of desaturation. The amount of tone compression may be quantified by the log-log slope of the tone curve. As an illustrative example, the sigmoidal tone curve function, as plotted by curve 55 in FIG. 3, has a steeper log-log slope substantially in the substantially linear mid-tone region 56C than its log-log slope near the maxima 56A and minima 56B. Thus, the compression of the hue from input (horizontal coordinate) to output (vertical coordinate) is greater around the values 56A and 56B than for the substantially linear mid-tone region 56C.

Applying the global re-saturation technique may re-saturate all pixels without considering the amount of de-saturation caused by the tone compression. Some embodiments re-saturate the transformed image data pixels according to an amount of tone compression of the transformed image data pixels. Assuming that the amount of tone compression corresponds to the log-log slope of the tone curve, one would apply to the input value L_inIs determined as a transfer function L_out＝f(L_in) At the input value L_inThe derivative of (c). Can be set by setting L_in＝e^xAnd L_out＝e^yAnd solved for dy/dx, which represents the log-log slope, to determine the log-log slope of this transfer function. For the tone curve according to the above equation (1),

y can be expressed as:

y＝log(c₁+c₂e^nx)-log(1+c₃e^nx)(26)

and the log-log slope c (L) at any point on the tone curve_in) Can be calculated as being at L_inDerivative of y with respect to x:

for color channels R, G and B, the re-saturated drive value (R) may be determined from the normalized drive values as follows_re-sat,G_re-sat,B_re-sat)：

Wherein f (c) is given as:

and k is₁And k₂Is a constant. In some embodiments, k is₁1.6774. In some embodiments, k is₁1.677. In some embodiments, k is₁1.68. In some embodiments (including, but not limited to, where k is₁＝1.6774，k₁1.677 or k₁Some embodiments of 1.68), k₂0.9925. In some embodiments (including but not limited to where k is₁＝1.6774，k₁1.677 or k₁Some embodiments of 1.68), k₂0.992. In some embodiments (including, but not limited to, where k is₁＝1.6774，k₁1.677 or k₁Some embodiments of 1.68), k₂0.99. It will be appreciated that k is used₁And k₂Other values of (a) may yield acceptable results. It will also be appreciated that the linear luminance value (R) may be based on the display of each of the red, green and blue channels_out、G_outAnd B_out) To calculate the driving value R for re-saturation_re-sat、G_re-satAnd B_re-sat。

It will be appreciated that the above-described techniques for hue compression related re-saturation may be implemented in a parameter-free (automatic) manner.

Fig. 5 is a flow chart illustrating a method 80 according to another example embodiment. The method 80 incorporates several optional steps. At block 81, method 80 converts the image data into the color space of the target display. In the example shown, the target display has the primary colors red, green and blue, and the color space is an RGB color space. Block 81 may include performing a transformation that takes into account the white point and primary colors of the target display. Block 81 is optional in the case where the image data is already in the local color space of the target display.

Block 82 determines the chromaticity white points of the source and target. The white point may be represented, for example, as chromaticity coordinates in any suitable color space, and converted to the local color space of the target display.

Block 83 establishes an initial black level anchor point and white level anchor point for the transfer function. The initial anchor point may be set based on the black and white levels of the source display and the black and white levels of the target display.

Block 84 establishes an initial mid-tone anchor point for the transfer function. The mid-tone anchor point may be determined by analyzing the source image data (e.g., determining a geometric mean of the luminance of the source image data) and determining characteristics of the target display (or characteristics of the target display and the current viewing environment at the target display).

Block 85 adjusts the anchor point based on the white point determined in block 82 (application example is as in equations (5) to (10)).

Block 86 maps the image data using the transfer function specified by the adjusted anchor point determined in block 85.

Block 87 calculates drive values for the target display based on the mapped image data from block 86.

Optional block 88 adjusts the color saturation (block 88 may apply equations (20) through (22) or (28) through (30), for example).

A block 89 gamma corrects the drive values.

The drive values generated by applying the method 80 may be used to drive the target display to display an image on the target display and/or may be stored or transmitted for later display on the target display.

The apparatus and methods as described herein may be used to optimize a target display for a particular ambient viewing condition. A transfer function of the general type described above may be dynamically offset to accommodate changes in ambient lighting and the resulting level of Human Visual System (HVS) adaptability. The ideal luminance midpoint of the target display may be a function of the ambient light. The vertical component of the mid-tone anchor point may be selected based on ambient lighting conditions.

In some embodiments, the operation of fixing the intermediate anchor point 57C is done based in part on the ambient lighting or the estimated adaptability of the viewer's eyes (which adaptability itself may be based at least in part on the measured ambient lighting or a combination of the measured ambient lighting and past display content) and the characteristics of the target display. For example, the vertical coordinate of point 57C may be adjusted based on ambient lighting near the target display. For example, if the display is in a dark ambient lighting condition (or the viewer's eyes are estimated to be dark-adapted), the vertical coordinate may be reduced to a lower luminance value, and the value may be increased to a higher value if the target display is in an environment with high ambient lighting (or the viewer's eyes are estimated to be adapted to a brighter condition).

In some embodiments, the amount of saturation adjustment (e.g., according to equations (20), (21), and (22) and equations (28), (29), and (30)) is based at least in part on the ambient lighting or on an estimated adaptability of the viewer's eyes (which adaptability itself may be based at least in part on the measured ambient lighting or on a combination of the measured ambient lighting and past display content) and characteristics of the target display. For example, the parameter S may be adjusted based on ambient lighting near the target display or a combination of measured ambient lighting and past display content. For example, if the display is in a dark ambient lighting condition (or the viewer 'S eyes are estimated to be dark-adapted), the value of the parameter S may be set relatively low, and in case the target display is in an environment with high ambient lighting (or the viewer' S eyes are estimated to be adapted to a brighter condition), the value may be set relatively high. Some embodiments provide a re-saturation control unit that receives signals from ambient light sensors and/or signals containing past image content and/or signals indicating overall brightness of past display content. The re-saturation control unit may be configured to set a new value of a parameter (e.g., parameter S) that affects the amount of re-saturation based on the received signal (S).

In some embodiments, the spectral characteristics of the ambient lighting are taken into account. For example, the location of point 57C in the transfer function for each color channel may be individually set based at least in part on the amount of ambient illumination in the spectral range corresponding to the color channel.

Additionally or in the alternative, the slope of the transfer function may be controlled based on ambient lighting (or an estimate of the adaptability of the eyes of the viewer). In the case of brighter ambient light, the reflection from the display surface tends to increase the black level. This effectively narrows the range of the target display. In a condition where the ambient lighting is high (the viewer's eyes are estimated to be brightly adapted), the slope of the transfer curve in the mid-tone region may be reduced to provide an enhanced viewing experience in this ambient condition. For example, for conditions where the ambient lighting is low (dark), the perception of contrast is reduced. This may result in the image appearing "flat". Thus, the slope of the mid-tone portion of the transfer function may be increased from a slope of 1:1 to a greater slope (e.g., a slope of 1.3), such as a slope up to about 1:1.5, to improve the contrast level for dark-adapted eyes. In case a transfer function of the type shown in equation (1) is applied, this can be done by changing the value of the free parameter n. The slope may be controlled in response to input from an ambient light sensor.

In some embodiments, a light adaptation circuit is provided that estimates the level of adaptation of the human visual system in response to inputs, which may include signals from ambient light sensors, signals representing a weighted average or other indicator of the brightness of historical image content, or the like. The light adaptation circuit may be based on, for example, a model of the human visual system. Various algorithms for estimating the level of adaptability of the human visual system are known in the art. The light adaptation circuit may implement such algorithms in any suitable manner, including software executing on one or more programmable data processors, fixed logic circuitry, or a combination thereof. The value of the mid-tone contrast in the transfer function and/or the position of the dot 57C may be automatically controlled in response to the output of the light adaptation circuit.

In some embodiments, the transfer function may be set once for the target display. The transfer function may for example be built into the target display and implemented in the following form: one or more programmable processors executing firmware or other software that performs mapping according to the transfer functions described above; a lookup table for implementing the transfer function; a hard-wired or configurable logic circuit arranged to provide an output based on a transfer function as described above; and so on.

In some embodiments, the drive values for the red, green, and blue channels of the target display may be converted to a bit depth that matches the bit depth of the display. For example, the display may use an 8-bit drive value. If the transfer function is applied using floating point calculations or other higher precision calculations, the conversion may involve, for example, rounding the drive values to the nearest corresponding 8-bit value.

In the foregoing embodiments, the maximum luminance value and the minimum luminance value of the input video data may be mapped to the maximum luminance value and the minimum luminance value of the pixels of the display, respectively. Further, the selected halftone dots from the input video signal may be mapped to selected halftone dots for the display. The mid-tone contrast remains a free parameter. Another feature of the above transfer functions is that they provide compression or expansion for both low and high values while preserving local contrast in the mid-tone range.

In some embodiments, the average brightness of a particular image (e.g., a particular video frame or sequence of video frames) is relatively low (base dimming), while the average brightness of other images (e.g., frames or groups of frames) may be intentionally made relatively high (base dimming). In some embodiments, information about the expected mood of the image may be provided in the form of metadata. The metadata may be created and associated with the image data, for example, during a color grading operation. For example, the metadata may be embedded in or otherwise associated with a signal carrying color-graded video data. In such embodiments, the mood of the image as indicated by the metadata may be used in determining the mid-tone anchor point(s) used in the transfer function. In the case of an image where the metadata indicates a low key, the vertical coordinate of the anchor point may be moved to a lower value, thereby recreating the key in the target display.

Different video content may be color graded for different reference displays. When employing the above approach, it may be desirable to map content differently in any particular target display according to the characteristics of the reference display on which color grading is performed. Information identifying the reference display or a characteristic thereof may be carried, for example, in metadata embedded in or otherwise associated with the image data. The target display may store a plurality of different sets of parameters of the transfer function. Different sets of transfer functions may correspond to and be available for video data that has been color adjusted using different reference displays.

Another feature of an example transfer function having the form provided by equation (1) is that the same transfer function may provide compression or expansion at the high and low ends of the range depending on the selected parameters. For example, where the target display has a larger luminance range than the input data, the target display may be configured with transfer functions that extend the range of the image data to match or more closely approximate the range of the image data of the target display.

One advantage of the methods and apparatus according to some embodiments described herein is that the mapping is performed in the RGB color space of the target display. This may save a very large amount of computation and/or reduce the complexity of the hardware needed to perform the mapping.

The mapping may be performed in real time.

The method according to some embodiments provides direct control for each of the following aspects: 1) average image brightness ("adaptation point"), 2) mid-tone local contrast (set by the tone curve slope), 3) input black mapping to minimum display brightness, and 4) input white mapping to maximum display brightness. It has been found that these variables are essential for providing an image that reproduces the creative intent as embodied in the original image data. In an example embodiment, these variables explicitly correspond to independent parameters. Such a method results in providing a simple and efficient way to perform color mapping that takes raw image data (which may, for example, comprise High Dynamic Range (HDR) data and/or color graded image data) and maps it to a limited 3-dimensional color gamut of a specified output display.

The color mapping methods and apparatus as described herein may also be used for color grading/content authoring, or may be used in the alternative for color grading/content authoring. The colorist may be provided with filters that implement the transformations described above. The filter may have controls that allow the colorist to directly set the parameters of the transfer function. The colorist may use these controls, for example, to adjust the black level, etc. In some embodiments, these controls include controls that allow one or more of the following aspects to be set directly: one or more coordinates of one or more of a white level anchor point, a black level anchor point, and an intermediate level anchor point (e.g., points 57A, 57B, and 57C, respectively), and an intermediate level contrast (e.g., parameter n). Such control may allow the colorist to set the white level, black level, and key without significantly affecting the halftone slope, and vice versa.

In some embodiments, the apparatus is arranged to automatically determine a starting set of parameters that is close to the intentions of the colorist. These starting parameters may be generated, for example, based on information characterizing the input video content (e.g., minimum and maximum values of pixel color/brightness coordinates) and information characterizing the target display (e.g., white level, black level, and optionally metadata (e.g., metadata indicating the mood of the image data being processed)).

Video production involves authoring different versions for displays with greater and lesser capabilities. For example, Standard Dynamic Range (SDR) ratings may be performed to generate video for display on legacy displays. Tools as described herein may be used to automatically create an SDR version of a video. The colorist may direct the operation of the tool to generate optimized results.

Further, where the colorist has set parameters for providing a version for viewing on the lower-capability display, the parameters used in performing the mapping for the display having the intermediate capability may be determined from the parameter values selected by the colorist for the lower-capability display. This may be done, for example, by interpolating the parameter values established by the colorist for displays having higher and lower capabilities than displays having intermediate capabilities.

The methods and apparatus described herein are not limited to use in connection with professional level color mediation. Tools for color mediation are available to amateurs and, even where color mediation is performed on an uncalibrated monitor (e.g., a home computer display, a television, etc.), the methods and apparatus described herein can be used to translate content authored on the uncalibrated monitor to another display (e.g., by evaluating the capabilities of the uncalibrated color mediation display). The techniques as described herein may also be applied to signals that are not color-mediated.

Combining a global tone mapping operator with a local multi-scale tone mapping operator

The color mapping methods and apparatus as described herein may also be combined with other tone mapping techniques, such as a local Tone Mapping Operator (TMO). Fig. 6 depicts an example embodiment of a global tone mapping operator combined with a local multi-scale tone mapping operator as described herein, such as the local multi-scale tone mapping operator described in g.j.ward, U.S. provisional application 61/448,606 filed on 3/1/2012, also filed as international patent application No. pct/US2012/027267, "alocal multiscale tone-mapping operator" (referred to herein as the "Ward" reference), the entire contents of which are incorporated herein by reference. This example embodiment combines the predictability and stability of global TMO with the ability to preserve highlight portions and color fidelity when using local multiscale operators (MSTMO).

As depicted in fig. 6, the method 60 begins in step 62 with accessing input image or video data. These data may be stored or transmitted in various color formats (such as YCbCr, RGB, XYZ, etc.). In step 63, a luminance component (e.g., Y) may be extracted from the input data. Depending on the format of the input data (e.g., RGB), this step may require a color transformation (e.g., RGB to XYZ). Step 64 may apply global tone mapping operator 55 as described in equation (1) to the Y color component of the input data. In an example embodiment, anchor points of global TMO55 may be selected such that luminance ranges of input data are mapped to the range [4 × L_dmin,1/2*L_dmax]Wherein L is_dminAnd L_dmaxRepresenting the minimum and maximum brightness of the target display. Scaling factors 4 and 1/2 are typical, but adjustable.

In an example embodiment, the output of step 64 may be represented as global tone mapped luminance data Y_TM. In step 65, a local multiscale tone mapping operator (MSTMO) as described by "Ward" may be applied to Y_TMAnd (4) data. For example, first, a global logarithmic ratio image may be calculated

The global log ratio image is defined as the log of the global tone mapped luminance data divided by the original luminance pixels. At a given global logarithmic ratio image R_LIn this case, the output of the MSTMO (e.g., step 65) may be represented as Y, as described by Ward_MSThe local tone mapped luminance image of (a). By using Y_MSCan calculate

And is

In step 66, X may be added_MS、Y_MSAnd Z_MSConvert back to R with primary color, white level, and black level determined by the target display_MS、G_MSAnd B_MS(RGB_MS). Negative or out-of-gamut RGB can be used_MSThe values are clamped to very small positive values, or negative or out-of-gamut RGB may be mapped by using any one of the known gamut mapping algorithms_MSThe values are remapped to RGB values within the gamut.

RGB within the given gamut from step 66_MSIn this case, step 67 may reapply the global tone mapping operator 55 to all color components to output the global tone mapped correction data RGB_G-MS. Application of the second global tone mapping operation ensures that the output of the MSTMO is within the range of the target display. Finally, in step 68, the RGB may be paired for the output display as needed before displaying the image data (step 68)_G-MSThe data is gamma corrected.

Some implementations of the invention include computer processors executing software instructions that cause the processors to perform the methods of the invention. For example, one or more processors in a display, color grading station, set-top box, decoder, etc. may implement the image data transformation methods described above by executing software instructions in program memory accessible to those processors. The present invention may also be provided as a program product. The program product may comprise any medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to carry out the method of the invention. The program product according to the invention may be in any of a number of forms. The program product may include, for example, physical media such as magnetic data storage media (including floppy disks, hard drives), optical data storage media (including CDROMs, DVDs), electronic data storage media (including ROMs, flash RAMs), and so forth. Optionally, the computer readable signal on the program product may be compressed or encrypted.

Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is discussed above, unless otherwise indicated, a discussion of that component (including a discussion of a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (i.e., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated exemplary embodiments of the invention.

Some non-limiting embodiments may provide (e.g., as the case may be) one or more of the following advantages:

mapping according to a tone mapping curve with black anchor points may avoid excessive tone compression of dark input content;

mapping according to a tone mapping curve with black and/or white anchor points may make more use of the luminance range of the target display than tone mapping according to a curve without one or both of such anchor points;

a color channel specific mapping function that maximizes the luminance range may be applied in the RGB color space of the target display (e.g., after conversion from the input color space to the RGB color space of the target display); and

the white point, brightness, and/or intermediate contrast of the output video data of the target display may be adjusted in the transfer function (e.g., rather than before or after mapping according to the transfer function).

Some embodiments may not provide any of the above advantages; some embodiments may provide different advantages (e.g., advantages different from or in addition to the above advantages)

As will be apparent to those skilled in the art in the light of the foregoing disclosure, many alterations and modifications are possible in the practice of this invention without departing from the spirit or scope thereof. The scope of the invention should, therefore, be construed in accordance with the substance defined by the following claims.

Claims (12)

1. An apparatus for display management of an image, the apparatus comprising:

an input section for receiving an input image signal;

an input for accessing first information data of a reference display, the first information data including a black point level, a white point level, and an intermediate point level of the reference display;

an input for accessing second information data of the target display, the second information data including a black point level, a white point level, and an intermediate point level of the target display;

a processor for determining a transfer function mapping pixel values of an input image signal to corresponding pixel values of an output image signal using the first information data and the second information data, wherein the transfer function comprises three anchor points, wherein a first anchor point is determined using a black point level of the reference display and a black point level of the target display, a second anchor point is determined using a white point level of the reference display and a white point level of the target display, and a third anchor point is determined using an intermediate point level of the reference display and an intermediate point level of the target display;

a processor for mapping the input image signal to the output image signal using the determined transfer function; and

an output section for outputting the generated output image signal.

2. The apparatus of claim 1, wherein the processor for determining the transfer function is further configured to apply a free parameter, wherein the free parameter adjusts a slope of the transfer function at the third anchor point.

3. The apparatus according to claim 1, wherein the first information data is received as part of metadata of the input image signal.

4. The apparatus of claim 3, wherein the first information data is determined for a scene of the input image signal.

5. The apparatus of claim 1, further comprising a processor for generating the first information data based on a characteristic of the input image signal.

6. The apparatus of claim 5, wherein the black dots of the reference display are determined as small percentiles of luminance channels of the input signal.

7. The apparatus of claim 5, wherein a white point of the reference display is determined as a maximum value of any color channel of the input signal.

8. The apparatus of claim 1, wherein the first anchor point has horizontal and vertical coordinates equal to a black point level of the reference display and a black point level of the target display, respectively, the second anchor point has horizontal and vertical coordinates equal to a white point level of the reference display and a white point level of the target display, respectively, and the third anchor point has horizontal and vertical coordinates equal to an intermediate point level of the reference display and an intermediate point level of the target display, respectively.

9. The apparatus of claim 1, wherein the transfer function comprises a transformation according to:

where V is the input pixel value, V' is the output pixel value, C₁、C₂And C₃Is a parameter determined using the three anchor points, and n is a free parameter.

10. The apparatus of claim 1, wherein the pixel values of the input image comprise color values of two or more color components, and the transfer function is determined for each of the two or more color components.

11. The apparatus of claim 9, wherein the free parameter is determined from a slope of the transfer function in the mid-range region.

12. The apparatus of claim 9, wherein the free parameter is determined according to a desired mid-range contrast, independent of the determined anchor point.