WO2014118032A1

WO2014118032A1 - Method and device for modifying the dynamic range of an image

Info

Publication number: WO2014118032A1
Application number: PCT/EP2014/051116
Authority: WO
Inventors: Dominique Thoreau; Kadi BOUATOUCH; Rémi COZOT; Ronan BOITARD
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2013-01-29
Filing date: 2014-01-21
Publication date: 2014-08-07
Anticipated expiration: 2015-07-29

Abstract

The invention relates to a method and device for modifying the dynamic range of an image (IM) by multiplying the luminance values of the pixels of the image by a modification coefficient determined from a value, called a key, which defines an indication of the brightness of the pixels. The method is characterised in that it comprises the following steps: - obtaining (10) at least one image region (Rj), - associating (20) a key (kj) with each image region, said key defining an indication of the brightness of the pixels belonging to this image region, and - modifying (30) the dynamic range of the pixels of at least one image region by multiplying the luminance values of these pixels by a coefficient determined from the key thus associated with this image region.

Description

METHOD AND DEVICE FOR MODIFYING THE DYNAMIC RANGE

OF AN IMAGE

1 . Field of the invention

The invention relates to the general field of modifying the dynamic range of an image.

The invention relates to a device and a method for modifying the dynamic range of an image. This device and method can be used, for example, to reduce the dynamic range of an image, that is to say to modify the luminance values of the pixels of this image which belong to a given dynamic value range so as to obtain luminance values which belong to a lower dynamic value range that the initial image.

2. Prior art

It is known to use tone mapping operators (TMOs) or tone reproducers to modify the dynamic range of an image called the original image which can be, for example, acquired by a high dynamic range camera so as to obtain an image whose dynamic range is lower (a low dynamic range image) so as to adapt the dynamic range of the original image to that, for example, of a screen on which this image is displayed. If the adapted original image is intended for a display system, the luminance component of this adapted image is quantised and encoded so as to be compatible with a display standard (BT 709, etc.). In this case, we usually refer to luma components rather than luminance components. The luminance, for its part, corresponds to a physical unit expressed in cd/m². The invention is equally applicable to a luminance component and a luma component.

One of these TMOs is that developed by Reinhard which is commonly called a PTR operator (Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J., \Photographic tone reproduction for digital images, " ACM Transactions on Graphics 21 (July 2002)) .

The principle of this operator is to modify a luminance component L_w of an original image so as to obtain a modified luminance component L_d of the image by using a sigmoid mapping curve given by equation (1 ): Ld =— . (l + ^-) (1 )

1+L_S ^Lwhite

where L_white is a luminance value used to ignore zones with high luminance values, L_d is a matrix whose size is that of the image and which comprises the luminance values of the pixels of the image which are expressed in a lower dynamic value range than that of the original image and L_s is a matrix whose size is that of the image and which comprises the luminance values obtained by equation (2):

L_s = -_k . L_w (2)

where a is a chosen exposure value, and k, commonly called a key, defines an indication of the brightness of the image given by equation (3):

k = exp ( .∑ ₁ log(5 + L_w(i))) (3)

where N is the number of pixels in the image, 5 is a value which avoids any singularity and L_w(i) is the luminance value of a pixel i of the luminance component L_w of the image.

The values a and L_white are two parameters of this TMO which are fixed, for example, at 18% for parameter a and at the maximum luminance value of the original image for parameter L_white.

An extension of this operator makes it possible to avoid using a matching curve and instead to modify each luminance component Lw of an original image so as to obtain a modified luminance component L_d of the image by using a weighted average over the spatial neighbourhood of each pixel. For this purpose, a pyramid, called a Gaussian pyramid, is created. A Gaussian pyramid corresponds to such a weighted average. Its successive application corresponds to doubling the size of the spatial neighbourhood. It is defined by the equation:

L, = L_s_, ® G_a (4)

where s is the index of the current level of the pyramid and L is a matrix whose size is that of the image and which comprises the averaged luminance values. G corresponds to the weighting applied to each pixel during the averaging.

A difference of Gaussian (DoG) is thus defined by:

H_S = L - L _, (5) where Hs is the difference of Gaussian and s is the index of the neighbourhood size.

The size of the neighbourhood used is determined when the weighted difference of Gaussian exceeds a certain threshold:

where Vs is the weighted difference of Gaussian at level s, a is the exposure chosen in equation (2) and Φ is a parameter which controls the degree of conservation of the fronts.

The modification is therefore made by the equation:

Many TMOs of the prior art use a spatial neighbourhood to modify the dynamic range of a pixel (Kuang, J., Johnson, G. M., & Fairchild, M. D. (2007). iCAM06: A refined image appearance model for HDR image rendering. Journal of Visual Communication and Image Representation, 18(5), 406-414, Li, Y., Sharan, L, & Adelson, E. H. (2005). Compressing and companding high dynamic range images with subband architectures. ACM SIGGRAPH 2005 Papers on - SIGGRAPH Ό5 (p. 836). New York, New York, USA: ACM Press).

These operators give better results than operators defined as global (the dynamic range modification does not depend on the spatial neighbourhood). However, these TMOs of the prior art are not optimal and introduce artefacts into the modified image as they do not respect both the spatial coherence of the brightness of the original image and the brightness contrasts which exist in the original image.

The invention aims to overcome the disadvantages of the prior art. In particular, it aims to define a method for modifying the dynamic image of an image which preserves the spatial coherence of the brightness of the image while preserving the brightness contrasts which exist in the image.

3. Summary of the invention

The purpose of the invention is to overcome at least one of the disadvantages of the prior art.

For this purpose, in a general way, the invention relates to a method for modifying the dynamic range of an image (IM) by multiplying the luminance values of the pixels of the image by a modification coefficient determined from a value, called a key, which defines an indication of the brightness of the pixels. The method is characterised in that it comprises the following steps:

- obtaining at least one image region,

- associating a key with each image region, said key defining an indication of the brightness of the pixels belonging to this image region, and

- modifying the dynamic range of the pixels of each image region by multiplying the luminance values of these pixels by a coefficient determined from the key thus associated with this image region.

Associating a key per image region and not a key with the entire image makes it possible to preserve the brightness contrasts of the original image.

4. List of figures

The invention will be better understood and illustrated by means of non-restrictive embodiments and advantageous implementations, with reference to the accompanying drawings, wherein:

- Figure 1 shows a diagram of the method for modifying the dynamic range of an image according to a preferred embodiment of the invention;

- Figure 2 shows a diagram of an example of a method for segmenting an image into regions;

- Figure 3 shows an example of a histogram of the luminance values of an image;

- Figure 4 shows an example of the internal architecture of a device according to a preferred embodiment of the invention; and - Figure 5 shows a block diagram of a device for converting an image which implements a method according to the invention.

5. Detailed description of the invention

According to one of its aspects shown in Figure 1 , the invention relates to a method for modifying the dynamic range of an image IM by multiplying the luminance values of the pixels of the image by a modification coefficient determined from a value, called a key, which defines an indication of the brightness of the pixels of the image.

The method comprises a step 10 of obtaining at least one image region

Rj.

According to an embodiment of step 10, at least one image region Rj is obtained from a memory. This implies that at least one image region Rj has been previously determined and stored.

Each image region groups together pixels of image IM which are homogeneous in the sense of a criterion. More than one image region can thus be obtained for the image.

According to an embodiment of step 10, at least one image region Rj is determined by image segmentation.

According to an embodiment of step 10, the criterion for homogeneity of the pixels of a same image region is the luminance value of these pixels and the segmentation of the image is thus based on luminance values of the pixels of this image.

Segmenting the image based on the luminance values of the pixels rather than on spatial characteristics of the images such as salience points or other contours ensures that two image portions spatially separated from each other and which have the same intensity will undergo a same dynamic range correction. Moreover, segmentation based on the intensity of the pixels avoids certain imperfections of the spatial segmentation methods such as the occurrence of outlier values in segmented zones which would influence the dynamic range modification.

According to an embodiment of the segmentation of an image based on the luminance values of the pixels, shown in Figures 2 and 3, a histogram H(Lw) of the luminance values (luminance component Lw) of the image to be segmented is calculated (step 1 10) in the logarithmic domain. Local maxima P1 to P4 are then determined (step 1 1 1 ). According to an example, a local maximum is determined when the number of occurrences exceeds a threshold T1 equal, for example, to 5% of the total number of pixels in the image. Those local maxima which are too close to other local maxima in the sense of a metric are removed. For example, when a local maximum (for example P2) is separated from another local maximum (for example P3) by a distance less than a threshold T2 (expressed in the logarithmic scale) then this local maximum (here P2) is removed. Here, the local maxima P1 , P3 and P4 are conserved. Local minima are then determined between each pair of local maxima (step 1 12). Here two local minima are determined, one between the local maxima P1 and P3 and the other between P3 and P4. These local minima define delimitations of image regions Ri. Here, three image regions R1 , R2 and R3 are determined. They are delimited by dashed vertical lines defined, for example, at the first local minimum found between the two local maxima.

The thresholds T1 and T2 are parameters of the segmentation and can be chosen by a user.

The segmentation of the image is not restricted to that above which uses a histogram H(Lw) of the image luminance values. Indeed, it can also extend to any other segmentation which does not use a histogram. As an example, it can also extend to segmentations which use other types of histograms but also to any method which defines region delimiters from local maxima, or which defines local maxima from which are defined the image region delimiters.

The method also comprises, as shown in Fig. 1 , a step 20 of associating a value, called a key kj, with each image region obtained. A key kj associated with an image region defines an indication of the brightness of the pixels belonging to this image region Rj.

According to an embodiment of step 20, a key kj associated with an image region Rj is defined by:

kj = exp ( - . ^ logtS + Z/_W(T))) (9) where Nj is the number of pixels in image region Rj and L^J _w(i) is the luminance value of a pixel of image region Rj originating from luminance component L_w of image IM.

The method also comprises a step 30 of modifying the dynamic range of the pixels of at least one image region by multiplying the luminance values of these pixels by a modification coefficient CRj determined from key kj associated with this image region Rj:

Lj' = CRj. Lj (10)

where Lj is the luminance component of image region Rj of image IM and Lj is the modified luminance component of image region Rj.

The dynamic range of one or all of the regions of an image can thus be modified.

According to a variant, the luminance component L is given by:

Lj' = (x + (1— x). CRj)Lj (1 1 )

where x is an offset in the modification coefficient fixed for example by a user.

This variant makes it possible to modify the gradient of the modification function when the modification coefficient is too small.

According to an embodiment of step 30, shown in Figure 5, it is determined if a pixel belonging to an image region Rj is close, in the sense of a metric, to another image region Rk.

If this is not the case, the dynamic range of the image region Rj is modified by equation (10) or (1 1 ). This is referred to as direct modification.

However, if the pixel is considered as being close to another image region Rk, the modification coefficient CRj,k is then equal to the weighted sum of the modification coefficients relating to each of the image regions Rj and Rk. This is referred to as weighted modification.

In mathematical terms, the modification coefficient CRj,k is given by:

CR_jik = CRj. Wj + CR_k. W_k (12)

where CRj and CR_k are the modification coefficients relating to image regions Rj and Rk and Wj and W_k are weighting coefficients given by:

[ lo_g(L_w)-lo_g(B)l \

W_x = ^ (13) where x designates either index j or index k, N is a normalisation factor and B designates a parameter corresponding to the width of an intersection band situated on either side of a border delimiting image regions Rj and Rk as shown in Figure 3, and σ is

This embodiment makes it possible to enhance the spatial contrasts as the brightest image region then has a modification coefficient equal to 1 .

The methods described in relation to Figures 1 to 3 can, for example, be applied to an image IM originating from a TMO thus used to reduced the dynamic range of an original image IM.

According to an embodiment, the modification coefficient CRj is given by the equation:

^CRi ⁼^ <¹⁴>

where kj is the key calculated from equation (9) and k_v is a key which gives an indication of the brightness of image IM.

This latter embodiment, which corresponds to the particular case where image IM is the outcome of the application of the PTR operator to an original image, makes it possible to achieve spatial coherence of the brightness of the original image.

According to an embodiment, the brightness indication of image IM is given by a key kv which is calculated by:

kv = exp (± .∑f₌₁ log(S + L_w(i))) (15) where N is the total number of pixels in image IM.

According to an embodiment, the modification coefficient CR_t is given by:

.HDH .LDR

CRj = _HDR ff£ (16)

j.max j

where kf^DR is the key which is associated with an image region Rj and calculated according to equation (9) where L^J _w k) is the luminance value of a pixel of image region Rj originating from luminance component L_w of image ^{I M} _> ^{is tne} maximum key from among all keys associated with image region Rj, each of said keys being calculated according to equation (9) for luminance values of the pixels of the image regions originating from luminance component L_w of image IM, k ^DR is the key which is associated with an image region Rj and calculated according to equation (9) where L^J _w k) is the luminance value of a pixel of image region Rj originating from the luminance component modified according to equation (10) or (1 1 ), and kf _ax is the maximum key from among all keys associated with the image regions, each of said keys being calculated according to equation (9) for luminance values of the pixels of the image regions originating from the luminance component modified according to equation (10) or (1 1 ).

This latter embodiment ensures the spatial coherence of the brightness of the original image, that is to say that the relative brightnesses of the different modified image regions follow the original relative brightnesses of these regions of image IM.

This latter embodiment is particularly advantageous in the case where image IM is the outcome of the application of a TMO to an original image as any type of TMO which applies to an image can then be used.

According to one of its hardware aspects, the invention relates to a device 400 for modifying the dynamic range of an image IM described with reference to Figure 4.

Device 400 comprises the following elements, interconnected by a digital address and data bus 81 :

- A calculation unit 43 (also called a central processing unit);

- A memory 45;

- A network interface 44, for interconnections between device 400 and other remote devices connected via a connection 41 ;

The calculation unit 43 can be implemented by a (possibly dedicated) microprocessor, a (possibly also dedicated) microcontroller, etc. The memory 45 can be implemented in a volatile and/or non-volatile form such as a RAM (random access memory), a hard disc, an EPROM (erasable programmable ROM), etc. Device 400 is configured to implement a method according to the invention described in relation to Figures 1 to 3.

For this purpose, means 43, 44 and possibly 45 cooperate with each other to obtain at least one image region, in order to associate a key with an image region, each key defining an indication of the brightness of the pixels belonging to this image region. Means 43, 44 and possibly 45 also cooperate with each other to modify the dynamic range of the pixels belonging to an image region by multiplying the luminance values of these pixels belonging to this image region by a coefficient determined from the key associated with this image region.

According to another hardware aspect, the invention relates to a device CONV for converting an image whose luminance values belong to a given dynamic value range (HDR) to an image whose luminance values belong to a lower dynamic value range (LDR) than that of the original image. This is usually referred to as HDR to LDR dynamic range reduction.

Figure 5 shows a block diagram of such an image conversion device which implements an image modification method according to the invention.

Device CONV comprises a dynamic range conversion operator TMO which is applied to luminance component Lw of an original image SIO to obtain a modified image whose luminance values belong to a lower dynamic value range LDR than that of image SIO (HDR).

In the case of a colour image, device CONV comprises means GLW for obtaining the luminance component Lw of this colour image SIO. For example if an image SIO is expressed in an (R,G,B) colour space, the image is transformed in order to be expressed in the (Χ,Υ,Ζ) colour space so as to recover the Y channel of the (Χ,Υ,Ζ) space which forms the luminance component Lw. It is widely known to use such colour space transformation means. Other examples of means GLW can be used without leaving the scope of the invention. Device CONV also comprises means DIV and MULT for the purpose of conserving a constant saturation and the hue of the colours. These means DIV are configured to divide the R, G and B colour components corresponding to a colour image SIO by component Lw and means MULT are configured to multiply the R, G and B colour components thus modified by the modified luminance component. The three components originating from this multiplication are then expressed in floating values. To obtain a modified image LDR, these three components originating from this multiplication are submitted at the input of means Ftol of device CONV and undergo a conversion of their values to whole values which belong to a dynamic value range for the screen on which the modified colour image must be displayed.

The operator TMO can be any TMO of the prior art which applies to a still image. The PTR operator described in the introductory section can for example be used. In this case, the modified luminance component is obtained from equation (5) or (6).

According to the invention, the conversion device also comprises means for modifying the dynamic range represented on the diagram by module C and means IAN for obtaining characteristics of the luminance component Lw. These means are configured so that the dynamic range of the values of the modified luminance component is in turn modified by one of the methods described in relation to one of Figures 1 to 3.

In Figures 4 and 5, the modules shown are functional units that may or may not correspond to physically distinguishable units. For example, these modules or some of them can be grouped together in a single component or circuit, or constitute functions of the same software. On the contrary, some modules may be composed of separate physical entities. The inter-image prediction devices compatible with the invention are implemented according to a purely hardware embodiment, for example in the form of a dedicated component (for example in an ASIC (application specific integrated circuit) or FPGA (field-programmable gate array) or VLSI (very large scale integration) or of several electronic components integrated into a device or even in the form of a mixture of hardware elements and software elements.

Although not explicitly described, the present embodiments can be used in combination or sub-combination.

Claims

1 . Method for modifying the dynamic range of an image (IM) by multiplying the luminance values of the pixels of the image by a modification coefficient determined from a value, called a key, which defines an indication of the brightness of the pixels, characterised in that it comprises the following steps:

- obtaining (10) at least one image region (Rj),

- associating (20) a key (kj) with each image region, said key defining an indication of the brightness of the pixels belonging to this image region, and

- modifying (30) the dynamic range of the pixels of at least one image region by multiplying the luminance values of these pixels by a coefficient determined from the key thus associated with this image region.

2. Method according to claim 1 , wherein at least one image region is obtained by image segmentation.

3. Method according to claim 2, wherein at least one image region is obtained by using an image segmentation based on the luminance values of the pixels of this image.

4. Method according to one of the preceding claims, wherein during the step of modifying the dynamic range of the pixels whose luminance values belong to an image region,

- it is determined if a pixel belonging to an image region Rj is close, in the sense of a metric, to another image region (Rk), and

- if a pixel belonging to an image region (Rj) is considered as being close to another image region (Rk), the modification coefficient is then equal to the weighted sum of the modification coefficients relating to each of these two image regions (Rj and Rk).

5. Device for modifying the dynamic range of an image (IM) by multiplying the luminance values of the pixels of the image by a modification coefficient determined from a value, called a key, which defines an indication of the brightness of the pixels, characterised in that it comprises the following means for:

- obtaining at least one image region (Rj),

- associating a key (kj) with each image region, said key defining an indication of the brightness of the pixels belonging to this image region, and

- modifying the dynamic range of the pixels belonging to an image region by multiplying the luminance values of these pixels by a coefficient determined from the key thus associated with this image region.

6. Device according to claim 5, which also comprises means for segmenting an image.

7. Device according to claim 6, wherein the means for segmenting an image are configured to implement a segmentation based on the luminance values of the pixels of this image.

8. Device according to one of claims 5 to 7, wherein the means for modifying the dynamic range of the pixels whose luminance values belong to an image region are configured to,

- determine if a pixel belonging to an image region is close, in the sense of a metric, to another image region (Rk), and