WO2008102296A2

WO2008102296A2 - Method for enhancing the depth sensation of an image

Info

Publication number: WO2008102296A2
Application number: PCT/IB2008/050577
Authority: WO
Inventors: Michel W. Nieuwenhuizen
Original assignee: Koninklijke Philips Electronics NV
Current assignee: Koninklijke Philips NV
Priority date: 2007-02-23
Filing date: 2008-02-18
Publication date: 2008-08-28
Anticipated expiration: 2009-08-23
Also published as: WO2008102296A3

Abstract

Method for enhancing the depth sensation of an image (22),which method comprises the steps of: obtaining the depth of pixels depicting objects (24, 26) located in at least one part of the image (22), calculating a weight for pixels depicting objects (24, 26) located in said at least one part of the image (22) as a function of their depth, generating a depth-dependent histogram (17), using the weighted pixel data, calculating a correction curve (19) from the depth-dependent histogram (17), and applying the correction curve (19) to the entire image (22).

Description

Method for enhancing the depth sensation of an image

FIELD OF THE INVENTION

The present invention relates to a method for enhancing the depth sensation of an image. The invention also relates to a computer program product, an image-processing system, and an image display device.

BACKGROUND OF THE INVENTION

Three-dimensional images to be viewed on two-dimensional displays are often enhanced by electronically processing the two-dimensional image so as to give the viewer an enhanced depth sensation. An image-processing system usually comprises means for generating a luminance histogram of the image being enhanced. A luminance histogram is a tabulation of pixel value populations displayed as a bar chart, in which the x-axis represents all possible pixel values and the y-axis is the total image count of each given pixel value. A histogram counts how many pixels in the image have a given intensity value or range of values. Each histogram intensity value or range of values is referred to as a bin. Each bin contains a positive number that represents the number of image pixels falling within the range of the bin. A typical 8-bit gray-scale histogram contains 256 bins. Each bin has a range of a single intensity value. Bin 0 contains the number of pixels in the image that have a grayscale value of 0 or black; bin 255 contains the number of white pixels. When the collection of bins is sorted (0-255) and charted, the histogram displays the intensity distributions of all pixels in the image. For a color image, three histograms are calculated for the red, green and blue components of the image.

The luminance histogram is analyzed to determine the most suitable luminance correction curve that should be applied to the image. Algorithms in the image- processing system then allow a user to adjust the brightness value of each pixel and to dynamically display the results as adjustments are being made. Improvements in brightness and contrast of an image, or an image sequence, can thus be obtained.

A cumulative histogram is normally used as a luminance correction curve. A cumulative histogram is the integral version of a normal histogram. It represents, level by level, the probability of finding a pixel that has a brightness which is lower than or equal to a particular value. The smoothness and form of the correction curve provides an indication of how uniformly grey levels are distributed. If the cumulative histogram provides too much correction, a combination of the cumulative histogram and a fixed gain may be used as the luminance correction curve. This method has the drawback that it does not always enhance an image in the way a human being would prefer to see the image. This drawback may be overcome by applying a heuristic algorithm wherein the luminance correction curve is not based on a cumulative histogram but on an algorithm that is based on experience/trial and error alone. However, a problem with this method is that it is difficult to obtain the desired image enhancement. For example, if a dark image contains small areas of very bright light, these small areas may be lost in the enhancement process.

European patent no. 0 648 043 discloses a picture signal enhancement circuit which comprises histogram means for measuring a histogram only at transitions in the picture signal, i.e. only areas with detail are measured. This approach better mimics the behavior of the human eye which also mainly focuses on details. However, this method encounters problems with images in which a person is standing in front of a detailed background, such as, for example, patterned wallpaper, which can be enhanced to a better extent than the person standing in front of it.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method for processing a two-dimensional input image so as to create a two-dimensional output image with an enhanced depth sensation.

This object is achieved by a method for enhancing the depth sensation of an image comprising the steps defined in claim 1. The method comprises the steps of: a) determining the depth of pixels depicting objects located in at least one part of the image by using any suitable technique, (wherein the expression "objects" throughout this document is understood to mean a material body, such as a person, animal, building, etc. or a feature thereof such as a person's eyes, an animal's ears, smoke coming from a chimney or the door of a building, etc.), thereby generating a depth map; b) calculating a weight for pixels depicting objects located in the at least one part of the image as a function of their depth. Each pixel or a group of pixels is multiplied by a weighting factor whose magnitude depends on the importance or assumed importance of the object that the pixel or pixels depict to a viewer of the image, the importance in casu being related to the depth information (depending on whether the object or objects of interest have been selected by the viewer or by an image-processing system); c) generating a depth-dependent histogram of a property of the image, such as luminance, using the weighted pixel data; d) calculating a correction curve from the depth-dependent histogram; and e) applying the correction curve to the entire image, thereby enhancing the depth sensation of the image. The property of the entire image is thus adjusted on the basis of the depth-dependent histogram.

The method according to the invention has the advantage that the depth map which is generated does not have to be highly accurate for generating a depth-dependent histogram, as depth errors will only slightly modify the depth-dependent histogram but will not result in the presence of visible artifacts in the enhanced image because the correction curve is applied to the entire image independently of the depth of objects in the image. This is not the case when a depth map is used during the histogram processing operation (i.e. when the histogram correction is applied to the image) because any error, for example, in discriminating between objects in the image, might result in the presence of artifacts in the enhanced image.

In accordance with an embodiment of the invention, the property is one of luminance, grey-scale magnitude, UV chrominance value or a property along one of the axes of a color space such as RGB, CMYK, YUV or HSL.

In accordance with an embodiment of the invention, the depth of pixels depicting objects located in the entire image is determined in step a).

In accordance with another embodiment of the invention, pixels depicting objects located within a predetermined depth range in the at least one part of the image are weighted more than pixels depicting objects located outside the predetermined depth range in step b). This predetermined depth range may be selected and changed by a user or by an image-processing system.

In accordance with a further embodiment of the invention, pixels depicting objects in the foreground of the image are weighted more than pixels depicting objects in the background of the image in step b). In a particularly favorable embodiment, objects whose distances are greater than a threshold value are considered to be in the background of an image, whereas objects that are closer than the threshold value are considered to be in the foreground of the image. A viewer is usually more interested in objects located in the foreground of an image than in objects located in the background. If a histogram primarily represents objects in the foreground of an image, such as a person standing in a countryside landscape, the influence of background objects in the image, such as distant mountains, will be diminished. The resulting enhanced image will therefore correspond more closely to the image that would be seen if the scene depicted in the image were viewed by a human being. However, even if there is an object, such as a tree, located behind an object in the foreground of the image, the pixels depicting the tree will be more weighted than pixels depicting objects behind the tree, and the sharpness, contrast, hue and/or brightness of the tree will therefore be enhanced more than objects behind the tree. The method according to the invention therefore avoids the problems that occur when using conventional enhancement techniques such as a cumulative histogram method, in which large non-detailed areas of an image may be enhanced too much, whereas other parts of the image containing important information may be degraded. Alternatively, objects at multiple depth ranges may be weighted more, which may be particularly beneficial when there are multiple regions of interest at different depths. In accordance with another embodiment of the invention, pixels depicting objects that are in focus are weighted more than pixels depicting objects that are out of focus. The parts of an image that are in focus usually have the highest resolution, i.e. the in- focus parts contain the highest spatial frequency components whose presence can therefore be detected, whereupon the depth sensation of the in- focus parts can be subsequently enhanced. In accordance with another embodiment of the invention, pixels depicting objects that are in one or more particular locations in the image are weighted more than pixels depicting objects in other locations.

In accordance with an embodiment of the invention, a depth-dependent histogram for one part of the image is generated in step c) and the entire image is enhanced on the basis of this depth-dependent histogram in step e). By selecting a relevant part of the image, the histogram calculation is simplified and can be fully tuned for this part of the image.

In accordance with another embodiment of the invention, the at least one part of the image is a central part of the image, as an image tends to be centered on what the image capturer considers to be the object of most interest. In accordance with a further embodiment of the invention, the image is a still image, such as a camera still, or an image sequence, such as a video or television signal. The present invention also relates to a computer program product which comprises a computer program stored on a computer-readable medium or a carrier wave, and computer program code means arranged to cause a computer or a processor to execute at least one of steps b) to e) of a method in accordance with any one of the embodiments of the invention.

The invention further relates to an image-processing system for enhancing the depth sensation of an image, which system comprises: - means for determining the depth of pixels depicting objects located in at least one part of an image; means for calculating a weight for pixels depicting objects located in the at least one part of the image as a function of their depth, wherein each pixel or a group of pixels is multiplied by a "weighting factor" whose magnitude depends on the importance, or assumed importance, of the object that the pixel or pixels depict to a viewer of the image; means for generating a depth-dependent histogram of a property of the image, using the weighted pixel data; means for calculating a correction curve from the depth-dependent histogram; and - means, such as a non- linear processing circuit, for applying the correction curve to the entire image, thereby enhancing the depth sensation of the image.

In accordance with an embodiment of the invention, the property is one of luminance, grey-scale magnitude, UV chrominance value or a property along one of the axes of a color space such as RGB, CMYK, YUV or HSL. In accordance with an embodiment of the invention, the image-processing system comprises means for determining the depth of pixels depicting objects located in the entire image.

In accordance with another embodiment of the invention, the means for calculating a weight for pixels depicting objects located in the at least one part of the image are arranged so that pixels depicting objects located within a predetermined depth range in the image are weighted more than pixels depicting objects located outside the predetermined depth range.

In accordance with a further embodiment of the invention, the means for calculating a weight for pixels depicting objects located in the at least one part of the image are arranged so that pixels depicting objects in the foreground of the image are weighted more than pixels depicting objects in the background of the image.

In accordance with another embodiment of the invention, pixels depicting objects that are in focus are weighted more than pixels depicting objects that are out of focus. In accordance with another embodiment of the invention, pixels depicting objects that are in one or more particular locations in the image are weighted more than pixels depicting objects in other locations.

In accordance with yet a further embodiment of the invention, the image- processing system comprises means for generating a depth-dependent histogram for one part of the image and means for enhancing the entire image on the basis of this depth-dependent histogram, wherein the one part of the image is, for example, a central part of the image.

In accordance with an embodiment of the invention, the image is a still image or a moving image. The invention also relates to an image display device, such as a mobile or non- mobile telephone, television, personal digital assistant (PDA), computer or camera, comprising means for capturing and/or receiving images. The image display device comprises at least part of an image-processing system in accordance with any one of the embodiments of the invention, or a computer program product in accordance with an embodiment of the invention, or means for communicating with the image-processing system or the computer program product.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will hereinafter be elucidated by way of non- limiting examples with reference to the appended Figures, in which;

Fig. Hs a block diagram of an embodiment of an image-processing system according to the invention;

Fig. 2 is a depth-dependent histogram; Fig. 3 shows a luminance correction curve; Fig. 4 is a depth-dependent histogram after correction;

Fig. 5 shows an input image for an image-processing system as captured by image-capturing means, such as a camera;

Figs. 6 and 7 show the image of Fig. 5 after it has been enhanced by means of conventional enhancement techniques; Fig. 8 shows the image of Fig. 5 after it has been enhanced by means of a method in accordance with an embodiment of the invention;

Fig. 9 shows a depth map of the input image of Fig. 5.

DESCRIPTION OF EMBODIMENTS Figure 1 is a block diagram of an embodiment of an image-processing system 10 according to the invention. The system receives an image signal, Y_1n, as an input signal either from image-capturing means within the system 10 or at the same location as the system 10, or from image-capturing means remote from the system 10. The system 10 comprises (commercially available or self- written) software or hardware means 12 for determining the depth of pixel objects located in at least one part of an image, for example, by generating a reference plane and determining the depth of pixels therefrom.

Examples of methods for depth map generation from a 2D image are disclosed in International Patent Application WO2005/013623 entitled "Multi-view Image Generation" and International Patent Application WO2005/083630 entitled "Creating a depth map", herein both incorporated by reference. A refined method for depth map generation is described in co-pending European patent application 06301011 by the same applicant, herein incorporated by reference.

Alternatively, depth information may be obtained from stereoscopic footage as e.g. described in "Depth Estimation from Stereoscopic Image Pairs Assuming Piecewise Continuous Surfaces" by L. Falkenhagen, published in Proc. of European Workshop on combined Real and Synthetic Image Processing for Broadcast and Video Production, Hamburg, November 1994. Depth information may also be obtained through remote sensing.

The image-processing system 10 also comprises means 14 for calculating a weight for pixels depicting objects located in the at least one part of the image as a function of their depth. The image-processing system 10 further comprises a histogram-generating means and histogram memory 16.

The image-processing system 10 further comprises means 18 for calculating a correction curve, or transfer curve, from the depth-dependent histogram, and processor means 20, such as a non-linear processing circuit, which may be programmed to apply such a correction to an input signal Y_1n and thereby adjust/correct the sharpness, contrast, hue and/or brightness of the image. The processor means 20 may adjust, for example, the luminance of an incoming luminance signal Y_1n and thus supply an enhanced output luminance signal Y_out to an image display device (not shown). In analog embodiments, the non-linear processing circuit 20 may be constituted by a non-linear amplifier, and in digital embodiments, the nonlinear processing circuit 20 may comprise a memory which is used as a look-up table.

A correction curve may be calculated from the depth-dependent histogram by calculating the cumulative histogram which is a mapping that counts the cumulative number of observations in all bins up to the specified bin:

cum_hist[0] = hist[O] cum_hist[l] = hist[O] + hist[l] cum_hist[2] = hist[O] + hist[l] + hist[2] cum_hist[32] = hist[O] + hist[l] + hist[2] + + hist[32].

If a cumulative histogram is used as look-up table for the luminance correction, the output will be an image picture with a "flat" histogram which normally provides too much correction if a natural- looking image is desired. A look-up table (lut), which is a combination of the cumulative histogram (cumhist) and a fixed gain (linear curve), is therefore usually used as the luminance correction curve so as to provide a more natural- looking image (when desired):

lut[n] = alpha * cumhist[n] + (1 -alpha) * linear[n]

in which linear[n] = n* scale factor. The scale factor is the number of levels of the luminance pixels (for example, 256 levels) divided by the number of bins (for example, 32 bins).

Alternatively, a heuristic algorithm may be applied to calculate a correction curve which is not based on a cumulative histogram but on an algorithm which is based on experience/trial and error alone. Such an approach is described, for example, in the book entitled "Video Processing" by Gerard de Haan (ISBN 9090140158).

It should be noted that, although the components 12, 14, 16, 18, 20 of the system 10 are illustrated as separate units, a plurality of components can be functionally and/or physically integrated with one another, for example, in a microprocessor.

Furthermore, all of the components 12, 14, 16, 18, 20 of the image-processing system 10 do not necessarily need to be located at the same location. The components may be located at a plurality of locations and comprise means for communicating with the other components via a network such as the Internet. The image-processing system 10 receives a signal such as a luminance signal

Y_1n from a pixel as input. The pixel is offered to the histogram memory 16 as the address (or "bin"). The depth of the pixel is determined by software means 12. The output of the software means 12 is a signal indicating how far to the front of the image the object depicted by this pixel is located. A weight for the pixel is calculated by means 14 using this information and information manually provided by a user of the system 10 or stored in the system 10. For example, the weight may be in the range of 0 to 10, wherein pixels depicting objects in the far background of the image are weighted "1" and pixels depicting objects in the foreground of the image are weighted "10". The degree of difference between the lightest and darkest parts of the objects of most interest to a viewer will therefore be increased. Weighted pixel data is then sent to the histogram memory 16. Instead of adding a constant value (typically 1) to a bin for each pixel with this specific luminance level (which is the case when there is no depth dependency), when a pixel has a luminance value that matches a particular bin, the weight of this pixel is added to this bin. The result is that objects in the foreground of an image are weighted more than objects in the background of the image. This better imitates how a human eye will perceive the scene depicted in the image, as human beings are generally more interested in what is happening nearby than what is happening in the distance.

A depth-dependent histogram 17, such as the one shown in Figure 2, is generated by using the weighted pixel data. The x-axis represents all of the possible pixel values and the y-axis shows the total image count of each given pixel value. The luminance level of an incoming image may be divided into 256 levels of, for example, 0 to 255 and an ideal image should contain approximately equal numbers of all 256 grey levels. The output of the histogram memory 16 (which may be read once per field, i.e. after measuring a complete image/video field) is supplied to means 18 which calculate the luminance correction curve, or transfer curve, to be applied to the image. The luminance correction curve re-distributes the grey levels in the depth-dependent histogram 17 so that it becomes substantially flattened (i.e. so that each bin has substantially the same value), thereby improving the quality of the resulting enhanced image. This flattening is achieved by adapting the gain of bins in the image. Figure 3 shows a luminance correction curve 19 and Figure 4 shows a depth- dependent histogram 21 after correction, which is substantially (but not completely) flattened. In general, the correction should only be partial, as complete flattening of the depth-dependent histogram may make the image look unnatural.

Use of such a method for enhancing an image has the advantage that the distribution of the grey levels in the image is more equalized, which results in an improved dynamic range for the objects in the image that are of most interest, rather than for those objects that are most prominent. The sharpness, contrast, hue or brightness of the objects of most interest will be increased. It is important to note that, even though the depth-dependent histogram is dominated by pixels located in only a part of an image, it is applied to the entire image. This has the advantage that, even if the depth determination is not completely accurate, this will not adversely influence the enhanced image as long as the errors in depth determination are not too large.

Figure 5 shows an input image 22 which can be processed by the image- processing system 10 according to the invention. It shows an original image 22 (in bitmapped graphics (bmp) format) which depicts a puppy dog 24 sitting on a floor 26. Due to the lack of depth sensation in the two-dimensional image, it is difficult to see the dog 24 in detail. The input image 22 is represented as an array of pixels. Each pixel depicts a particular object, such as the dog 24, the floor 26, the dog's eyes, ears or nose. Figure 6 shows the same image after it has been enhanced in a conventional cumulative histogram process. It can be seen that the floor 26 has been enhanced much more than the dog 24, contrary to what most viewers of the image would prefer.

Figure 7 shows the same image after it has been enhanced by means of a conventional heuristic algorithm. The more subtle facial features of the dog 24 are almost completely lost in the enhancement process, such as, for example, the highlights on the dog's face around the eyes and nose.

Figure 8 shows the image of the dog 24 sitting on the floor 26 after enhancement by means of a method in accordance with an embodiment of the invention. Such an enhanced image is obtained by determining the depth of pixels depicting the dog 24 and the floor 26 (a depth map 34 of the image 22 is shown in Figure 9), weighting pixels depicting the dog 24 more than pixels depicting the floor 26, generating a depth-dependent histogram by means of the weighted pixel data, calculating a luminance correction curve from the depth-dependent histogram, and applying the luminance correction curve to the entire image 22 so as to enhance the depth sensation of the image 22. Details of the dog's face are enhanced much more than details of the floor 26 (by a factor of about four or five), which floor details are largely ignored. This would also be the case if the dog 24 was sitting on a patterned floor.

Alternatively, if a viewer using an interactive processing system implementing the present invention, e.g. in the form of an Adobe Photoshop filter or similar software, wants to enhance an object that is not in the foreground or the center of an image, such as the leaf 28 on the floor 26 of the image shown in Figure 8, he can choose to weigh pixels depicting objects located in the background of the image more than pixels depicting objects located in the foreground or the center of the image. If the viewer is more interested in the floor 26 than in the dog 24, he can also choose to enhance objects that form the bulk of the image. Alternatively, the user may select multiple depth ranges so as to increase the weights of pixels at multiple depth ranges.

The enhanced image may be displayed on an image display device 30, such as a mobile or non-mobile telephone, television, personal digital assistant (PDA), computer or camera comprising display means, such as an LCD screen, a plasma display, a cathode ray tube display, etc. The image display device 30 may comprise at least part of an image- processing system 10 in accordance with an embodiment of the invention, or means for communicating with the image-processing system 10 via a network 32, such as the Internet. Further modifications of the invention within the scope of the claims will be apparent to a skilled person. For example, the invention is not only useful for users in the consumer field, such as television viewers and amateur photographers who can enhance images in accordance with personal preferences, but also for users in the field of professional image analysis and processing, such as in medical applications (NMR, X-ray scans), in criminal investigations (detection, identification and tracking of people), and in photographic art and computer-aided design applications.

Claims

CLAIMS:

1. A method for enhancing the depth sensation of an image (22), characterized in that said method comprises the steps of: a) obtaining the depth of pixels depicting objects (24, 26) located in at least one part of the image (22); b) calculating a weight for pixels depicting objects (24, 26) located in said at least one part of the image (22) as a function of their depth; c) generating a depth-dependent histogram (17) of a property of the image (22), using the weighted pixel data; d) calculating a correction curve (19) from the depth-dependent histogram (17); and e) applying the correction curve (19) to the entire image (22).

2. A method according to claim 1, characterized in that said property is one of luminance, grey-scale magnitude, UV chrominance value or a property along one of the axes of a color space such as RGB, CMYK, YUV or HSL.

3. A method according to claim 1, characterized in that the depth of pixels depicting objects (24, 26) located in the entire image (22) is determined in step a).

4. A method according to claim 1, characterized in that pixels depicting objects (24, 26) located within a predetermined depth range in said at least one part of the image (22) are weighted more than pixels depicting objects (24, 26) located outside said predetermined depth range in step b).

5. A method according to claim 1, characterized in that pixels depicting objects (24) relatively in the foreground of the image (22) are weighted more than pixels depicting objects (26) relatively in the background of the image (22) in step b).

6. A method according to claim 1, characterized in that pixels depicting objects that are in focus are weighted more than pixels depicting objects that are out of focus.

7. A method according to claim 1, characterized in that a depth-dependent histogram (17) for one part of the image (22) is generated in step c) and the correction curve (19) is applied to the entire image (22) in step e).

8. A method according to claim 7, characterized in that said at least one part of the image (22) is a central part of the image.

9. A method according to claim 1, characterized in that said image (22) is an image of an image sequence.

10. A computer program product, characterized in that it comprises a computer program stored on a computer-readable medium or a carrier wave, and computer program code means arranged to cause a computer or a processor to execute the steps a) to e) of a method according to any one of the preceding claims.

11. An image-processing system (10) for enhancing the depth sensation of an image (22), characterized in that said system comprises: means (12) for obtaining the depth of pixels depicting objects (24, 26) located in at least one part of an image (22); - means (14) for calculating a weight for pixels depicting objects (24, 26) located in said at least one part of the image (22) as a function of their depth; means for generating a depth-dependent histogram (17) of a property of the image (22), using the weighted pixel data; means (16) for calculating a correction curve (19) from the depth-dependent histogram (17); and means (20) for applying the correction curve (19) to the entire image (22).

12. An image-processing system (10) according to claim 11, characterized in that the means (12) for obtaining the depth of pixels is arranged to determine the depth of pixels depicting objects (24, 26) located in the entire image (22).

13. An image-processing system (10) according to claim 11, characterized in that the means (14) for calculating a weight for pixels depicting objects (24, 26) located in said at least one part of the image (22) are arranged so that pixels depicting objects (24, 26) located within a predetermined depth range in the image (22) are weighted more than pixels depicting objects (24, 26) located outside said predetermined depth range,

14. An image-processing system (10) according to claim 11, characterized in that the means (14) for calculating a weight for pixels depicting objects (24, 26) located in said at least one part of the image (22) are arranged so that pixels depicting objects (24) in the foreground of the image (22) are weighted more than pixels depicting objects (26) in the background of the image (22).

15. An image-processing system (10) according to claim 11, characterized in that pixels depicting objects that are in focus are weighted more than pixels depicting objects that are out of focus.

16. An image-processing system (10) according to claim 11, characterized in that pixels depicting objects that are in one or more particular locations in the image are weighted more than pixels depicting objects in other locations in the image.

17. An image-processing system (10) according to claim 11, characterized in that said image (22) is an image of an image sequence.

18. An image display device (30), characterized in that it comprises at least part of an image-processing system (10) according to claim 11.