JP2012042804A - Image processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of offering an observer near an image display device an image suitable for him/her when a plurality of observers are detected.SOLUTION: The image processing apparatus which corrects colors of a display image according to a position of an observer includes an acquisition means for acquiring photographic image data which can be acquired by imaging an observation range of the image display device, a viewpoint distance calculation means for calculating a viewpoint distance of the observer who is closest to the device from the photographic image data, an observation angle calculation means for calculating an observation angle of the observer closest to the device by using an image position in the photographic image data which shows the observer closest to the device and the viewpoint of the observer, and a color correction means for performing color correction corresponding to the calculated observation angle to an input image data.

Description

  The present invention corrects the color of the display image in accordance with the position of the observer.

  In recent years, with the spread of personal computers, various people input images with image input devices such as digital cameras and color scanners, display them on image display devices such as CRTs and LCDs, and use image output devices such as color printers. Output. Under such circumstances, attention has been paid to color matching processing for matching the reproduced colors of the input device, display device, and output device. The color matching process is performed by associating the reproduction colors of each device based on a profile in which the color reproduction characteristics (data obtained by outputting a predetermined color chart and measuring the color) of the device to be matched are described.

  The color reproduction characteristics are generally calculated using a colorimeter. In general, color measurement of an image display apparatus measures a light emitted in a vertical direction from the front of the image display apparatus with a colorimeter. However, in general, an image display device has a viewing angle characteristic that a display color changes depending on a viewing angle. For this reason, the result of matching by the color matching process does not match the color when the image display device is viewed from a direction other than the front.

  As a method for correcting the change in display color due to such viewing angle characteristics, a technique for correcting the display color of the image display device according to the observation angle of the observer is known as a conventional technique (for example, Patent Document 1). ). This technique captures an image within a range assumed to be an observation position of an observer who observes an image display device. Then, the observation angle of the observer is calculated from the captured image, and the saturation and brightness of the display image are corrected according to the observation angle.

JP 2009-128381 A

  However, the conventional technique has a problem of displaying an appropriate image at which observer's observation position when there are a plurality of observers.

  Therefore, an object of the present invention is to provide an appropriate image to an observer close to an image display device when a plurality of observers are detected. This is because a person who is most interested in an image displayed by the image display apparatus often observes the display image in the immediate vicinity of the image display apparatus.

  In order to solve the above-described problems, an image processing apparatus according to the present invention is an image processing apparatus that corrects the color of a display image according to the position of an observer, and is obtained by imaging an observation range of the image display apparatus. An acquired means for acquiring captured image data, a viewpoint distance calculating means for calculating a viewpoint distance of an observer who is closest to the captured image data, and an observer who is closest to the captured image data Using the image position and the viewpoint distance of the observer at the closest distance, an observation angle calculation means for calculating the observation angle of the observer at the closest distance, and a color corresponding to the calculated observation angle Color correction means for performing correction on the input image data.

  In order to solve the above problem, an information processing method according to the present invention is an image processing method for correcting the color of a display image according to the position of an observer, and images an observation range of an image display device. An acquisition step of acquiring the captured image data obtained by the step, a viewpoint distance calculation step of calculating a viewpoint distance of an observer who is closest to the captured image data, and an observer who is at the closest distance in the captured image data An observation angle calculating step of calculating an observation angle of the observer at the closest distance using an image position indicating the viewpoint distance of the observer at the closest distance, and according to the calculated observation angle And a color correction step for performing the color correction on the input image data.

  According to the present invention, it is possible to provide an image that is appropriate for the closest observer.

1 is a block diagram illustrating an image processing apparatus in Embodiment 1. FIG. 3 is a flowchart illustrating a processing flow of the image processing apparatus according to the first exemplary embodiment. It is a figure which shows the positional relationship of an image display apparatus, an image imaging device, and an observer. It is explanatory drawing of the space | interval calculation of eyes. It is a flowchart which shows the flow of color conversion LUT creation. It is a schematic diagram of the positional relationship of the observer from which an image display apparatus and an observation angle differ. It is a figure which shows the example of the color reproduction range of the image display apparatus by a different observation angle. 6 is a flowchart illustrating a processing flow of the image processing apparatus according to the second exemplary embodiment. It is explanatory drawing of the area calculation of a face area | region.

[Example 1]
Hereinafter, an embodiment according to the present invention will be described in detail with reference to the drawings.

  The image display device generally has a viewing angle characteristic in which the saturation and contrast of the display color decrease as the observation position deviates from the front. FIG. 6 shows a schematic diagram of the positional relationship between observers having different observation angles from the image display device 2. The observer 61 is at a position where the angle formed by the normal direction from the center of the image display device 2 and the line-of-sight direction is 0 degrees, and the observer 62 is formed by the normal direction from the center of the image display apparatus 2 and the line-of-sight direction. At an angle of 22.5 degrees, the observer 63 observes at an angle of 45 degrees between the normal direction from the center of the image display device 2 and the line-of-sight direction. FIG. 7 is a diagram showing a color reproduction range that can be seen when the viewers 61, 62, and 63 of FIG. 6 observe the display color of the image display device 2, and the color reproduction range of the image display device 2 in the CIELAB space. * -A * shows a cross section. A line 71 indicates a color reproduction range visible from the observer 61, a dotted line 72 indicates a color reproduction range visible from the observer 62, and a broken line 73 indicates a color reproduction range visible from the observer 63.

  As can be seen from FIG. 7, the larger the observation angle, the narrower the color reproduction range of the image display device 2 that can be seen by the observer. Therefore, when the same display image is observed at the position of the observer 61 and the position of the observer 63, When the image is observed at the position of the observer 63, the saturation and contrast of the image are lowered and appear.

  FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. In FIG. 1, reference numeral 1 denotes an image processing apparatus, which controls the entire apparatus by a processor (not shown) controlling each unit. Reference numeral 2 denotes an image display device such as a display. Reference numeral 3 denotes an image capturing device such as a digital camera that captures a range assumed to be an observer's observation position. The memory 4 stores measurement values of the image display device 2 and the like. The image input unit 101 inputs image data. The color correction unit 102 performs color correction according to the viewpoint position on the input image data. The image output unit 103 outputs the color-corrected image data to the image display device 2. The observation angle calculation unit 104 calculates the observation angle of the observer. The viewpoint distance calculation unit 105 calculates the viewpoint distance of the observer. The captured image acquisition unit 106 acquires a captured image captured by the image capturing device 3.

FIG. 2 is a flowchart of exemplary processing executed by the image processing apparatus 1.
First, in step S <b> 21, the image processing apparatus 1 inputs original image data through the image input unit 101. The acquired original image data is stored in the memory 4.

  In step S <b> 22, the image processing apparatus 1 causes the image capturing apparatus 3 to capture the range that is assumed to be the observation position of the observer observing the image display device 2, and acquires captured image data via the captured image acquisition unit 106. The image pickup device 3 includes an image pickup device such as a CCD and is attached to the upper portion of the image display device 1. The configuration of the image pickup device 3 is not particularly limited as long as the image pickup device 3 can pick up an image that can recognize the face and eyes of the observer, and may be a digital still camera or a digital video camera. Also good.

  In step S23, the captured image data is analyzed to detect the face area of the observer. For example, the face area can be detected by detecting a skin color area from the captured image data and specifying the face area from the detection result. The method for detecting the face area is not limited to this, and various known methods may be used.

  In step S24, the image processing apparatus 1 determines whether or not a plurality of face regions are detected from the captured image in step S23. When it is determined that a plurality of face areas are detected, the process proceeds to step S25, and when it is determined that a plurality of face areas are not detected, the process proceeds to step S26.

  In step S25, the viewpoint distance calculation unit 105 calculates a viewpoint distance corresponding to each detected face area, determines an observer with the closest distance from the image display device 2 from the observers corresponding to the plurality of face areas, Determine the target observer. The viewpoint distance calculation unit 105 detects an eye region from the detected face region using template matching or the like. The method for detecting the eye region is not limited to this, and various known methods may be used. The viewpoint distance calculation unit 105 calculates the number of pixels between the detected left and right eyes in the captured image. Then, the observation distance is calculated from the number of pixels of the left and right eye intervals using a one-dimensional LUT in which the relationship between the number of pixels of the left and right eye intervals determined in advance and the observation distance is determined. The closer the observer is to the image display device 2, the larger the distance between the left and right eyes is, so that the observation distance is estimated using this relationship. A one-dimensional LUT in which the relationship between the number of pixels between the left and right eyes and the observation distance is determined is stored in the memory 4. The observation distances of the detected plurality of face areas are calculated, and the face area with the smallest observation distance is set as the observer with the shortest distance from the image display device 2.

A method for detecting the observer with the closest distance will be described in detail with reference to FIGS. FIG. 3 is a diagram illustrating a positional relationship between the image display device 2, the image capturing device 3, and the viewers 31 and 32. When the image capturing apparatus 3 captures a range that is assumed to be the observation position of the observer, a captured image shown in FIG. 4 is obtained. In step S23, the face regions of the viewers 31 and 32 are detected from the captured image data indicating the captured image, and the pixel numbers d ₁ and d ₂ between the left and right eyes are calculated. When the number of pixels between the left and right eyes is obtained, the observation distances L ₁ and L ₂ in FIG. 3 are calculated using a one-dimensional LUT of the relationship between the number of pixels between the left and right eyes and the observation distance.

  In step S <b> 26, the observation angle calculation unit 104 calculates an observation angle from the captured image data captured by the image capturing unit 107.

  First, the observation angle calculation unit 104 acquires the distance between the image display device 2 and the target observer. If it is determined in step S24 that a plurality of face regions have been detected, the observation distance of the target observer calculated in step S25 is acquired. When it is determined in step S24 that a plurality of face areas are not detected, the same processing as the observation distance calculation processing in step S25 is performed to calculate the observation distance of one observer (target observer) included in the captured image. .

  Next, the number of pixels in the horizontal interval from the center position in the horizontal direction of the captured image to the midpoint of the left and right eyes of the target observer is calculated. Then, a two-dimensional LUT is used to obtain an observation angle from a combination of the observation distance of the observer and the number of pixels in the horizontal interval from the horizontal center position of the captured image to the midpoint of the left and right eyes of the observer. Thus, the observation angle of the target observer is calculated from the observation distance of the target observer and the number of pixels in the horizontal interval from the horizontal center position of the captured image to the midpoint of the left and right eyes of the target observer. In the case of an input value that is not determined in the two-dimensional LUT, the input value is obtained from the determined value around the input value using interpolation processing. In addition, the position of the image capturing device 3 is set so that the horizontal center position of the captured image and the horizontal center of the image display device 2 coincide.

  When there are a plurality of observers, a method of calculating an average value of observation angles of a plurality of observers and setting the average value as the position of the observer is conceivable. However, in this method, when a person is far away from the display screen, it is impossible to display an image suitable for an observer who strictly observes the image at a short distance due to the effect of detecting the person. There is a possibility that. Therefore, in this embodiment, when a plurality of observers are detected, the distance of the observer from the image display device is calculated, and an image appropriate for the observer closest to the image display device is displayed. This is because a person who is most interested in the image displayed by the image display device is likely to observe the display image in the immediate vicinity of the image display device.

  In step S27, the color correction unit 102 performs color correction on the input image data based on the calculated observation angle. The color correction unit 102 converts input image data using a color conversion LUT for the calculated observation angle. The color conversion LUT is obtained in advance every 5 degrees and stored in the memory 4, and in the case of an observation angle that is not at the prepared angle, the color conversion LUT of the two angles that are closest to the observation angle among the prepared angles. Then, a color conversion LUT corresponding to the calculated observation angle is calculated using interpolation processing. In the case of input image data not stored in the color conversion LUT, interpolation processing such as tetrahedral interpolation is performed using lattice point data corresponding to the stored input image data around the input image data. Obtain corrected image data. Here, the color conversion LUT is a three-dimensional LUT in which the correspondence between the RGB value of the input image and the RGB value output by the image display device is described. In this embodiment, the angle interval for preparing the color conversion LUT is 5 degrees. However, the present invention is not limited to this, and a finer interval or a larger interval may be used.

Hereinafter, a processing procedure for creating a color conversion LUT will be described with reference to the flowchart of FIG.
In step S51, in order to acquire the relationship between the device RGB value and the XYZ value of the image display device 2, the image display device 2 displays a patch according to a plurality of predetermined device RGB values, measures the patch, and measures a plurality of XYZ values. Get the value. As the measurement value, a value obtained by measuring the patch from the position where the observation angle is 0 degree and a value measured from the position of the observation angle where the color conversion LUT is to be created are acquired.
In step S52, the measured value (XYZ value) acquired in S50 is converted into an L * a * b * value using (Formula 1).

  In step S53, first, a color reproduction range at a certain observation angle is calculated from L * a * b * values corresponding to a plurality of measurement values at a certain observation angle.

  Then, using the relationship between the device RGB value and the XYZ value obtained at step S51 at the observation angle of 0 degree, the device RGB value corresponding to the lattice point constituting the LUT is represented as L corresponding to the observation angle of 0 degree. * A * b * value is converted. Next, the L * a * b * value corresponding to 0 degrees is mapped to the L * a * b * value within the color reproduction range of the certain observation angle. As shown in FIG. 7, since the color reproduction range varies depending on the observation angle, the color gamut compression process is performed for each observation angle. Depending on the observation angle, the L * a * b * value of the color at 0 degree may be outside the color gamut of the image display device 2 at the observation angle, so that all colors are displayed at the observation angle by color gamut compression processing. The image is compressed into a color gamut of 2 and converted into a color reproducible by the image display device 2 at an observation angle.

  In step S54, the mapped L * a * b * values are converted into XYZ values. In step S55, the XYZ value is converted into the device RGB value of the image output apparatus at the certain observation angle using the relationship between the device RGB value and XYZ value of the image display apparatus 2 at the certain observation angle acquired in step S51. .

  In step S56, the device RGB value of the image output device at the certain observation angle obtained in step S55 is obtained at the lattice point constituting the LUT corresponding to the device RGB value when the observation angle is 0 degrees used in step S51. Is stored. By performing the processing of step S53 to step S56 for all the lattice points constituting the LUT, the LUT corresponding to the certain observation angle can be created. Then, an LUT corresponding to each of the plurality of observation angles can be created by performing the LUT creation processing corresponding to the observation angle for each of the plurality of observation angles measured in step S51.

  In step S <b> 8, the image output unit 103 outputs the image data corrected by the color correction unit 102 to the image display device 102.

  According to this embodiment described above, it is possible to display an image in which color reproduction appropriate for the closest observer is performed when a plurality of observers are detected.

[Example 2]
In the first embodiment, an example is shown in which the distance of the observer is calculated from the number of pixels between the left and right eyes. However, the distance between the left and right eyes cannot be calculated when the observer overlaps another object on the captured image and one eye is not captured. Therefore, in this embodiment, an example in which the distance of the observer is calculated from the area of the face area will be described.

  The block diagram showing the logical configuration of the color processing apparatus that implements the present invention is the same as that in FIG. FIG. 8 is a flowchart of exemplary processing executed according to this embodiment. In FIG. 8, steps other than step S85 are the same as those in FIG.

  In step S <b> 85, the image processing apparatus 1 calculates a viewpoint distance from the viewpoint distance calculation unit 105, and detects an observer who is closest to the image display apparatus 2. Specifically, the viewpoint distance calculation unit 105 calculates the area of the detected face area (the number of pixels included in the face area), and uses a one-dimensional LUT of the relationship between the area of the face area and the observation distance obtained in advance. To calculate the observation distance. If the observer overlaps on the captured image and a part of the face area is not shown, the area is calculated by predicting the hidden part using the face template. The closer the observer is to the image display device 2, the larger the face area is imaged. Therefore, the observation distance is estimated using the relationship between the observation distance and the face area. That is, a one-dimensional LUT of the relationship between the area of the face area and the observation distance is stored in the memory 4. Then, the observation distance of each of the detected plurality of face areas is calculated, and the face area with the smallest observation distance is set as the observer with the shortest distance from the image display device 2.

  FIG. 9 is a conceptual diagram illustrating the area of the detected face area of the captured image. The face area detected by the observer 91 is indicated by a dotted line 93, and the face area detected by the observer 92 is indicated by a dotted line 94. The area of the face area of the observer 91 is calculated by x1 × y1, and the area of the face area of the observer 92 is calculated by x2 × y2. In the present embodiment, the detected face area is rectangular, but the shape is not limited to this and may be a circle or an ellipse.

  According to the present embodiment described above, the distance can be calculated even when a part of the face area of the observer is not captured on the captured image, and an image suitable for the closest observer can be displayed. It becomes possible.

  In the first and second embodiments, the distance between the left and right eyes and the area of the face area are used to calculate the observation distance. However, the method for calculating the observation distance is not limited to this. For example, two imaging devices 3 are provided. A method may be used in which stereo shooting is performed and the distance is estimated based on an image shift caused by parallax.

  The present invention also provides a system or apparatus with a storage medium storing software program codes for realizing the functions of the above-described embodiments (for example, the functions shown in the flowcharts of FIGS. 2, 5, and 8). Can also be realized. In this case, the functions of the above-described embodiments are realized by the computer (or CPU or MPU) of the system or apparatus reading and executing the program code stored in the storage medium so that the computer can read the program code.

Claims (7)

An image processing apparatus that corrects the color of a display image according to the position of an observer,
Acquisition means for acquiring captured image data obtained by imaging the observation range of the image display device;
Viewpoint distance calculating means for calculating a viewpoint distance of an observer who is closest to the captured image data;
Observation for calculating an observation angle of the observer at the nearest distance using an image position indicating the observer at the nearest distance in the captured image data and a viewpoint distance of the observer at the nearest distance An angle calculation means;
An image processing apparatus comprising: a color correction unit that performs color correction on input image data according to the calculated observation angle.   The viewpoint distance calculating unit detects a face area from the captured image data, and calculates a viewpoint distance of the observer using an interval between left and right eyes of the detected face area. The image processing apparatus according to 1.   The image according to claim 1, wherein the viewpoint distance calculation unit detects a face area from the captured image data, and calculates the viewpoint distance of the observer using the area of the detected face area. Processing equipment.   The color correction unit converts a device value of the image display apparatus according to an observation angle of 0 degrees into a device value of the image display apparatus according to the calculated observation angle. 4. The image processing device according to any one of 3.   5. The image processing according to claim 1, wherein the color correction unit corrects the input image data to color data within a color reproduction range of the image display device at the calculated observation angle. apparatus. An image processing method for correcting the color of a display image according to the position of an observer,
An acquisition step of acquiring captured image data obtained by imaging the observation range of the image display device;
A viewpoint distance calculating step of calculating a viewpoint distance of an observer who is closest to the captured image data;
Observation for calculating the observation angle of the observer at the closest distance using the image position indicating the observer at the closest distance in the captured image data and the viewpoint distance of the observer at the closest distance An angle calculation step;
And a color correction step of performing color correction according to the calculated observation angle on the input image data.   A program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 5.