JP2013012979A - Image processing apparatus and image processing method - Google Patents

Abstract

An image processing apparatus for obtaining a color space suitable for white balance correction by adjusting the shape of spectral sensitivity by color conversion without using a color chart.
An image processing apparatus that performs color conversion of an image photographed by an image sensor, and is present under a different type of light source different from a standard light source obtained based on information related to spectral sensitivity of the image sensor. A primary conversion unit that primarily converts color information including chromatic color into a color space for converting to color information under the standard light source, and a matrix used for conversion processing by the primary conversion unit includes a plurality of channels of the image sensor; It is optimized so that the evaluation value based on the overlap amount between them and the width index of each channel is minimized.
[Selection] Figure 7

Description

  The present invention relates to an image processing apparatus and an image processing method.

  When a person sees an object, it looks almost the same color when viewed under a different light source (color constancy). However, since the light from the object is obtained by multiplying the spectral radiation intensity of the illumination and the spectral reflectance of the object, when the light is detected by the image sensor, the signal obtained changes when the light source changes. Therefore, as a correction based on the light source, white balance correction is performed in which a gain is applied so that the R, G, and B values of the image sensor signal are the same when an achromatic object is photographed. However, even if white balance correction is performed on an image shot with a test light source to bring it closer to color reproduction with a reference light source, chromatic color correction cannot be performed sufficiently, so color reproduction of the shot image is not possible with other than a specific light source. May be accurate. In order to cope with this, if a different color conversion process is performed for each light source, a process for that is added, and it takes more time for image processing. If an image shot with a light source different from the reference light source can be achieved by performing only white balance correction to achieve the same color reproduction as the reference light source, the color processing of the shot image under various light sources is a simple process. It is convenient because it can be done uniformly. For example, when adjusting the color balance with the development software after shooting, the light source can be corrected and the desired color reproduction can be achieved simply by taking the gray balance, thereby improving efficiency. It is known that the color reproduction error is reduced when processing is performed after converting the color space in which white balance correction is actually performed into a color space different from camera RGB (see, for example, Patent Document 1).

Japanese Patent No. 4767699

  However, in Patent Document 1 described above, a color obtained by weighting and averaging color differences based on photographing output values obtained by photographing color charts for each of a plurality of types of different types of light sources when the color conversion is obtained. Since it is necessary to perform optimization by repeatedly performing arithmetic processing so that the constant prediction error is minimized, the calculation load is heavy. Further, there is a possibility that the color conversion depends on how to select the color chart and is not necessarily optimal for the actual photographing conditions.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of obtaining a color space suitable for white balance correction by adjusting the shape of spectral sensitivity by color conversion without using a color chart.

  An image processing apparatus of the present invention is an image processing apparatus that performs color conversion of an image photographed by an image sensor, and is a different kind of different type from a standard light source obtained based on information related to spectral sensitivity of the image sensor. A matrix for primary conversion of color information including a chromatic color under a light source into a color space for converting the color information under the standard light source into a color space; It is characterized by being optimized so that the evaluation value based on the overlap amount between the plurality of channels and the width index of each channel is minimized.

  The image processing method of the present invention is an image processing method for processing an image photographed by an image sensor to obtain a color balance, and is different from a standard light source obtained based on information related to spectral sensitivity of the image sensor. Including a conversion step of primary conversion of color information including chromatic colors under different types of light sources into a color space for conversion to color information under the standard light source, and the conversion process by the conversion step is obtained by the conversion step. A gain adjustment step of adjusting the white balance by adjusting the gain of the converted color information, and the matrix used for the primary conversion processing includes an overlap amount between a plurality of channels of the image sensor and a width index of each channel It is characterized by being optimized so that the evaluation value based on and is minimized.

  An image processing method of the present invention is an image processing method for processing an image in a predetermined color space to obtain a color balance, and a standard light source obtained based on information related to spectral sensitivity corresponding to the predetermined color space Includes primary conversion of color information including chromatic colors under different types of different light sources into a color space for converting into color information under the standard light source, and the conversion process by the conversion step includes the conversion step. The matrix used for the primary conversion process includes an overshoot between a plurality of channels having spectral sensitivities corresponding to the predetermined color space. It is optimized to minimize the evaluation value based on the lap amount and the width index of each channel. .

  According to the present invention, it is possible to obtain a color space suitable for white balance correction by adjusting the spectral sensitivity shape by color conversion without using a color chart.

It is a figure which shows model RGB spectral sensitivity (w = 60). It is a figure which shows the spectral sensitivity in case color temperature is low by model RGB spectral sensitivity (w = 60). Spectral sensitivity when white balance correction is performed when the spectral sensitivity is wide (w = 80), intermediate (w = 60), narrow (w = 40), model RGB spectral sensitivity and color temperature is low It is a figure which shows a sensitivity. It is an evaluation value calculation block diagram. It is a figure which shows the spectral sensitivity of type C, type N, and CIE XYZ. Type C after conversion into peaky shape, type N, is a diagram showing spectral sensitivity of _{CIE XYZ (k O = 50,} k M = 1). It is a figure which shows the processing method of white balance correction | amendment. It is a figure which shows the color space by an overlap minimization color conversion matrix. It is explanatory drawing which shows the color space image output by overlap minimization color conversion from a digital camera. It is a block diagram which shows an image processing apparatus.

  Hereinafter, an image processing apparatus and an image processing method according to embodiments of the present invention will be described with reference to the drawings. First, a spectral sensitivity shape suitable for white balance correction will be described. Consider the error in color reproduction when the difference in light source is corrected with white balance. For example, assume that there is an image sensor with spectral sensitivity as shown in FIG. This is the sensitivity with which the white balance is matched in the daylight and CIE illuminant E. When photographing with a light source having a low color temperature such as CIE A with this spectral sensitivity sensor, the spectral sensitivity of the sensor substantially increases in the long wavelength region and the spectral sensitivity of the sensor decreases in the short wavelength region ( (See FIG. 2). Performing white balance adjustment (gain correction for each RGB channel) is equivalent to adjusting the sensitivity of each of R, G, and B. However, as shown in FIG. 3, even if gain correction is performed, a peak shift in spectral sensitivity occurs, and correction cannot be made completely. The effect of this peak shift increases as the spectral sensitivity width increases. Therefore, if color conversion is performed so that the spectral sensitivity width of each channel is narrowed and white balance correction is performed, color reproduction misregistration due to different light sources can be reduced.

As shown in FIG. 3, when the range of spectral sensitivity is wide, it is easily affected by the difference in the light source. In addition, in the overlap portion between channels, a plurality of channels have sensitivity to light of the same wavelength, but when each channel is gain-corrected, different correction is performed in different channels. Therefore, an attempt is made to convert the spectral sensitivity of the camera by “color conversion in which the channels do not overlap and the width of each spectral sensitivity is narrowed”. Since the channels do not overlap, the size of the overlap between the RGB channels is calculated by the following equation (1).

R (λ), G (λ), and B (λ) are the spectral sensitivities of RGB. Then, the moment is calculated by the following formula (2) as an evaluation value related to the spectral sensitivity range.

Note that λ _Rg , λ _Gg , and λ _Bg are the centroids of the absolute values of the spectral sensitivities of the R, G, and B channels, respectively. The optimization evaluation value m is defined by the following formula (3).

K _O and k _M are weighting factors. The evaluation value calculation block diagram is shown in FIG. Then, a color conversion matrix is calculated so that the optimization evaluation value m becomes small.

  Type C and type N are used as models of the RGB spectral sensitivity of the camera (see Document A1). Type C is a model with small colorimetric color reproduction error, type N has a large colorimetric color reproduction error, but noise amplification in color conversion is small, and subjective evaluation experiment considering color reproduction and noise is more than type C Is also a preferred spectral sensitivity model. In addition, three types of spectral sensitivity models including color matching functions are used.

  [Reference A1] H. Kuniba and RS Berns, “The Effects of Sensor Spectral Sensitivity, Pixel Pitch, Photon Shot Noise, and Dark Noise on Perceived Image Quality,” IEICE Trans. Fundamentals, Vol. E92-A, pp. 3321- 3327 (2009)

  FIG. 5 shows the respective spectral sensitivities. When color conversion that minimizes the optimization evaluation value m shown in Expression (3) is obtained, the spectral sensitivities of type C, type N, and XYZ are converted into shapes as shown in FIG. Both have a shape that is more peaky than the original spectral sensitivity. X of XYZ is a double peak before conversion, but sub-peaks disappear after color conversion. In type N, since RGB is well separated before conversion, the change in shape is small even after color conversion.

  For these spectral sensitivities, we investigated how the color reproduction errors of the reference light source and test light source change with and without white balance color conversion. The reference process shown in FIG. 7A is a color correction suitable for a reference light source by photographing with a reference light source (Illumini1) and multiplying the obtained image signal (CameraRGB1) by a white balance coefficient (WB1) suitable for the reference light source. Color correction is performed using the matrix (Mtx1).

  In the first test process shown in FIG. 7 (2), a test light source (Illumini2) different from the reference light source is photographed, and the obtained image signal (CameraRGB2) is multiplied by a white balance coefficient (WB2) suitable for the test light source. Color correction is performed using a color correction matrix (Mtx1) suitable for the reference light source.

  In the second test process shown in FIG. 7 (3), a test light source (Illumini2) is photographed, and the obtained image signal (CameraRGB2) is multiplied by a white balance coefficient (WB1) suitable for the reference light source, and then the reference light source. Color correction is performed using a color correction matrix (Mtx1) suitable for the above. Further, the color conversion matrix (Mtx) is used to perform color conversion to the white balance color space, and the white balance correction suitable for the test light source is performed by gain correction (white balance) for each channel, and the white balance color space is reversed. Convert to obtain an output after color correction.

For color reproduction error, the difference between the reference process and each test process is measured by the color difference CIE 94. And in the column of WB without color conversion in Table 1 below, the color constant error between the reference process shown in FIG. 7 (1) and the first test process shown in FIG. 7 (2) is shown. In the column of WB, a color constant error between the reference process shown in FIG. 7A and the second test process shown in FIG. 7C is shown.

  As shown in Table 1, the color constant error shown in WB after color conversion is smaller than the color constant error shown in WB without color conversion. That is, the second test process has a smaller color reproduction error with respect to the reference process than the first test process. Type N has a narrow spectral sensitivity range and a small overlap between channels, so it can be said that it is advantageous for light source correction by white balance.

  Next, a color space suitable for white balance correction will be described. A color space by color conversion suitable for white balance is shown in FIG. “Type C ko50km1,” “Type N ko50km1,” “XYZ ko50km1” are the color conversions calculated to minimize the optimization evaluation value m shown in Equation (3) based on typeC, typeN, and XYZ, respectively. It is a chromaticity point.

  The color space based on any of typeC, typeN, and XYZ is wider than sRGB. TypeC and XYZ are almost the same. This is because typeC becomes almost the same shape as XYZ by matrix transformation. And typeN is narrower than typeC. The spectral sensitivity of Type N is considered to be affected by the small overlap between R and G.

  A digital camera normally outputs an image that has undergone image processing inside the camera, but a digital single-lens reflex camera or the like has a mode for outputting a “raw” file suitable for adjusting a detailed color and gradation after shooting. However, raw files are closely linked to the characteristics of image sensors for each manufacturer and model. Further, a commonly used single-plate color imaging device is not easy to handle because it requires demosaicking processing in order to handle it as image data. Therefore, it would be nice to have an explicitly defined file format that allows detailed color and gradation adjustments after shooting. As the color space of the file format, the color space of FIG. 8 has a wide color gamut, and is suitable because it can correct color reproduction for each light source satisfactorily only by white balance correction.

  Since the color space suitable for white balance correction differs between typeC and typeN, it depends on the spectral sensitivity of the image sensor, but it is reasonable to use a color matching function as the basis for defining the color space with chromaticity points. Therefore, a color space based on overlap-minimized color conversion based on XYZ is appropriate.

  A digital camera using this color space outputs a file by a process as shown in FIG. That is, the digital camera 2 converts the “raw” data (cameraRGB data) into a color space by overlap minimization color conversion, that is, a color space suitable for white balance processing, and outputs it as an image file. Output by unit 6. When the image sensor of the digital camera 2 is a single-plate color image sensor, demosaic processing is performed before color conversion. In this image file, the gradation characteristics are preferably linear or explicitly defined gradation characteristics. Furthermore, it is preferable that the dynamic range is larger than that of a normal image. In addition, it is preferable to add information indicating that the data has been converted into a color space suitable for white balance processing and output.

  Here, FIG. 10 is a block diagram showing the image processing apparatus. An image processing apparatus 10 shown in FIG. 10 processes the above-mentioned image file, and detects whether or not the image file has been converted into a color space suitable for white balance processing by the color space detection unit 12. If the color space has not been converted to a color space suitable for white balance processing, the primary conversion unit 14 performs the above-described primary conversion on the image file. Then, the image file subjected to the primary conversion is sent to the white balance adjustment unit 18. An image file that is detected to have been converted to a color space suitable for white balance processing is sent to the white balance adjustment unit 18 without performing primary conversion.

  The white balance adjustment unit 18 performs white balance processing on the image file converted into a color space suitable for white balance processing in accordance with the gray information from the gray instruction unit 16. The output color space conversion unit 20 converts the image file that has been subjected to the white balance processing into an output color space. The gradation correction unit 22 performs gradation correction on the image file converted into the output color space. The display unit 24 displays an image based on the image file subjected to gradation correction.

  Since the color space of the image file is a color space that can appropriately correct the difference in the light source only by performing the white balance processing, it can be satisfactorily performed only by performing the correction by the white balance adjustment unit 18 based on the gray information of the gray instruction unit 16. An image in which a difference in color reproduction due to a difference in light source is corrected can be displayed on the display unit 24.

  Note that an image in a standard color space such as the sRGB color space may be input to the image processing apparatus that processes the image file. In the standard color space, RGB primary colors and white chromaticity points are defined, and RGB spectral sensitivity can be defined based on these. Therefore, primary conversion is preferably obtained in advance based on the RGB spectral sensitivity corresponding to the standard color space, and color conversion is performed by the primary conversion to perform white balance correction.

  DESCRIPTION OF SYMBOLS 2 ... Digital camera, 4 ... Color conversion part, 6 ... Output part, 10 ... Image processing apparatus, 12 ... Color space detection part, 14 ... Primary conversion part, 16 ... Gray instruction | indication part, 18 ... White balance adjustment part, 20 ... Output color space conversion unit, 22 ... gradation correction unit, 24 ... display unit.

Claims (5)

An image processing device that performs color conversion of an image captured by an image sensor,
Color information including chromatic colors under a different type of light source different from a standard light source, obtained based on information related to spectral sensitivity of the image sensor, is primary in a color space for converting color information under the standard light source into color information. A primary conversion unit for conversion,
The matrix used for the conversion processing by the primary conversion unit is optimized so that the evaluation value based on the overlap amount between the plurality of channels of the image sensor and the width index of each channel is minimized. An image processing apparatus. The image processing apparatus according to claim 1.
Along with image output, information indicating that the color space of the image output is primary converted into color space for converting color information including chromatic colors under the different light source into color information under the standard light source is output. An image processing apparatus. An image processing method for obtaining a color balance by processing an image captured by an image sensor,
Color information including chromatic colors under a different type of light source different from a standard light source, obtained based on information related to spectral sensitivity of the image sensor, is primary in a color space for converting color information under the standard light source into color information. Including a conversion step to convert,
The conversion process by the conversion step includes a gain adjustment step of adjusting the white balance by adjusting the gain of the color information after conversion obtained by the conversion step,
The matrix used for the primary conversion process is optimized so that an evaluation value based on an overlap amount between a plurality of channels of the image sensor and a width index of each channel is minimized. Image processing method. The image processing method according to claim 3.
When information indicating that the color space of the image is primary converted into color space for converting color information including chromatic colors under the different light source into color information under the standard light source is added An image processing method, wherein the primary conversion is not performed. An image processing method for obtaining a color balance by processing an image in a predetermined color space,
Color for converting color information including chromatic colors under a different type of light source different from the standard light source, obtained based on information related to spectral sensitivity corresponding to a predetermined color space, to color information under the standard light source Including a transformation step for primary transformation into space,
The conversion process by the conversion step includes a gain adjustment step of adjusting the white balance by adjusting the gain of the color information after conversion obtained by the conversion step,
The matrix used for the primary conversion processing is optimized so that the evaluation value based on the overlap amount between the plurality of channels of spectral sensitivity corresponding to the predetermined color space and the width index of each channel is minimized. An image processing method characterized by being a thing.

Images