JP2006013567A - Imaging apparatus and signal processing method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus and a signal processing method thereof capable of properly applying spectral sensitivity correction to any of green colors of coloring matters or the like and green colors of leaves of naturally grown trees or the like so as to be capable of obtaining excellent color reproducibility without the need for an IRCF. <P>SOLUTION: The imaging apparatus comprises: a color signal generating means (2) that includes red, green, blue color filters (26, 27, 28) and a color filter (29) the peak of the spectral transmittance of which exists in a wavelength region shorter than that of the green color filter and having nearly the same spectral transmittance in the near infrared ray region as that of the green color filter and outputs first to fourth color signals; a spectral sensitivity correction means (4) that carries out a matrix arithmetic operation for multiplying a matrix coefficient with the first to fourth color signals to generate a corrected color signal; and a coefficient setting means (6) that discriminates whether or not a combination of the first to fourth color signals indicates an exceptional color and switches the matrix coefficient depending on a result of the discrimination. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image pickup apparatus having a signal processing for performing an image correction with a good color reproducibility by performing a relative visibility correction without using an infrared removal filter required for the relative visibility correction in the image pickup apparatus, and The present invention relates to a signal processing method.

A conventional imaging device has a lens that forms incident light, an imaging device that converts an optical image formed by the lens into an electrical signal, and a signal process performed on the electrical signal obtained from the imaging device to obtain a predetermined signal. Video signal processing means for obtaining a video signal.
When an imaging device is configured with only one CCD (Charge Coupled Device) sensor or CMOS (Complementary Metal Oxide Semiconductor) sensor used as a normal imaging device, that is, in a single-plate sensor, as a color filter that performs color separation, Different colors for each pixel are provided on the sensor.
In order to obtain red (R), green (G), and blue (B) color signals, R, G, and B primary color filters that transmit light bands corresponding to R, G, and B are used. In some cases, magenta (Mg), cyan (Cy), yellow (Ye), and G complementary color filters are used. Each of the above color filters is designed to have a spectral transmission characteristic so as to transmit a target color using a dye or pigment, but has a certain transmittance even in the near infrared region. Further, since the photoelectric conversion unit of the image sensor is mainly composed of a semiconductor such as silicon (Si), the spectral sensitivity characteristic of the photoelectric conversion unit is sensitive to near infrared light having a long wavelength. Therefore, the signal obtained from the image sensor provided with the color filter also reacts to light in the near infrared region.

  On the other hand, the color vision characteristic, which is a sensitivity characteristic for human colors, and the relative visual sensitivity characteristic, which is a sensitivity characteristic for brightness, are sensitivity characteristics from 380 nm to 780 nm, which are referred to as the visible range, and in a wavelength range longer than 700 nm. Has little sensitivity. Therefore, in order to match the color reproducibility of the image pickup device with human color vision characteristics, an infrared ray removal filter for correcting visibility (hereinafter referred to as IRCF: Infrared Cut Filter) that does not allow light in the near infrared region to pass in front of the image pickup device. It was necessary to provide.

  On the other hand, in the case where sensitivity is more important than color reproducibility, for example, in a surveillance camera, it is better to make the imaging device receive near infrared light without providing IRCF in order to use light in the near infrared region.

  Therefore, in order to be able to cope with both color image pickup that emphasizes color reproducibility and high-sensitivity image pickup, when importance is attached to color reproducibility, an IRCF is installed in front of the image sensor and importance is attached to sensitivity. Sometimes, in order to receive near-infrared rays, a mechanism means for moving the IRCF is provided, or an IRCF is provided in a part of the diaphragm for adjusting the amount of incident light, and the IRCF is installed in front of the image sensor in accordance with the amount of light. Various techniques for removing and removing these have been proposed (see, for example, Patent Document 1).

  Further, white balance is taken without installing the IRCF, and sensitivity is improved by generating a luminance signal with a color mixture ratio different from the color mixture ratio of the R, G, and B signals that generate the luminance signal when the IRCF is installed. Technology has also been proposed (see Patent Documents 2 and 3).

JP 2001-36807 A (page 3-4, FIGS. 1 and 2) JP2003-134522A (page 4-5, FIG. 1) JP 2003-264843 A (page 4, FIG. 1)

However, the conventional imaging device described in Patent Document 1 requires a mechanism means for moving the IRCF, which is disadvantageous for downsizing the unit including the imaging element, and is desired to have a configuration that does not use the IRCF. .
In addition, a simple image pickup device (for example, a PC (PC) camera, a mobile phone camera, a toy (TOY) camera, a consumer monitoring camera) that adjusts the amount of light using an electronic shutter of the image pickup element has an aperture mechanism. In many cases, a new mechanism for attaching and detaching the IRCF must be provided.

  Furthermore, the imaging devices described in Patent Documents 2 and 3 have no problem when obtaining a black and white video signal, but when obtaining a color video signal, the color signal only adjusts the white balance, and the luminance signal is also human. Therefore, the color image signal is different from the R, G, and B values determined by the human color vision characteristic or the spectral sensitivity characteristic obtained from the linear conversion. That is, the video signal has a large color difference ΔE * ab (JIS Z8730), and accurate color reproducibility cannot be obtained.

  Furthermore, even if the spectral sensitivity can be corrected appropriately for the green color of the paint, the green color of the green leaves of the tree that appears to be the same color (same color) as the human eye. There is a problem that the spectral sensitivity cannot be appropriately corrected for the color of the image.

  The present invention has been made in order to solve the above-described problems, and appropriately corrects spectral sensitivity for any of the green color of a paint or the green color of a tree that grows naturally without IRCF. An object of the present invention is to provide an image pickup apparatus and a signal processing method that can obtain good color reproducibility.

This invention
The first, second, and third color filters that extract red, green, and blue light, and one of the first to third color filters has a correlation in the visible region, and has a maximum spectral transmittance. A fourth color filter having a spectral transmittance that is shorter than the wavelength at which the spectral transmittance of the one color filter is maximized and having substantially the same spectral transmittance as that of the one color filter in the near-infrared region. Color signal generating means for outputting first to fourth color signals corresponding to the spectral transmittances of the respective color filters;
Spectral sensitivity correction means for generating a corrected color signal by performing a matrix operation of multiplying at least three color signals among the first to fourth color signals by a matrix coefficient;
In response to the first to fourth color signals output from the color signal generation means, it is determined whether or not a combination thereof represents a predetermined specific exceptional color. An image pickup apparatus having coefficient setting means for switching matrix coefficients is provided.

  According to the present invention, the sensitivity of the near-infrared region is corrected without IRCF, and the same color (equal color) is seen by the human eye. In any case, good color reproduction can be realized.

Embodiment 1 FIG.
FIG. 1 is a diagram showing a schematic configuration of an imaging apparatus according to Embodiment 1 of the present invention. As shown in the figure, the imaging apparatus includes a color signal generation unit 2, a spectral sensitivity correction unit 4, a coefficient setting unit 6, and a video signal processing unit 8.

The color signal generation means 2 receives incident light and outputs first video signals corresponding to the incident light, for example, first to fourth color signals Rb, G1b, G2b, and Bb.
The spectral sensitivity correction unit 4 is configured to detect the near infrared component, that is, the color signal generation unit 2 has a spectral sensitivity in the near infrared region from the first video signal output from the color signal generation unit 2. A signal component included in the video signal is removed, and second video signals, for example, red, green, and blue color signals Rc, Gc, and Bc are generated.
The coefficient setting unit 6 receives the first to fourth color signals Rb, G1b, G2b, and Bb output from the color signal generation unit 2, and these combinations represent a predetermined specific exceptional color. And the coefficient matrix is switched according to the determination result.
The video signal processing unit 8 converts the second video signal output from the spectral sensitivity correction unit 4 into a signal (third video signal) suitable for output to the outside.

  For example, as shown in FIG. 1, the color signal generation unit 2 includes an imaging unit 11, an amplification unit 12, an A / D converter (ADC) 13, a direct current component reproduction unit (DC reproduction unit) 14, and a front white And a balance (WB) means 15.

For example, as shown in FIGS. 2 and 3, the imaging unit 11 includes an optical system 21 including a lens and an imaging element 22 having a plurality of photoelectric conversion elements that are two-dimensionally arranged and each constitute a pixel. The plurality of photoelectric conversion elements of the imaging element 22 are covered with a color filter group 23 as color separation means, for example, as shown in FIG.
The plurality of photoelectric conversion elements are divided into first to fourth groups.
The color filter group 23 includes a plurality of red filters (R filters) 26 provided for each of the first group of photoelectric conversion elements, and a plurality of color filters provided for each of the second group of photoelectric conversion elements. Green filter (G1 filter) 27, a plurality of blue filters (B filters) 28 provided for each of the third group of photoelectric conversion elements, and each of the fourth group of photoelectric conversion elements A plurality of green filters (G2 filters) 29 are provided, and these are arranged as shown.

As shown in FIG. 2, in every other row (the first, third and fifth rows from the top in FIG. 2), the B filter 28 and the G1 filter 27 or G2 filter 29 are alternately arranged, that is, “B−”. G1-B-G2 "is repeated, and in a row (second and fourth columns from the top in FIG. 2) located between the other rows, the G2 filter 29 or the G1 filter 27 The R filters 26 are provided alternately, that is, so as to repeat “G2-R-G1-R”.
In every other column (first, third and fifth columns from the left in FIG. 2), the B filter 28 and the G2 filter 29 or the G1 filter 27 are alternately arranged, that is, “B-G2-B-G1”. In the column (second and fourth columns from the left in FIG. 2) located between the other columns, the G1 filter 27 or the G2 filter 29 and the R filter 26 are alternately arranged. That is, it is provided to repeat “G1-R-G2-R”.

  The R filter 26 mainly passes only light in the first wavelength band corresponding to red, and the G1 filter 27 mainly passes only light in the second wavelength band corresponding to green. The B filter 28 mainly passes only light in the third wavelength band corresponding to blue, and the G2 filter 29 mainly passes only light in the fourth wavelength band corresponding to green.

FIG. 4 shows the spectral sensitivity characteristic of the photoelectric conversion element provided with the R filter (that is, the spectral sensitivity characteristic of the combination of the R filter and the pixel) r (λ), and the spectral sensitivity characteristic of the photoelectric conversion element provided with the G1 filter. g1 (λ), spectral sensitivity characteristic b (λ) of the photoelectric conversion element provided with the B filter, and spectral sensitivity characteristic g2 (λ) of the photoelectric conversion element provided with the G2 filter are shown.
The spectral sensitivity characteristic shown in FIG. 4 is a characteristic of a combination of the transmittance of the color filter and the spectral sensitivity characteristic of a photoelectric conversion element that constitutes each pixel of the imaging means 11, for example, a photodiode. Since the photoelectric conversion element constituting each pixel of the image pickup means 11 has sensitivity up to around 1000 nm, r (λ), g1 (λ), g2 (λ), and b (λ) are substantially equal to the spectral transmittance of the color filter. It corresponds.

  The G1 filter 27 and the G2 filter 29 both transmit light in the wavelength region corresponding to green as described above, and g1 (λ) in the visible region (approximately 400 nm to approximately 700 nm) visible to the human eye. And g2 (λ) have a strong correlation (the envelope shape is similar), but g2 (λ) has a shorter peak position (wavelength at which the spectral transmittance is maximum) than g1 (λ). It is shifted by about 50 nm to the wavelength side. Therefore, the G2 filter 29 has a transmittance in a region slightly cyan than the G1 filter 27. On the other hand, at about 700 nm or more, the G1 filter 27 and the G2 filter 29 have substantially the same transmittance characteristics.

  Light incident from the optical system 21 including the lens forms an image on the light receiving surface of the image sensor 22. As described above, the image sensor 22 is covered with the color filter group 23, and the color components corresponding to the spectral transmittance of the color filter group 23, that is, R, G 1, G 2, and B, from each photoelectric conversion element. An analog video signal is output.

  In this way, R, G1, G2, and B analog signals output from the imaging means 11 (hereinafter referred to as “R signal”, “G1 signal”, “G2 signal”, and “B signal”, respectively). ) Is amplified by the amplification means 12. The video signal output from the amplifying means 12 is converted into a digital signal by the ADC 13.

  The DC level of the video signal converted into the digital signal is reproduced by the DC reproducing means 14. In DC reproduction, the offset level held before the A / D conversion by the ADC 13 is DC-shifted or clamped so that the black level of the normal video signal becomes “0”. Outputs Ra, G1a, G2a, Ba of the DC reproducing means 14 are supplied to the front white balance means 15.

  As shown in FIG. 5, the front white balance means 15 amplifies the signals Ra, G1a, G2a, Ba and outputs four amplified signals Rb, G1b, G2b, Bb, respectively. 31g2, 31b, and amplifying signals Rb, G1b, Bb are integrated for all the pixels in one screen, and integrated values 32r, 32g1, 32b for outputting integrated values ΣRb, ΣG1b, ΣBb, and integrating means 32r, Based on the integrated values ΣRb, ΣG1b, and ΣBb output from 32g1 and 32b, for example, amplification is performed so that the values obtained by dividing the integrated values ΣRb, ΣG1b, and ΣBb by the number of pixels that are targets of integration are equal to each other. While controlling the amplification factors of the means 31r, 31g1, 31b, the amplification factor of the amplification device 31g2 is the same as the amplification factor of the amplification device 31g1. And a gain control unit 33 for controlling the.

  The white balance means usually makes the R, G, B signals corresponding to the achromatic portion of the subject equal to each other. Normally, when the white balance is a general subject, the color in one screen is By using the statistical result that, on average, the color is close to an achromatic color (Evans principle), the amplification factor for each color signal is controlled so that the integrated values are the same for all pixels on the screen. Do.

  However, this is based on the premise that the R, G, and B signals respectively correspond to the color vision characteristics of the person, and the R, G1, and B signals to be subjected to white balance processing are in the near infrared region as shown in FIG. In the case of having spectral sensitivity characteristics, even if the integrated values of the R, G1, and B signals are equal to each other, the white balance is not taken within the visible range that can be seen by human eyes, and the color reproducibility is not always good. is not. The white balance by the previous white balance means 15 is not to improve the color reproducibility but to make the ratios of R, G1, G2, and B uniform for illumination of various color temperatures.

  That is, the values of R, G1, G2, and B correspond to the sum of the products of the spectral sensitivity of the image sensor, the reflectance of the subject, and the spectral characteristics of the illumination, so even if the same achromatic subject is imaged, R The ratio of the G1, G2, and B signals changes with the change in the illumination color temperature, but the white balance processing by the previous white balance means 15 can remove the influence of the illumination color temperature.

  The spectral sensitivity correction means 4 receives R, G1, G2, and B signals Rb, G1b, G2b, Bb after white balance output from the previous white balance means 15 as input, and performs matrix operations to be described in detail later on this input signal. As a result, color signals R, G, and B signals Rc, Gc, and Bc in which the influence on the color reproducibility due to the near-infrared sensitivity characteristic of the image sensor is corrected are obtained.

  As described above, the video signal processing unit 8 converts the signals Rc, Gc, and Bc output from the spectral sensitivity correction unit 4 into signals suitable for output to the outside. For example, FIG. As shown, it has a post-white balance means 16, a gamma (γ) correction means 17, and a luminance / color difference signal generation means 18.

  The post-white balance means 16 performs white balance processing on the R, G, B signals Rc, Gc, Bc corrected by the spectral sensitivity correction means 4, and as shown in FIG. 6, Rc, Gc, The three amplifying means 51r, 51g, 51b for amplifying the Bc signal and outputting the signals Rd, Gd, Bd and the signals Rd, Gd, Bd are integrated for all the pixels in the position screen, respectively, and integrated values ΣRd, ΣGd Based on the integrated values ΣRd, ΣGd, and ΣBd output from the integrating means 52r, 52g, and 52b that output ΣBd and the integrating means 52r, 52g, and 52b, for example, the integrated values ΣRd, ΣGd, and ΣBd are to be integrated. Gain control means for controlling the amplification factors of the amplification means 51r, 51g, 51b so that the values obtained by dividing by the number of pixels obtained are equal to each other With a 3 and.

  Here, the R, G, and B signals Rc, Gc, and Bc input to the amplifying units 51r, 51g, and 51b are signals in which the near-infrared sensitivity characteristics are corrected by the spectral sensitivity correcting unit 4, and are almost visible. It consists only of ingredients. Therefore, the Rd, Gd, and Bd signals obtained by the white balance processing by the post white balance means 16 have good color reproducibility.

  The gamma correction unit 17 performs non-linear tone conversion on the video signals Rd, Gd, and Bd output from the rear white balance unit 16.

The luminance color difference signal generation means 18 converts the R, G, B signals Re, Ge, Be output from the gamma correction means 17 into a luminance signal (Y signal) and two color difference signals (Cr signal, Cb signal).
In this conversion (YCrCb conversion), the luminance / chrominance signal generation means 18 generates a Y, Cr, Cb signal by performing a linear matrix operation of the following equation (1), which is usually multiplied by a coefficient matrix of 3 rows and 3 columns. .

  In the formula (1), the matrix coefficient of 3 rows and 3 columns is defined by, for example, IEC (International Electrotechnical Commission) 61966-2-1, y1 = 0.2990, y2 = 0.5870, y3 = 0. 1140, cr1 = −0.1687, cr2 = −0.3313, cr3 = 0.5000, cb1 = 0.5000, cb2 = −0.4187, cb3 = −0.0813.

Hereinafter, the configuration and principle of the spectral sensitivity correction unit 4 will be described.
For example, the spectral sensitivity correction unit 4 performs a matrix operation by multiplying the matrix coefficient of 3 rows and 4 columns shown in the following equation (2), and R, G1, G2, B signals Rb, G1b, G2b, Bb to R, G, B signals Rc, Gc, and Bc are generated. In Expression (2), e11 to e34 are predetermined constants.

  In order to generate the R, G, and B signals Rc, Gc, and Bc, it is not essential to use the four signals R, G1, G2, and B. R, G1, B, or R, G2, B It is also possible to generate R, G, and B signals from these three signals. When only three signals R, G1, and B are used, e13 = e23 = e33 = 0 in the equation (2). When only three signals R, G2, and B are used, e12 = e22 = e32 = 0 in equation (2), and in either case, it is generally expressed by equation (2). be able to.

Next, correction by the spectral sensitivity correction unit 4 of the present invention will be described in detail.
FIG. 7 shows spectral sensitivity characteristics representing human color vision characteristics. The characteristic shown in FIG. 7 is an average value of the color matching function of a normal color vision person, and is defined by CIE (Commission Internationale de l'E'claage) 1931. Colors that humans feel ignore the functions such as chromatic adaptation, and can be simply expressed as R, G, and B spectral sensitivity characteristics (color matching functions), reflection spectral characteristics of the subject, and spectral characteristics of illumination. Can be expressed as a value obtained by multiplying the multiplication results in the visible range. As shown in FIG. 7, the sensitivity characteristic of human beings has a sensitivity only from about 380 nm to 780 nm, so-called visible range, in particular, the sensitivity in the range of 400 nm to 700 nm is high, and the sensitivity is almost in the longer wavelength side than 700 nm. There is no.

  On the other hand, when the color filter group 23 that performs color separation is provided in the imaging unit 11, a signal corresponding to the product of the spectral transmittance of the color filter and the spectral sensitivity of the imaging element is output from the imaging unit 11. The means has a spectral sensitivity from the visible region to the near-infrared region (near 1000 nm) because an image pickup device that performs photoelectric conversion, such as a photodiode, is formed of a semiconductor such as Si (silicon). Further, since the R color filter has a relatively high transmittance in the near-infrared region, the near-infrared light is incident on the image sensor 22. Further, the B color filter for incident B light and the G color filter for incident G light similarly have a constant transmittance in the near infrared region. This is because RGB color filters are usually constructed using dyes and pigments containing each color, but the spectral transmittance depends on the material to be constructed, and from the visible region on the long wavelength side to the near infrared region. This is because it has the characteristic of increasing the transmittance again when applied.

  Accordingly, the spectral sensitivity characteristics of the imaging means given by these products are as shown by solid lines r (λ), g (λ), and b (λ) in FIG. When two types of green filters are provided as in the present embodiment, the spectral sensitivity characteristics of the imaging means 11 are indicated by solid lines r (λ), g1 (λ), g2 (λ ), B (λ).

  Spectral sensitivity characteristics r (λ), g (λ), b (λ) of the image pickup means indicated by a solid line in FIG. 8, and spectral sensitivity characteristics r (λ), g1 (of the image pickup means 11 indicated by a solid line in FIG. Since λ), g2 (λ), and b (λ) are different from the color matching functions shown in FIG. 7 and are particularly different in the near-infrared region, ordinary imaging devices pass light in the near-infrared region. An infrared cut filter (IRCF) to be removed is provided in front of the image sensor to correct the spectral sensitivity. The spectral transmission characteristic IRCF (λ) of IRCF is also shown by a solid line in FIG. The characteristic obtained by multiplying the IRCF (λ) and the RGB spectral sensitivity characteristics (r (λ), g (λ), b (λ)) corresponds to the RGB signal of the image pickup means provided with the conventional IRCF. The spectral sensitivity characteristics r ′ (λ), g ′ (λ), and b ′ (λ) of the respective colors are shown, and the characteristics are shown by broken lines in FIG.

  Further, even if the conventional imaging device has the spectral sensitivity characteristic represented by the broken line in FIG. 8, the negative characteristic (a negative value of r (λ) from about 450 nm to 540 nm) cannot be realized. In some cases, the RGB signal obtained from the image pickup means is subjected to a matrix calculation by multiplying a coefficient matrix of 3 rows and 3 columns as shown in Expression (3), thereby performing color correction.

  However, when IRCF is not used, the signal output by the sensitivity characteristic by near infrared rays has a great influence on the color reproducibility, and even if linear matrix calculation is performed by multiplying the coefficient matrix of 3 rows by 3 columns as described above. In some cases, good color reproducibility could not be obtained.

  If an extra signal component obtained from the near-infrared region can be removed from the original signal by processing the signal obtained from the imaging means, good color reproducibility can be realized without using IRCF.

  In the present embodiment, the color signal generation unit 2 generates two G1 signals and G2 signals as green color signals, and the spectral sensitivity correction unit 4 performs correction by matrix calculation represented by the above equation (2). That is why we do it. The matrix coefficient used in the matrix calculation of the above equation (2) is determined as follows.

  The spectral sensitivity characteristics of the imaging means indicated by the solid line in FIGS. 8 and 4 and the spectral sensitivity characteristics indicated by the broken line in FIG. 8 as the color target differ greatly in the spectral characteristics particularly in the near infrared region. By appropriately determining the matrix coefficients shown in (2), satisfactory color reproducibility can be obtained while satisfying the condition color. Here, the condition equal color means that two color stimuli having different spectral characteristics appear to be the same color under a specific observation condition. Therefore, the matrix coefficient has a spectral sensitivity that gives the best color reproducibility when a specific subject is imaged using a specific illumination, that is, a signal substantially equal to that when using the IRCF without using the IRCF. It is determined so as to be obtained on the output side of the correction means 4.

A specific method for obtaining the matrix coefficient is, for example, as follows.
As the illumination, a specific color temperature, for example, 5000K illumination is used. FIG. 9 shows the spectral characteristics of illumination with a color temperature of 5000K.
A standard color chart such as a Macbeth color checker is used as the subject. Hereinafter, 24 color patches (color charts) of Macbeth Color Checker are used.

  The Macbeth color checker includes 24 color patches that represent existing colors as subjects and are selected with emphasis on human memory colors (skin color, plant green, sky blue, etc.). The spectral reflectance from 300 nm to 1200 nm of the color patch of color is shown in FIG. In FIG. 10, color patch numbers 1 to 24 correspond to the following colors, respectively.

1: Dark skin color (Dark skin),
2: Bright skin color (Light Skin),
3: Blue sky color (Blue sky),
4: Grass color (Foliage),
5: Blue flower,
6: Blue green
7: Orange (Orange),
8: Purple blue
9: Middle red (Moderate red)
10: Purple
11: Yellow green
12: Orange yellow,
13: Blue (Blue),
14: Green,
15: Red (Red),
16: Yellow,
17: Magenta,
18: Cyan,
19: White (White),
20: Gray 8 (Neutral 8),
21: Gray 6.5 (Neutral 6.5),
22: Gray 5 (Neutral 5),
23: Gray 3.5 (Neutral 3.5),
24: Black
(The Japanese translation of each color patch described above is based on the 2nd edition of the new edition of Color Chemistry Handbook, edited by the Japan Society of Color Science.)

  An integrated value of the spectral characteristic of the specific illumination shown in FIG. 9, the spectral reflectance of the subject shown in FIG. 10, and the spectral sensitivity characteristic shown by the broken line in FIG. 8 is obtained.

The signals obtained on the output side of the spectral sensitivity correction means 4 when each color patch of the Macbeth color checker is imaged by the image pickup apparatus of FIG. 1 are the spectral characteristics of illumination (for example, those shown in FIG. 9), and the Macbeth color checker. The spectral reflectance of each color patch (shown in FIG. 10), the spectral sensitivity characteristic of the video signal generation means 2 (substantially equal to the spectral sensitivity characteristic shown by the solid line in FIG. 4), and the response of the spectral sensitivity correction means 4 It is obtained by integrating the product with the characteristics over all wavelengths.
On the other hand, signals (target signals) Rt, Gt, and Bt to be obtained on the output side of the spectral sensitivity correction means 4 when imaging each color patch of the Macbeth color checker with the imaging apparatus of FIG. 9), the spectral reflectance (shown in FIG. 10) of the 24-color patch of the Macbeth color checker, and the spectral sensitivity characteristic indicated by the broken line in FIG. It is obtained by accumulating.
Therefore, the values of the coefficients e11 to e34 are determined so that the values of the signals Rc, Gc, and Bc obtained on the output side of the spectral sensitivity correction unit 4 are closest to the values of the target signals Rt, Gt, and Bt. Whether or not they are the closest is determined by the least square error method, that is, by determining the sum of the squares of the difference between the two corresponding values and determining whether or not the sum is the minimum.

  As described above, the matrix coefficient for the normal color is obtained, for example, for the 24 colors of the Macbeth color checker through the matrix calculation using the spectral sensitivity characteristic of FIG. 4 and the matrix coefficient shown in Expression (2). The matrix coefficient is calculated by the least square error method so that the RGB value is closest to the RGB value obtained through the target spectral sensitivity characteristic (for example, the broken line in FIG. 8) for the 24 colors of the same Macbeth color checker. calculate.

  Note that although the matrix coefficient of 3 rows and 4 columns is shown in Equation (2), it can also be realized by a matrix coefficient of 3 rows and 3 columns using only G1 as an input term. When the matrix coefficient is a matrix coefficient of 3 rows and 3 columns, R, G1 (or G2) is approximated to the RGB value obtained via the target spectral characteristic (for example, the broken line in FIG. 8). , B can be similarly calculated by the least square error method using the input light.

  It is possible to approximate the RGB value that should be the target color even if the matrix coefficient of 3 rows and 3 columns using only G1 is used as the conventional imaging device has three types of RGB color filters. However, when both G1 and G2 are used as input terms and the matrix coefficient is 3 rows by 4 columns, the spectral sensitivity characteristic peak is shifted to the cyan side of G2, and therefore the matrix coefficient of G2 is negative. 7 has an advantage that the characteristic of the portion where the R value of the color matching function is negative can be easily realized, and the color difference ΔE can be reduced in color reproducibility.

  However, the matrix coefficients determined as described above are calculated using a standard color chart, for example, a Macbeth color checker, as an object, and are not necessarily appropriate for the colors of all actual objects. In particular, the Macbeth color checker has been created to cover as many characteristic colors as possible in nature, but it can be said that its characteristics have been created considering only the visible range. On the other hand, when considering the near-infrared region, there may be a color different from the color of the Macbeth color checker. Since the Macbeth Color Checker is a chart, pigments such as paints are used for the color patches. On the other hand, for example, a leaf that looks the same green color in the visible range has a spectral reflectance different from the green color of the paint.

  In FIG. 11 and FIG. 12, “grass color (Foliage)”, “yellow green”, and “green”, which are listed as green examples in the Macbeth color checker, are naturally occurring in nature. The spectral reflection characteristics of two types of tree leaves are shown. For example, “Foliage” is a color patch created to reproduce the color of the leaves of a tree, but its spectral characteristics are different from those of an actual tree, particularly from 700 nm in the near infrared region. Is characterized in that its reflectance increases rapidly from 700 nm. As described above, since the human eye has almost no sensitivity from 700 nm, the color of the color patch and the color of the leaves of the tree appear to be almost the same color, but FIG. 8 and FIG. 4 that are sensitive to light in the near infrared region. The obtained color reproduction differs greatly in the spectral sensitivity characteristics of the image pickup means shown in FIG. 5, and specifically, the R signal becomes larger than necessary, so that green is a brown color reproduction.

  As described above, the color of the subject represented by pigments such as paint and paint can achieve good color reproducibility with the above matrix coefficient, but the formula ( Even if the matrix coefficient of 2) is determined, the color of the leaves of the tree, etc., is significantly different from that of the target color in the range of 700 nm or more. That is, even if the matrix coefficient of equation (2) is determined so as to be the same color as the green color patch shown in FIG. When correction is performed using the matrix coefficients defined above, color reproducibility is poor.

  Therefore, a color having a large reflectance from 700 nm, such as a leaf color, is an exceptional color, and for the exceptional color, a matrix coefficient different from the matrix coefficient of the normal color is determined and applied to the tree leaf. Even good color reproducibility is achieved.

The matrix coefficients suitable for the exceptional colors are the same exceptional colors when the RGB values obtained through the matrix operation using the spectral sensitivity characteristics of FIG. 4 and the matrix coefficients shown in Equation (2) are the same for the exceptional colors. On the other hand, it can be calculated by the least square error method so as to be closest to the RGB value obtained through the spectral sensitivity characteristic (for example, the broken line in FIG. 8) as a color target.
For distinction, the matrix coefficient obtained as described above for the normal color is called a first matrix coefficient, and the matrix coefficient obtained for the exceptional color is called a second matrix coefficient.

  The second matrix coefficient is also realized by a matrix coefficient of 3 rows and 3 columns using R, G1 (or G2), and B instead of being realized by a matrix coefficient of 3 rows and 4 columns as in the case of the first matrix coefficient. You can also.

The coefficient setting unit 6 receives the first to fourth color signals Rb, G1b, G2b, and Bb output from the color signal generation unit 2, and determines whether or not these combinations represent predetermined exceptional colors. If it is determined that the color is an exceptional color, the second matrix coefficient is selected. Otherwise, the first matrix coefficient is selected, and the selected matrix coefficient is supplied to the spectral sensitivity correction means 4. .
The coefficient setting unit 6 includes a color identification unit 42 and a coefficient determination unit 43 as shown in FIG.
The color identification unit 42 receives the first to fourth color signals Rb, G1b, G2b, and Bb output from the color signal generation unit 2, and determines whether these satisfy a predetermined condition, A determination signal DS indicating the determination result is output. That is, when a predetermined condition is satisfied, it is determined that the color is an exceptional color, and the determination signal DS is set to a first value, for example, “1”. In other cases, the determination signal is set to a second value, for example, “0”.
The coefficient determination unit 43 outputs an appropriate matrix coefficient to the spectral sensitivity correction unit 4 according to the value of the determination signal supplied from the color identification unit 42.

For example, the color identification unit 42 determines that the color is an exceptional color when both of the following inequalities (4) and (5) are satisfied.
Rb / G1b> k1 (4)
G1b / G2b> k2 (5)
In equations (4) and (5), k1 and k2 are predetermined constants.

Hereinafter, the reason why the exceptional color can be appropriately detected by such determination will be described.
Since the reflectance of the leaves of the tree suddenly increases from 700 nm, the ratio of the spectral sensitivity g1 (λ) value G1b and the value Rb of r (λ) of the imaging means 11 shown in FIG. It is possible to distinguish between green leaves that are growing naturally. That is, when Rb / G1b is larger than a predetermined value, it can be determined as an exceptional color.

  On the other hand, since the spectral sensitivity band of r (λ) shown in FIG. 4 is very wide from about 570 nm to 1000 nm, if only the discriminant of Rb / G1b> k1 is used, other colors are also discriminated as green leaves. There is a possibility. For example, FIG. 13 shows spectral reflectances of colors such as skin color (“light skin color”) and brown (“dark skin color”). When looking only at the spectral reflectance up to 700 nm, the bright skin color and the color of the leaves of the leaves are clearly different, but when including up to the near infrared, both colors are very similar and the value of R / G1 is almost the same. Has the same value. On the other hand, when comparing the region of g1 (λ) (near 540 nm) and the region of g2 (λ) (region close to cyan, ie, near 500 nm), in the case of leaves, the value in the region of g1 (λ) is g2 Although it is larger than the value in the region (λ), there is no such tendency in the case of “light skin color” or “dark skin color”. That is, in the case of “bright skin color”, the value in the g2 (λ) region is larger than the value in the g1 (λ) region. In the case of “dark skin color”, the value in the g1 (λ) region and the value in the g2 (λ) region are substantially the same. Therefore, it is possible to distinguish a leaf from “light skin color” or “dark skin color” depending on whether G1b is larger than G2b.

  The color discriminating unit 42 discriminates the color according to the above equations (4) and (5), and outputs an identification signal to the coefficient determining unit 43 when it is determined that the color is an exceptional color (leaves of the tree). The coefficient determination unit 43 supplies the second matrix coefficient to the spectral sensitivity correction unit 4 when the color is an exceptional color, and supplies the first matrix coefficient to the spectral sensitivity correction unit 4 when the color is not an exceptional color.

  The spectral sensitivity correction unit 4 performs matrix calculation using the matrix coefficient supplied from the coefficient determination unit 43, thereby performing spectral sensitivity correction. That is, the matrix calculation is normally performed using the first matrix coefficient, and the matrix calculation is performed using the second matrix coefficient when the color is an exceptional color.

  In this way, colors different from pigments such as leaves are treated as exceptional colors, and the matrix determination coefficient is switched to an appropriate matrix coefficient by the coefficient determination means 43, so that appropriate color reproducibility can be achieved for all colors. This can be realized, and it becomes possible to correct an unnecessary signal incident near infrared rays.

  In the above example, the coefficient determination unit 43 outputs the first matrix coefficient or the second matrix coefficient in accordance with the detection signal of the color identification unit 42, and the spectral sensitivity correction unit 4 calculates the supplied matrix coefficient. The matrix calculation is performed by using the first matrix coefficient and the second matrix coefficient stored in the spectral sensitivity correction unit 4 instead of the above configuration, and the detection signal of the color identification unit 42 is used as a detection signal. Accordingly, any one of the first matrix coefficient and the second matrix coefficient may be selected and used for the matrix calculation. In this case, the coefficient setting means 6 does not have to be provided with the coefficient determination means 43. However, in this case, it can also be seen that the coefficient determination means is built in the spectral sensitivity correction means 4.

  In the above example, the front white balance means 15 and the rear white balance means 16 integrate the color signals for one screen, but they may be integrated over one screen or more.

  In the above example, the fourth color filter has a strong correlation in the visible region with the second color filter that extracts green light, and the spectral transmittance peak is the spectral spectrum of the second color filter. The one having a wavelength shorter than the transmittance and having substantially the same spectral transmittance as that of the second color filter in the near-infrared region is used, but the first color filter or the third color filter is used. A filter having the above relationship may be used as the fourth color filter.

  Furthermore, in the above example, the signal obtained when the imaging unit 11 has the spectral sensitivity characteristic indicated by the broken line in FIG. 8 is used as the target signal. However, the spectral sensitivity characteristic equal to the color matching function of FIG. A signal selected in the case of having a signal may be used as a target signal. That is, the total characteristics from the color signal generation means 2 to the spectral sensitivity correction means 4 are CIE1931 color matching functions or spectral sensitivity characteristics approximated to color matching functions obtained by linearly transforming them, or human color vision characteristics or A color signal obtained when having a spectral sensitivity characteristic obtained by linearly transforming it may be used as the target signal.

Embodiment 2. FIG.
In the first embodiment, when the processing target is a still image, for example, the processing after the front white balance means 15 can be realized by software, that is, by a programmed computer.

Embodiment 3 FIG.
The imaging device of the above embodiment can be applied to a video camera that captures moving images and still images, a camera-integrated VTR, a digital still camera, a PC camera, and a digital still camera built in a mobile phone or a mobile terminal. Therefore, the present invention can be applied to surveillance cameras and in-vehicle cameras that do not require IRCF and are often used for night vision.

  A configuration when applied to a digital still camera will be described below with reference to FIG. As shown in FIG. 14, the digital camera includes a color signal generating unit 61 instead of the color signal generating unit 2 among the elements constituting the imaging apparatus shown in FIG. Means 63, display drive means 64, monitor 65, image compression means 66, and writing means 67 are added.

  The shutter driving unit 63 drives the shutter in the color signal generating unit 61 according to the operation of the shutter button 62. The display driving means 64 receives the output of the video signal processing means 8 and displays an image on the monitor 65 as a viewfinder. The monitor 65 is composed of, for example, a liquid crystal display device, and is driven by the display driving unit 64 to display an image captured by the imaging unit in the color signal generation unit 61. The image compression unit 66 receives the output of the video signal processing unit 8 and performs image compression in accordance with, for example, JPEG. The writing unit 67 writes the data compressed by the image compression unit 66 to the recording medium 68.

  When the image pickup apparatus is used for moving image shooting and image data is transmitted to a device (not shown), the output of the video signal processing means 8 is encoded to generate and output an NTSC signal.

1 is a block diagram illustrating an imaging apparatus according to Embodiment 1 of the present invention. It is the figure which showed the arrangement | sequence of the color filter on the image pick-up element 22 in the imaging means 11 of the imaging device of FIG. It is a figure which shows arrangement | positioning of the image pick-up element 22, a color filter, and an optical system. It is a figure which shows the spectral sensitivity characteristic of the imaging means 11 using the color filters 26, 27, 28, and 29 of FIG. It is a block diagram which shows the structure of the front white balance means 15 of FIG. FIG. 2 is a block diagram showing a configuration of rear white balance means 16 in FIG. 1. It is a figure which shows the color matching function shown in CIE1931. It is a figure which shows the conventional image sensor, IRCF, and the spectral sensitivity characteristic multiplied. It is a figure which shows the spectral characteristic of the illumination in the case of 5000K in black body radiation. It is a figure which shows the spectral reflectance characteristic of each color patch of a Macbeth chart. It is a figure which shows the spectral reflection characteristic of "grass color (Foliage)", "Yellow green (Yellow green)", and "Green (Green)" mentioned as an example of green by the Macbeth color checker. It is a figure which shows the spectral reflection characteristic of the leaf of a tree naturally growing in nature. It is a figure which shows skin color ("light skin color" or brown ("dark skin color") spectral reflectance. 6 is a block diagram illustrating a configuration of a camera according to Embodiment 3. FIG.

Explanation of symbols

2 color signal generating means, 4 spectral sensitivity correcting means, 6 coefficient setting means, 8 video signal processing means, 11 imaging means, 12 amplifying means, 13 ADC, 14 DC reproducing means, 15 front white balance means, 16 rear white balance means , 17 Gamma correction means, 18 Luminance color difference signal generation means, 22 Imaging device, 23 Filter group, 26 Red color filter, 27 Green (G1) color filter, 28 Blue color filter, 29 Green (G2) color filter 31r, 31g1, 31g2, 31b amplifying means, 32r, 32g1, 32g2, 32b integrating means, 33 gain controlling means, 42 color identifying means, 43 coefficient determining means, 51r, 51g, 51b amplifying means, 52r, 52g, 52b integrating Means, 53 gain control means.

Claims (14)

The first, second, and third color filters that extract red, green, and blue light, and one of the first to third color filters has a correlation in the visible region, and has a maximum spectral transmittance. And a fourth color filter having a spectral transmittance which is shorter than the wavelength at which the spectral transmittance of the one color filter is maximum and which has substantially the same spectral transmittance as that of the one color filter in the near infrared region. Color signal generating means for outputting first to fourth color signals corresponding to the spectral transmittances of the respective color filters;
Spectral sensitivity correction means for generating a corrected color signal by performing a matrix operation of multiplying at least three color signals among the first to fourth color signals by a matrix coefficient;
In response to the first to fourth color signals output from the color signal generation means, it is determined whether or not the combination represents a predetermined specific exceptional color, and according to the determination result, An image pickup apparatus comprising: coefficient setting means for switching matrix coefficients.   The spectral sensitivity correction unit is configured such that a total spectral sensitivity characteristic from the color signal generation unit to the spectral sensitivity correction unit is a human color vision characteristic, or a spectral sensitivity characteristic obtained by linearly converting it, or CIE 1931 color matching The imaging apparatus according to claim 1, wherein the corrected color signal is generated by performing correction so as to approximate a spectral sensitivity characteristic obtained by converting a function or linearly the function. The first color filter is a color filter that allows red light to pass through;
The second color filter is a color filter that transmits green light;
The third color filter is a color filter that transmits blue light;
The fourth color filter has a correlation with the second color filter up to about 700 nm in spectral transmittance, and the wavelength at which the spectral transmittance is maximum is the second color filter. The imaging apparatus according to claim 1, wherein the imaging apparatus is shifted to a shorter wavelength side by about 50 nm and has substantially the same spectral transmittance as that of the second color filter when the wavelength exceeds about 700 nm. The spectral sensitivity correction unit is configured to use the following formula for the first to fourth color signals Rb, G1b, Bb, and G2b output from the color signal generation unit.

The imaging apparatus according to claim 1, wherein the corrected color signals Rc, Gc, and Bc are generated by performing an operation according to claim 1. The coefficient setting means includes
Expressed by the first to fourth color signals output from the color signal generation means by determining whether or not the first to fourth color signals output from the color signal generation means satisfy a predetermined condition. Color identification means for determining whether the color to be processed is the exceptional color;
The imaging apparatus according to claim 1, further comprising a coefficient determination unit that determines a matrix coefficient based on a determination result in the color identification unit. The color identification means includes
The ratio Rb / G1b of the third color signal Rb to the second color signal G1b is larger than a first predetermined value (k1), and the ratio G1b of the second color signal G1b to the fourth color signal G2b. When / G2b is greater than a second predetermined value (k2), it is determined that the color represented by the first to fourth color signals output from the color signal generation means is the exceptional color The imaging apparatus according to claim 5. When the first to fourth color signals from the color signal generating means represent the exceptional colors, matrix coefficients for the exceptional colors are used,
When the first to fourth color signals from the color signal generation means do not represent the exceptional colors, matrix coefficients for normal colors are used,
The matrix coefficient defined for the normal color is determined using a standard color chart,
The imaging apparatus according to claim 1, wherein the matrix coefficient for the exceptional color is determined in advance using the exceptional color. The first, second, and third color filters that extract red, green, and blue light, and one of the first to third color filters has a correlation in the visible region, and has a maximum spectral transmittance. And a fourth color filter having a spectral transmittance which is shorter than the wavelength at which the spectral transmittance of the one color filter is maximum and which has substantially the same spectral transmittance as that of the one color filter in the near infrared region. In the signal processing method of the imaging apparatus including color signal generation means for outputting the first to fourth color signals corresponding to the spectral transmittance of each color filter,
A spectral sensitivity correction step of generating a corrected color signal by performing a matrix operation of multiplying at least three color signals among the first to fourth color signals by a matrix coefficient;
In response to the first to fourth color signals output from the color signal generation means, it is determined whether or not a combination thereof represents a predetermined specific exceptional color. And a coefficient setting step for switching matrix coefficients.   In the spectral sensitivity correction step, a comprehensive spectral sensitivity characteristic from color signal generation in the color signal generation means to spectral sensitivity correction in the spectral sensitivity correction step is obtained by human color vision characteristics or by linearly converting it. The corrected color signal is generated by performing correction so as to approximate a spectral sensitivity characteristic or a CIE1931 color matching function or a spectral sensitivity characteristic obtained by linearly converting the same. 9. The signal processing method according to 8. The first color filter is a color filter that allows red light to pass through;
The second color filter is a color filter that transmits green light;
The third color filter is a color filter that transmits blue light;
The fourth color filter has a correlation with the second color filter up to about 700 nm in spectral transmittance, and the wavelength at which the spectral transmittance is maximum is the second color filter. 9. The signal processing method according to claim 8, wherein the signal processing method is shifted to a shorter wavelength side by about 50 nm, and has substantially the same spectral transmittance as that of the second color filter when the wavelength exceeds about 700 nm. In the spectral sensitivity correction step, the following equations are used for the first to fourth color signals Rb, G1b, Bb, and G2b output from the color signal generation unit.

The signal processing method according to claim 8, wherein the corrected color signals Rc, Gc, and Bc are generated by performing an operation according to the above. The coefficient setting step includes
Expressed by the first to fourth color signals output from the color signal generation means by determining whether or not the first to fourth color signals output from the color signal generation means satisfy a predetermined condition. A color identification step for determining whether the color to be processed is the exceptional color;
The signal processing method according to claim 8, further comprising: a coefficient determination step that determines a matrix coefficient based on a determination result in the color identification step. The color identification step includes
The ratio Rb / G1b of the third color signal Rb to the second color signal G1b is larger than a first predetermined value (k1), and the ratio G1b of the second color signal G1b to the fourth color signal G2b. When / G2b is greater than a second predetermined value (k2), it is determined that the color represented by the first to fourth color signals output from the color signal generation means is the exceptional color The signal processing method according to claim 12. When the first to fourth color signals from the color signal generating means represent the exceptional colors, matrix coefficients for the exceptional colors are used,
When the first to fourth color signals from the color signal generation means do not represent the exceptional colors, matrix coefficients for normal colors are used,
The matrix coefficient defined for the normal color is determined using a standard color chart,
The signal processing method according to claim 8, wherein the matrix coefficient for the exceptional color is predetermined using the exceptional color.

Images