JP2010161455A - Infrared combined imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an infrared combined imaging device capable of obtaining a color image having high color reproducibility. <P>SOLUTION: In imaging under infrared radiation imaging conditions, the infrared combined imaging device processes white balance per color of an imaging signal generated based on visible light imaging conditions in which an infrared cut filter is inserted into an optical path, and generates a white balance imaging signal. The infrared combined imaging device measures the visible light quantity per color for an imaging signal generated under the visible light imaging condition, and separates the imaging signal generated under the infrared imaging condition into a visible light component imaging signal and an infrared component imaging signal based on the visible light quantity per color. Then, the infrared combined imaging device matches color balance of the visible light component imaging signal based on the white balance imaging signal to generate a color image signal. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an infrared mixed imaging device that enables imaging of a subject under low illuminance by irradiating infrared rays.

  A technique relating to an imaging apparatus and method is disclosed in, for example, Patent Document 1 below. In Patent Document 1, before generating a luminance signal from an input pixel signal, an infrared light component mixed in the color signal of the pixel is estimated and calculated for each color filter, and the result of the color signal subjected to the estimation calculation process is calculated. A signal processing method for generating a luminance signal in response and generating an image signal using the generated luminance signal and color difference signal is described.

On the other hand, the following Patent Document 2 includes a detection unit that detects a position of a filter that changes the spectrum, and a white balance control unit that performs white balance control based on the position of the filter detected by the detection unit. A technique relating to an imaging apparatus characterized by this is disclosed.
JP 2007-88873 A JP 2005-130317 A

  In recent years, there has been a demand from the market for an imaging device that can obtain a color image having a color close to the appearance, not a monochrome image, even when an object is irradiated with infrared rays and imaged under low illuminance. Yes.

  However, infrared rays are not in the wavelength range of human visual characteristics. For example, it is possible to obtain a monochrome image by irradiating a subject with infrared rays and imaging the reflectance thereof. However, when a color image is obtained, it is necessary to further process color information that does not exist in infrared rays. is there.

  For example, as disclosed in Patent Document 1, there is a signal processing method for obtaining color reproducibility by estimating and calculating an infrared component mixed in an imaging signal for each color filter. This signal processing method corrects the color data so as to cancel the infrared component when the color temperature decreases and the infrared component increases in the evening, and k × (R + B + GR + GB) / 4 is calculated from each color data. Is a circuit that performs an operation of subtracting.

  However, the signal processing method disclosed in Patent Document 1 corrects an imaging signal containing a large amount of infrared components, such as imaging an object by irradiating the subject with infrared rays using an infrared irradiating instrument or the like, using the arithmetic circuit described above. There is a first problem that cannot be met.

  On the other hand, as shown in Patent Document 2, when an infrared cut filter is off the optical axis in imaging including an infrared component, black body curve data LB considering the infrared component is read. The black body curve control is performed. Alternatively, there is an imaging apparatus in which optimum white balance control is performed according to shooting conditions by performing gray world control such that the ratio of the integration values of the video signals R, G, and B is 1.

  However, the black body curve control based on the black body curve data LB in consideration of the former infrared component is performed by the arrangement of the infrared cut filter while the ratio and / or the difference between the visible light amount and the infrared light amount is unknown. Since the body curve data LA and LB are selected, the black body curve is not optimized, and thus the first problem described above is not solved.

  The latter gray world control is based on the premise that there is no color bias in the subject to be imaged. If there is a color bias in the subject, this white balance is greatly disrupted, resulting in good color reproduction. There is a second problem that it is not possible to obtain sex.

  Therefore, the present invention has been made in view of the above-described problems, and it is possible to obtain good color reproducibility when imaging by irradiating infrared rays on a subject under low illuminance. An object is to provide an infrared mixed imaging device.

  In addition, the present invention can obtain good color reproducibility even in the case where the subject under low illumination is irradiated with infrared rays and the subject has a color bias. It is an object of the present invention to provide an infrared mixed imaging device that is possible.

  In an infrared mixed imaging device, an infrared irradiation means for irradiating a subject with infrared rays, an optical system for forming an optical image by forming an image of the subject, and a structure capable of moving forward and backward with respect to the optical path of the optical system, An infrared cut filter for generating the optical image composed of visible light by blocking infrared contained in the visible light, and a visible light imaging condition for inserting the infrared cut filter on the optical path without irradiating the subject with infrared light, Imaging condition control means for controlling the infrared irradiation imaging condition for irradiating the subject with infrared rays and causing the infrared cut filter to exit the optical path, and imaging for photoelectrically converting the optical image and generating an imaging signal for each color , A timing generator capable of adjusting the exposure time of the imaging unit for the optical image, an imaging signal branch circuit for branching the imaging signal according to imaging conditions, and visible light imaging White balance processing means for processing the white balance for each color of the imaging signal generated in the process and generating the white balance imaging signal, and the visible light amount for each color with respect to the imaging signal generated under the visible light imaging condition And a visible light component separating unit that separates an imaging signal generated under an infrared irradiation imaging condition into a visible light component imaging signal and an infrared component imaging signal based on the visible light amount for each color. The white balance imaging signal is integrated for each color, the ratio of the integrated value is calculated and stored, and the color balance of the visible light component imaging signal is matched based on the visible light color ratio And a color balance matching unit that generates a color matching imaging signal, an image processing unit that generates a color image signal by performing image processing on the color matching imaging signal and the infrared component imaging signal.

  According to the present invention, there is an effect that good color reproducibility can be obtained even when imaging is performed by irradiating a subject under low illuminance with infrared rays.

  According to the present invention, when a subject under low illuminance is imaged by irradiating infrared rays, even if the subject has a color bias, an effect of obtaining good color reproducibility is obtained. There is.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a functional block diagram of an infrared mixed imaging device (hereinafter referred to as “imaging device”) according to an embodiment of the present invention. The imaging device is, for example, a digital still camera, a movie camera, a medical camera, or the like, and includes an infrared irradiation unit 1, an optical system including a lens group 3, an imaging unit 4, an infrared cut filter 5, a filter driving unit 6, and imaging condition control. Unit 7, branching unit 8, visible light amount measuring unit 9, white balance processing unit 10, separation unit 11, color balance matching unit 12, image processing unit 14, operation unit 15, and timing generator 16.

  Each configuration will be described. The infrared irradiation unit 1 includes an infrared light emitting diode that irradiates the subject 2 with infrared rays and a concave reflector. Infrared light emitted from the infrared light emitting diode is, for example, near infrared light whose main component is 800 to 900 nm.

  The optical system including the lens group 3 generates an optical image from the subject 2 and forms an image on the imaging surface of the imaging unit 4.

  The infrared cut filter 5 is a filter having a characteristic of transmitting a visible light component (350 to 700 nm) and attenuating or blocking the infrared component (700 nm or more). The infrared cut filter 5 moves between a position on the optical path of the optical system (hereinafter referred to as “blocking position”) A and a position off the optical path of the optical system (hereinafter referred to as “retraction position”) B. Configured to be able to. When the infrared cut filter 5 is at the blocking position A, the infrared cut filter 5 removes the infrared component from the optical image generated by the lens group 3 and causes only the visible light component to reach the imaging unit 4. When the infrared cut filter 5 is in the retracted position B, both the visible light component and the infrared component reach the imaging unit 4.

  The filter driving unit 6 includes an actuator such as a motor connected to the infrared cut filter 5. The filter driving unit 6 drives the actuator to move the infrared cut filter 5 to the blocking position A or the retracted position B.

  The imaging unit 4 is configured by an imaging element such as a CCD or a MOS sensor having sensitivity to both a visible light component and an infrared component. A color filter composed of a plurality of colors is provided in a regular arrangement on the front surface of the image sensor. Such a spectral sensitivity characteristic of the imaging unit 4 is as shown in FIG. 2, for example, and an integrated value of the spectral sensitivity characteristic of the color filter and the spectral sensitivity characteristic of the imaging element is a main component. When an optical image of the subject 2 is formed on the light receiving surface of the image sensor via the lens group 3, the image sensor photoelectrically converts the optical image for each color to generate an image signal. The imaging unit 4 outputs this imaging signal to the branching unit 8.

  The imaging condition control unit 7 controls the infrared irradiation unit 1, the filter driving unit 6, the branching unit 8, and the timing generator 16. When an operation for starting imaging is performed on the operation unit 15, the imaging condition control unit 7 receives this operation signal and starts control of each block.

  First, the infrared cut filter 5 is disposed at the blocking position A by the filter driving unit 6 and the infrared irradiation from the infrared irradiation unit 1 to the subject 2 is not performed, thereby realizing the visible light imaging condition. Then, the imaging condition control unit 7 instructs the timing generator 16 to perform imaging control, and the imaging unit 4 generates an imaging signal including a visible light component (hereinafter, this signal is referred to as a visible light imaging signal RGBxy). And output to the branching unit 8.

  Subsequently, the imaging condition control unit 7 places the infrared cut filter 5 at the retreat position B by the filter driving unit 6 and causes the infrared irradiation unit 1 to perform infrared irradiation from the infrared irradiation unit 1, thereby performing the infrared irradiation imaging condition. Is realized. Then, the imaging condition control unit 7 instructs the timing generator 16 to perform imaging control, and an imaging signal composed of both a visible light component and an infrared component (hereinafter, this signal is referred to as an infrared mixed imaging signal RGBAxy). ) Is generated by the imaging unit 4 and output to the branching unit 8.

  In addition, when the above imaging is performed, the imaging condition control unit 7 outputs information regarding whether the imaging is performed under the visible light imaging condition or the infrared irradiation imaging condition to the branching unit 8.

  The branching unit 8 is a circuit that branches the visible light imaging signal RGBxy or the infrared mixed imaging signal RGBAxy according to the imaging condition in accordance with the imaging condition control unit 7.

  The timing generator 16 is capable of adjusting the exposure time (electronic shutter) of the imaging unit 4 for the optical image. The timing generator 16 has a first exposure time adjusted with respect to the infrared irradiation imaging condition and a second exposure time adjusted with respect to the visible light imaging condition.

The first exposure time needs to be adjusted in consideration of the movement of the dynamic subject, and the image blur amount is adjusted by the operator adjusting the first exposure time to an appropriate time. . Further, the second exposure time need not be adjusted in consideration of the movement of the dynamic subject. This is because the purpose of the second exposure time is to adjust the exposure amount appropriate for the calculation of the visible light color ratio (ΣwR: ΣwG: ΣwB) described later.

  The imaging data based on the second exposure time for the purpose of calculating the visible light color ratio is not image data used as pixels. Therefore, the visible light imaging signal RGBxy generated by the second exposure time has no problem even if there is an image blur.

  For the reason described above, for example, the second exposure time is automatically adjusted to a long exposure based on the visible light amount, and the amount of visible light is increased to an appropriate amount, thereby increasing the accuracy of calculation of the ratio of visible light colors, Reduce the effects of noise. On the other hand, the first exposure time should be adjusted to an exposure time based on the operator's intention, and image blur due to movement of the dynamic subject and camera shake can be reduced.

The visible light quantity measurement unit 9 performs visible light quantity ΣR for each RGB color with respect to the visible light imaging signal RGBxy
, ΣG, ΣB. At this time, the visible light amount ΣRGB (ΣR1 + ΣG1 + Σ
B1) is output to the timing generator 16 described above.

  Further, the visible light amount measuring unit 9 measures the visible light amount for each RGB color by the second exposure time, and estimates the measured visible light amount to the visible light amount for each RGB color that will be obtained by the first exposure time. Then, the converted value is output to the separation unit described later.

The separation unit 11 separates the infrared mixed imaging signal RGBAxy into a visible light component imaging signal [RGBxy] and an infrared component imaging signal [Axy] based on the visible light amounts ΣR, ΣG, ΣB for each RGB color. It is.

  The white balance processing unit 10 performs white balance processing on the visible light imaging signal RGBxy to generate a white balance imaging signal wRGBxy, and outputs the generated white balance imaging signal wRGBxy to the color balance matching unit 12.

  As the white balance processing, a method that can correctly perform white balance even when the subject color is uneven is used. For example, 1) A high-level signal estimated to have a high reflectance of the subject is extracted from the visible light imaging signal RGBxy, and the high-level signal is subjected to black body discrimination to extract a provisional white signal. Then, the provisional white signal is converged on the locus of the black body. Or 2) automatically discriminating the type of light source or allowing the user to select the type of light source. Or, 3) image a white or gray (achromatic) subject, and converge the RGB ratio to 1 at this time. A white balance processing method such as the above is used.

  Note that the imaging condition control unit 7 described above performs control to automatically switch between the visible light imaging condition and the infrared irradiation imaging condition in time after the generation of the white balance imaging signal wRGBxy.

The color balance matching unit 12 obtains values Σ [R], Σ [G], and Σ [, obtained by integrating the signals [Rxy], [Gxy], and [Bxy] of each color constituting the visible light component imaging signal [RGBxy] on the xy plane. B]
Are calculated by the following equations.
Σ [R] = ∬ [Rxy] dxdy
Σ [G] = ∬ [Gxy] dxdy
Σ [B] = ∬ [Bxy] dxdy

Similarly, values ΣwR, ΣwG, and ΣwB obtained by integrating the signals wRxy, wGxy, and wBxy of the respective colors constituting the white balance image pickup signal wRGBxy by the following expressions are respectively calculated by the following equations, and integration of signals of the respective colors of the white balance image pickup signal wRGBxy is performed. Ratio of values (ΣwR: Σw
G: ΣwB, which is referred to as a visible light color ratio. ) Is calculated.
ΣwR = ∬ (wRxy) dxdy
ΣwG = ∬ (wGxy) dxdy
ΣwB = ∬ (wBxy) dxdy

Then, the color balance matching unit 12 has the following relational expression:
Kr × Σ [R]: Kg × Σ [G]: Kb × Σ [B] = ΣwR: ΣwG: ΣwB
The color balance coefficients (Kr, Kg, Kb) that satisfy the above are calculated.

Based on the calculated color balance coefficients (Kr, Kg, Kb), the color balance matching unit 12 performs signals c [Rxy] and c [Gxy] of each color constituting the visible light component imaging signal c [RGBxy] after color balance matching. ] And c [Bxy] are calculated by the following equations, respectively.
c [Rxy] = Kr × [Rxy]
c [Gxy] = Kg × [Gxy]
c [Bxy] = Kb × [Bxy]

That is, the color balance matching unit 12 matches the color balance of the visible light component imaging signal [RGBxy] based on the ratio of each visible light color (ΣwR: ΣwG: ΣwB) of the white balance imaging signal wRGBxy, and the color matching imaging signal c [RGBxy] is generated. The color balance matching unit 12 outputs the generated color matching imaging signal c [RGBxy] to the image processing unit 14.

  The visible light color ratio may be calculated after further integrating the integral value obtained by the xy plane integration described above over a plurality of frames. This is because integration by a plurality of frames reduces the effect of optical shot noise, and stability of the ratio of visible light color to a dynamic subject can be obtained.

  The image processing unit 14 performs image processing on the color matching imaging signal c [RGBxy] and the infrared component imaging signal [Axy] to generate a color image signal.

  For example, the first luminance signal Y1xy is generated by re-sampling from the color matching imaging signal c [RGBxy], and the infrared component imaging signal [Axy] is used as the second luminance signal Y2xy, and the luminance signals Y1xy and Y2xy are generated. Are combined to generate a luminance signal Yxy. In addition, the color matching imaging signal c [RGBxy] is resampled with respect to hue and saturation to generate a color signal Cxy. In this way, the color image signal (Y, C) xy is generated.

  For this resampling process, for example, bicubic calculation, bilinear calculation, nearest neighbor processing, matrix calculation with n taps, Y matrix calculation, color difference matrix calculation, and the like are used.

  The image processing unit 14 is not limited to the resampling process as described above, but various image processes such as luminance gradation correction, detail correction, various noise cancellation filters, color correction, and resolution conversion. Is given.

  FIG. 3 is a block diagram showing an example of adjusting the second exposure time using the configuration of the infrared mixed imaging device shown in FIG.

In FIG. 3, this block diagram shows an example of adjusting the second exposure time at which an appropriate amount of imaging signal level is obtained in order to calculate the visible light color ratio ΣwR: ΣwG: ΣwB. In FIG. 3, the path for adjusting the second exposure time is indicated by a bold line.

  As shown in FIG. 3, when the second exposure time is adjusted, the imaging condition control unit 7 controls the visible light imaging conditions. Specifically, the imaging condition operation unit 7 controls the infrared irradiation unit 1 so as not to irradiate infrared rays, inserts the infrared cut filter 5 into the optical path, and then performs standard exposure with the timing generator 16. The subject is imaged in time (for example, 1/60 s).

The visible light imaging signal RGBxy thus generated is input to the visible light amount measurement unit 9 and the visible light amount ΣRGB is measured. Here, in the case of imaging an object under low illuminance, this visible light amount Since ΣRGB is a small amount, the visible light imaging signal RGBx as it is
In y, there is a concern that the calculation accuracy of the ratio ΣwR: ΣwG: ΣwB for each visible light color is deteriorated.

Therefore, based on the measured visible light amount ΣRGB, the visible light color ratio Σw
The timing generator 16 needs to adjust the second exposure time which is an appropriate amount for calculating R: ΣwG: ΣwB with respect to the imaging unit 4. The appropriate amount is preferably such that a visible light amount corresponding to at least half of the maximum accumulated charge capacity of the photoelectric conversion element provided in the imaging unit 4 can be obtained.

  FIG. 4 is a block diagram illustrating an example of signal processing of the visible light imaging signal RGBxy generated at the second exposure time using the configuration of the infrared mixed imaging device shown in FIG.

In FIG. 4, this block diagram shows a path for measuring the visible light amounts ΣR, ΣG, and ΣB for each color of RGB and a path for calculating the visible light color ratio ΣwR: ΣwG: ΣwB by bold lines.

In FIG. 4, the visible light imaging signal RGBxy is an imaging signal generated during the second exposure time described above. Here, the measured visible light amount ΣR, ΣG, ΣB for each RGB color is the first
Is converted into a visible light amount that will be obtained in the exposure time (infrared irradiation imaging condition) and output to the separation unit 11.

Further, after white balance processing is performed on the visible light imaging signal RGBxy to generate a white balance imaging signal wRGBxy, the white balance imaging signal wRGBxy is integrated on the xy plane for each RGB color, and the visible light color ratio ΣwR: ΣwG: ΣwB is calculated.

  FIG. 5 is a block diagram showing an example of signal processing of the infrared mixed imaging signal RGBAxy using the configuration of the infrared mixed imaging device shown in FIG. In FIG. 5, a path for performing signal processing of the infrared mixed imaging signal RGBAxy is indicated by a bold line.

  In FIG. 5, this block diagram is a timing after the imaging condition operation unit 7 controls the infrared irradiation unit 1 to irradiate the subject with infrared rays, and the infrared cut filter 5 leaves the optical path. It is shown that the generator 16 performs control so that the subject is imaged at the first exposure time. This first exposure time is an arbitrary exposure time desired by the user.

In FIG. 5, the infrared mixed imaging signal RGBAxy thus generated is separated into a visible light component imaging signal [RGBxy] and an infrared component imaging signal [Axy] based on the visible light amounts ΣR, ΣG, and ΣB for each RGB color. Has been shown to be.

Further, the visible light component imaging signal [RGBxy] obtained in this way is a visible light color ratio ΣwR:
Based on ΣwG: ΣwB, it is shown that the color matching imaging signal c [RGBxy] is generated by matching with the color balance under the visible light imaging condition.

  Further, in the block diagram of FIG. 5, it is shown that the color matching image signal c [RGBxy] and the infrared component image signal [Axy] are subjected to image processing to generate a color image signal. . Here, since the color matching imaging signal c [RGBxy] and the infrared component imaging signal [Axy] have the same imaging timing, image blur due to image synthesis does not occur. The infrared component imaging signal [Axy] may act on the luminance component and may be an invalid signal for the color component.

  FIG. 6 is a graph showing an example of imaging signal levels for each color of RGB when a white subject under low illuminance is imaged under visible light imaging conditions.

  FIG. 6 is an example in which the level of the imaging signal for each RGB color is shown for a pixel in which a white subject is imaged when an object under low illuminance is imaged under visible light imaging conditions. Note that the imaging signal shown in FIG. 6 is based on the above-described first exposure time (exposure time under infrared irradiation imaging conditions desired by the user).

  Here, when the image pickup unit 4 has a single-plate color filter specification with a Bayer arrangement, etc., the plane phase for each of the RGB colors is not uniform. For example, resampling processing (not shown) is performed in advance. The plane phase for each of the RGB colors may be aligned. This resampling process may be, for example, general bicubic interpolation or bilinear interpolation.

  In the case where the resampling process is provided in the image processing unit 14 shown in FIG. 1, the plane phase for each of the RGB colors is shifted in units of pixels, but the signal processing is performed with the pixel shift being substantially coincident. There may be a way.

  FIG. 7 is a graph showing an example of image signal levels for each color of RGB when a white subject under low illuminance is imaged under infrared irradiation imaging conditions.

  FIG. 7 shows an example in which the level of the infrared mixed imaging signal RGBAxy for each of the RGB colors for the pixel on which the white subject is imaged when an object under low illuminance is imaged under infrared irradiation imaging conditions. It is.

  In FIG. 7, the hatched portion indicates the level of the imaging signal photoelectrically converted with the visible light component, and the white frame indicates the photoelectric conversion with the infrared component. The level of the captured image signal is shown. As shown in FIG. 7, the signal level of the imaging signal is improved by imaging by irradiating infrared rays, even when imaging a subject under low illuminance.

  FIG. 8 is a graph showing an example in which white balance processing by gray balance control is performed on the imaging signal of the subject under the infrared irradiation imaging condition shown in FIG.

  As shown in FIG. 8, when the white balance processing as shown by the broken line in FIG. 8 is performed on the infrared mixed imaging signal RGBAxy, if the attention is paid to the hatched portion in FIG. There is essentially no balance.

Further, the graph of FIG. 8 is shown only when the subject has no color deviation (ΣR: ΣG: ΣB≈1: 1: 1), which is ideal for gray balance control. There,
If the subject 2 has a color bias, the white balance is greatly lost (not shown).

  FIG. 9 is a graph showing an example in which the imaging signal based on the infrared irradiation imaging condition shown in FIG. 7 is separated into a visible light component imaging signal and an infrared component imaging signal.

  In FIG. 9, this graph shows that the infrared mixed imaging signal RGBAxy shown in FIG. 7 is separated into the visible light component imaging signal [RGBxy] and the infrared component imaging signal [Axy] by the separation unit 11. It is shown.

Further, the graph shown in FIG. 9 is based on the visible light amount ΣR, ΣG, ΣB for each RGB color,
A ratio for each of the RGB colors of the visible light imaging signal RGBxy and the infrared mixed imaging signal RGBAxy is calculated, and the visible light component imaging signal [RGBxy] and the infrared component imaging signal [Axy] are separated by this ratio. .

  FIG. 10 is a graph showing an example in which color balance matching is performed on the visible light component imaging signal shown in FIG.

  As shown in FIG. 10, when the color balance matching as shown by the broken line 2 in FIG. 10 is applied to the visible light component imaging signal [RGBxy], attention should be paid to the hatched portion in FIG. For example, the white balance for the visible light component is essentially applied.

  FIG. 11 is a graph showing another example in which the imaging signal based on the infrared irradiation imaging condition shown in FIG. 7 is differentially separated into a visible light component imaging signal and an infrared component imaging signal.

  11, this graph shows that the infrared mixed imaging signal RGBAxy shown in FIG. 7 is separated into the visible light component imaging signal [RGBxy] and the infrared component imaging signal [Axy] by the separation unit 11. It is shown.

In the graph shown in FIG. 11, uniform differences are calculated for each RGB color of the visible light imaging signal RGBxy and the infrared mixed imaging signal RGBAxy based on the visible light amounts ΣR, ΣG, ΣB for each RGB color, By this difference, the visible light component imaging signal [RGBxy] and the infrared component imaging signal [Axy] are separated.

  FIG. 12 is a graph showing an example in which color balance matching is performed on the visible light component imaging signal and the infrared component imaging signal shown in FIG.

  As shown in FIG. 12, this visible light component imaging signal [RGBxy] is a signal including an infrared component as an error according to the infrared irradiation imaging condition.

  Here, when the graph of FIG. 10 is compared with the graph of FIG. 12, it is shown that the separation accuracy is better when separated by the ratio of each RGB color as shown in FIG. This is a case where the infrared rays to be applied are uniform with respect to the subject.

  However, since the actual amount of infrared irradiation is inversely proportional to the square of the distance of the subject, separation by ratio may not necessarily be accurately separated as shown in the graph of FIG. Therefore, the ratio separation shown in FIG. 10 and the difference separation shown in FIG. 12 are both pros and cons, and their performance is not generally superior or inferior.

Supplementing the separation accuracy of such ratio separation and / or difference separation, in any case, there is an effect that the separation accuracy is improved by the color balance matching according to the present embodiment. For example, even when the visible light component imaging signal [RGBxy] is separated to result in an overall reddish signal than the actual subject, the visible light color Since color matching is performed based on the ratio ΣwR: ΣwG: ΣwB, this redness is corrected.

Moreover, since this visible light color ratio ΣwR: ΣwG: ΣwB is generated on the white balance under the visible light imaging condition, a good white balance is indirectly reflected.

  As described above, the infrared mixed imaging device according to the embodiment of the present invention can obtain good color reproducibility when imaging by irradiating infrared rays onto a subject under low illuminance. An infrared mixed imaging device can be provided.

  In addition, according to the present invention, when a subject under low illuminance is imaged by irradiating infrared rays, good color reproducibility can be obtained even if the subject has a color bias. It is possible to provide an infrared mixed imaging device that can be used.

  The embodiment of the present invention has been described in detail above with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like without departing from the gist of the present invention.

  Further, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, the problem described in the column of the problem to be solved by the invention can be solved, and the effect described in the column of the effect of the invention Can be obtained as an invention.

It is a block diagram which shows the structure of the infrared mixing imaging device by embodiment of this invention. It is the figure which showed an example of the spectral sensitivity characteristic of an imaging part. It is a block diagram which shows an example which adjusts 2nd exposure time using the structure of the infrared mixing imaging device shown by FIG. It is a block diagram which shows an example about the signal processing of the visible light imaging signal RGBxy produced | generated in 2nd exposure time using the structure of the infrared rays mixed imaging device shown by FIG. It is a block diagram which shows an example about the signal processing of the infrared mixed imaging signal RGBAxy produced | generated on infrared irradiation imaging conditions using the structure of the infrared mixed imaging device shown by FIG. It is a graph which shows an example of the image pick-up signal level for every RGB color at the time of imaging the white photographic subject under low illumination on visible light image pick-up conditions. It is a graph which shows an example of the imaging signal level for every RGB color at the time of imaging the white photographic subject under low illumination on infrared irradiation imaging conditions. It is a graph which shows an example which performed the white balance process by gray balance control to the imaging signal of the to-be-photographed object under the infrared irradiation imaging conditions shown in FIG. It is a graph which shows an example by which the imaging signal by the infrared irradiation imaging conditions shown by FIG. 7 is ratio-separated into a visible light component imaging signal and an infrared component imaging signal. 10 is a graph illustrating an example in which color balance matching is performed on the visible light component imaging signal and the infrared component imaging signal shown in FIG. 9. It is a graph which shows an example by which the imaging signal by the infrared irradiation imaging conditions shown in FIG. 7 is differentially separated into a visible light component imaging signal and an infrared component imaging signal. 12 is a graph showing an example in which color balance matching is performed on the visible light component imaging signal and the infrared component imaging signal shown in FIG. 11.

1 Infrared irradiation unit 2 Subject 3 Lens group (optical system)
DESCRIPTION OF SYMBOLS 4 Imaging part 5 Infrared cut filter 7 Imaging condition control part 9 Visible light quantity measurement part 10 White balance processing part 11 Separation part 12 Color balance matching part 14 Image processing part 15 Operation part 16 Timing generator

Claims (4)

An infrared irradiation unit that irradiates the subject with infrared rays;
An optical system that forms an optical image by imaging the subject;
An infrared cut filter for generating an optical image composed of visible light by blocking infrared rays included in the optical image;
Visible light imaging conditions in which the infrared cut filter is inserted into the optical path without irradiating the subject with the infrared ray, and the infrared cut filter is exited with respect to the optical path by irradiating the subject with the infrared ray. An imaging condition control unit for controlling each of the infrared irradiation imaging conditions;
An imaging unit that photoelectrically converts the optical image to generate an imaging signal for each color;
A timing generator capable of adjusting an exposure time of the imaging unit for the optical image;
A branching unit for branching the imaging signal according to the imaging condition;
A white balance processing unit that processes white balance for each color of the imaging signal generated under the visible light imaging condition and generates a white balance imaging signal;
A visible light amount measuring unit that measures a visible light amount for each color with respect to the imaging signal generated under the visible light imaging condition;
A separation unit that separates the imaging signal generated under the infrared irradiation imaging condition into a visible light component imaging signal and an infrared component imaging signal based on the visible light amount for each color;
The white balance imaging signal is integrated for each color, and a ratio for each visible light color, which is a ratio of the integrated values, is calculated and stored, and the color of the visible light component imaging signal based on the ratio for each visible light color A color balance matching unit that generates a color matching imaging signal by matching the balance;
An image processing unit that performs image processing on the color matching imaging signal and the infrared component imaging signal to generate a color image signal;
An infrared mixed imaging device comprising: The timing generator can adjust the exposure time as follows:
A first exposure time adjusted for the infrared irradiation imaging conditions;
A second exposure time adjusted for the visible light imaging conditions;
Have
The infrared mixed imaging apparatus according to claim 1, wherein the second exposure time is adjusted to be longer than the first exposure time. The first exposure time is adjusted by an operator;
The infrared mixed imaging apparatus according to claim 2, wherein the second exposure time is automatically adjusted based on the visible light amount. 4. The infrared mixed imaging device according to claim 3, wherein the second exposure time is adjusted such that the charge capacity of the visible light amount is at least half or more of the maximum accumulated charge capacity of the photoelectric conversion.

Images