JP2013150051A - Image pickup device, image processing device, and image processing method and program - Google Patents

Abstract

To provide a color ratio correcting means capable of reducing deterioration of color reproducibility due to incomplete transfer, particularly deterioration of color reproducibility with high sensitivity.
For each settable imaging sensitivity, a color ratio correction coefficient for each color generated using an output signal of a different color included in an imaging signal output from an imaging unit corresponding to a different exposure amount is stored. The correction unit that corrects the imaging signal output from the imaging unit using the stored color ratio correction coefficient is controlled, and the color ratio correction coefficient corresponding to the shooting sensitivity set in the shooting of the subject is stored. An image pickup apparatus that corrects an output signal of a different color included in an imaging signal of a subject acquired from the imaging means and output from the imaging means using a color ratio correction coefficient of a corresponding color among the acquired color ratio correction coefficients .
[Selection] Figure 1

Description

  The present invention relates to an imaging apparatus using an imaging element such as a CCD or a CMOS image sensor, and more particularly to an image processing apparatus and an image processing method for processing an imaging signal output from the imaging apparatus.

  Currently, imaging devices such as digital cameras for recording and reproducing still images and moving images captured using a solid-state imaging device such as a CCD or CMOS, using a memory card having a solid-state memory device as a recording medium, are widely available on the market. Yes.

  A solid-state image sensor basically converts and accumulates irradiated light into signal charges by a photodiode (PD) as a photoelectric conversion means in each unit pixel arranged in a matrix, and transfers it to a subsequent circuit. It is configured to do.

  In an imaging device, in order to accurately capture an image of a subject from high luminance to low luminance, when the incident light is converted into an electrical signal by the solid-state image sensor and output, the amount of signal output according to the incident light amount is incident. It is desirable that it is exactly proportional to the amount of light (good linearity). For this purpose, the signal charge accumulated in the PD needs to be completely transferred to the subsequent circuit.

  In order to improve transfer efficiency and realize complete charge transfer, the charge transfer path of the solid-state imaging device needs to have an ideally uniform structure. However, in reality, potential pockets and the like are generated on the transfer path due to slight errors such as crystal defects and contamination, and the size and concentration of each part generated during manufacturing. Transfer residue occurs, and transfer efficiency deteriorates.

  Note that since the amount of trapped charge is finite, it can be said that the smaller the charge transferred from the PD, the greater the degree of transfer efficiency deterioration due to charge trapping.

  That is, when a certain amount of charge is captured, if a sufficient amount of light is incident, a sufficient amount of charge is transferred with respect to the captured amount of charge. The proportional relationship is generally maintained, and a signal of a desired level is output. However, when the amount of incident light is small, the ratio of the transferred charge amount to the trapped charge amount is small, so that the output signal is lower than the desired level. Therefore, when the amount of incident light is large and small, the proportional relationship between the incident light and the output signal is broken (poor linearity).

  By obtaining for each sensitivity that can set a gain such that the signal output when a certain amount of light is incident becomes a predetermined signal level, and applying the gain according to the sensitivity uniformly to the imaging signal, A correction method for correcting the level of the output signal is generally performed. However, this method cannot correct the deterioration of linearity due to incomplete transfer, the degree of which varies depending on the amount of incident light.

  As a means for reducing the influence of incomplete transfer as described above by signal processing, a correction that compensates for the lack of signal level due to incomplete transfer by adding a uniform offset signal to the image signal. Means have been proposed (see, for example, Patent Document 1).

Japanese Patent No. 046778824

  In recent years, solid-state imaging devices used in imaging devices such as digital cameras have been increasing in the number of pixels. Therefore, although the area of the imaging surface does not change, the area per pixel is reduced by the increase in the number of pixels, and accordingly, the amount of charge that can be captured per pixel is inevitably reduced. Yes.

  Correspondingly, an increase in sensitivity of an imaging apparatus capable of capturing an image with an appropriate exposure even when the amount of exposure is small in a low-brightness environment is progressing. In other words, an amplification unit is provided after the pixel unit, and the image signal is set to an appropriate level by amplification even with a small amount of charge. In the case of an imaging apparatus capable of setting sensitivity from low sensitivity to high sensitivity, the exposure amount of the image sensor is reduced by the sensitivity ratio when the high sensitivity is set compared to the equivalent exposure shooting when the low sensitivity is set. For example, when shooting with the same exposure, the sensitivity setting equivalent to ISO 51200 reduces the exposure amount by 1/512 compared to the sensitivity setting equivalent to ISO 100, and the amount of charge accumulated in each PD is also reduced accordingly. End up.

Furthermore, the image sensor is provided with a color filter for color separation for each unit pixel. Depending on the color (wavelength) of the color filter, the ratio of the amount of transmitted light to the amount of incident light varies, so even for pixels with the same exposure amount, depending on the subject light source color and the color filter color assigned to each pixel The amount of light reaching the PD will be different.
Further, even when the amount of light that passes through the color filter and enters the pixel surface is the same, there is a difference in photoelectric conversion rate depending on the wavelength of the light that enters the PD, so each color of the color filter (RGB, etc.) There is a difference in the amount of charge accumulated in the PD depending on the wavelength of light that passes through.

  As described above, the smaller the charge transferred from the PD, the greater the influence of incomplete charge transfer. In particular, when a low-brightness subject is photographed with a high-sensitivity setting using an imaging device that uses a solid-state imaging device with a large number of pixels, the effect of residual charge transfer appears significantly, and a sufficient exposure amount can be obtained for each color ratio. It will be very different compared to the case.

  Such deterioration of the color ratio results in poor color reproduction of the image pickup device, which degrades the image quality of the image acquired by the image pickup device.

  In the prior art described in Patent Document 1, means for adding a uniform offset signal to optical signals obtained from all pixels has been proposed. However, even if such correction is performed, the change in the color ratio that occurs depending on the exposure amount cannot be corrected appropriately, resulting in an image with poor color reproduction of the captured image. In addition, since a uniform offset signal is added, there is a problem that the offset level is floated even in a low luminance portion, particularly in a portion where there is no optical signal and the image signal is completely at the black level, and the black level cannot be reproduced. .

  The present invention has been made in view of the above-described problems, and provides a correction means that can reduce deterioration of color reproducibility due to incomplete transfer, in particular, deterioration of color reproducibility even when shooting at a high sensitivity setting. With the goal.

  In order to achieve the above object, according to the present invention, there are provided sensitivity setting means for setting shooting sensitivity and imaging means for performing photoelectric conversion at the shooting sensitivity set by the sensitivity setting means and outputting an imaging signal of a subject. The imaging apparatus corrects the color ratio for each color generated using output signals of different colors included in the imaging signal output from the imaging unit corresponding to different exposure amounts for each imaging sensitivity set by the sensitivity setting unit. The coefficient is stored in the storage means, the correction means for correcting the imaging signal output from the imaging means using the color ratio correction coefficient stored in the storage means, and the photographing sensitivity set by the sensitivity setting means in photographing the subject The corresponding color ratio correction coefficient is acquired from the storage means, the correction means is controlled, and an output signal of a different color included in the imaging signal of the subject output from the imaging means is acquired. Of, and a control means for correcting using the color ratio correction coefficient of the corresponding color.

A solid-state imaging device having photoelectric conversion means for converting optical signals having different wavelengths into electrical signals for each color and outputting the electrical signals;
Sensitivity setting means for switching a plurality of sensitivity settings;
Correction means for correcting an output signal output from the solid-state imaging device;
In an imaging system comprising:
For each of the plurality of sensitivities that can be set by the sensitivity setting means, an output signal from the solid-state imaging device is acquired with a plurality of exposures,
A correction coefficient for each color for correcting the ratio of the output signal for each color based on the output signal for each color obtained from the output signals acquired at the plurality of exposures is obtained and stored for each sensitivity. Leave
The output signal output from the solid-state imaging device is corrected using the correction coefficient for each color according to the sensitivity set by the sensitivity setting means.

  According to the present invention, it is possible to provide an image processing apparatus that can reduce deterioration of color reproducibility due to incomplete transfer or the like, particularly deterioration of color reproducibility in high-sensitivity shooting.

1 is a block diagram illustrating a configuration example of an imaging apparatus according to an embodiment of the present invention. 1 is an equivalent circuit diagram illustrating a configuration example of a solid-state imaging device according to an embodiment of the present invention. The figure which shows the timing chart which shows an example of the control pulse of the solid-state image sensor shown in FIG. Schematic diagram showing a cross section of the charge transfer path of the solid-state imaging device shown in FIG. The figure which shows an example of the input-output characteristic of the solid-state image sensor concerning embodiment of this invention The figure which shows the relationship of the output of a red (R) pixel and a blue (B) pixel with respect to the output of a green (G) pixel about the output of each sensitivity shown in FIG. The figure which shows the relationship of the output of a red (R) pixel and a blue (B) pixel with respect to the output of a green (G) pixel about the output of each sensitivity shown in FIG. The figure which shows the relationship of the output of a red (R) pixel and a blue (B) pixel with respect to the output of a green (G) pixel about the output of each sensitivity shown in FIG. The figure which shows a signal output when the correction | amendment of the image pick-up signal concerning 1st Example of this invention is applied to the signal shown to FIG. 6A and 6B. The figure which shows a signal output when the correction | amendment of the image pick-up signal concerning 1st Example of this invention is applied to the signal shown to FIG. 6A and 6B. The figure which shows a signal output when the correction | amendment of the image pick-up signal concerning 1st Example of this invention is applied to the signal shown to FIG. 6A and 6B. Schematic diagram of signal output for explaining in detail the correction of the imaging signal according to the first embodiment of the present invention. The figure which shows the flowchart of the color ratio correction | amendment process concerning 1st Example of this invention. The figure which shows the output signal by the signal correction concerning 2nd Example of this invention The figure which shows the flowchart of the modification of the color ratio correction process concerning 2nd Example of this invention. The figure which shows the flowchart of the modification of the color ratio correction process concerning 2nd Example of this invention. The figure which shows the output signal by the signal correction concerning the 3rd Example of this invention The figure which shows the flowchart of the color ratio correction process concerning the 3rd Example of this invention. Schematic diagram showing an example of arrangement of color filters of a solid-state imaging device according to an embodiment of the present invention

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The present embodiment is an example in which the present invention is applied to an imaging apparatus capable of acquiring an imaging signal using a solid-state imaging device.

  First, configuration examples of the imaging apparatus and the solid-state imaging device according to the present embodiment will be described. Next, each embodiment when the present invention is applied to the imaging apparatus will be described.

<Schematic configuration of imaging device>
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus as an imaging system in an embodiment of the present invention.

Reference numeral 1 denotes an optical system for guiding a light beam from a subject to the solid-state imaging device 3, which is composed of an optical member such as a lens and a diaphragm.
Reference numeral 2 denotes a mechanical shutter that shields light from the solid-state imaging device in order to control the exposure time of the solid-state imaging device described later.
Reference numeral 3 denotes a solid-state imaging device that converts irradiated light into an electrical signal and outputs the electrical signal. Details of the solid-state imaging device according to the embodiment of the present invention will be described later.
Reference numeral 4 denotes an A / D converter that converts an electric signal (an imaging signal that is an analog signal) output from the solid-state imaging device 3 into a digital image signal.
Reference numeral 5 denotes a timing signal generation circuit that generates a signal necessary for the drive circuit 6 in order to operate the solid-state imaging device 3.

Reference numeral 6 denotes a drive circuit for the optical system 1, the mechanical shutter 2, and the solid-state image sensor 3.
A signal processing circuit 7 performs signal processing such as various corrections necessary for the imaging signal output from the solid-state imaging device 3. In this embodiment, it functions as a correction means for performing signal correction according to the present invention.
Reference numeral 8 denotes an image memory for storing image-processed image data.
Reference numeral 9 denotes an image recording medium that is removable from the imaging apparatus and stores image data.
Reference numeral 10 denotes a recording circuit for recording the signal-processed image data on the image recording medium 9.

Reference numeral 11 denotes a display device that displays image data subjected to signal processing.
A display circuit 12 displays an image or the like on the display device 11.
Reference numeral 13 denotes a system control unit that controls the entire imaging apparatus.
Reference numeral 14 denotes a nonvolatile memory (ROM) as a storage means. The nonvolatile memory 14 includes a program describing a control method executed by the system control unit 13, control data such as parameters and tables used when executing the program, and data used for various corrections of the image signal. Remember.
Reference numeral 15 denotes a volatile memory (RAM). The volatile memory 15 transfers and stores the program, control data, and correction data stored in the non-volatile memory 14, and is loaded and used when the system control unit 13 controls the imaging apparatus.

Reference numeral 16 denotes a switch Sw0 that controls the power on / off state of the imaging apparatus.
Reference numeral 17 denotes a switch Sw1 that instructs the start of various shooting preparation operations for the imaging device to perform shooting operations.
Reference numeral 18 denotes a switch Sw2 for instructing the start of the photographing operation of the imaging apparatus.
Reference numeral 19 denotes a sensitivity setting unit as sensitivity setting means for instructing the setting of imaging sensitivity of the imaging apparatus.

  Note that the above-described configuration of the imaging device is merely an example of a configuration necessary for carrying out the present invention. The present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist.

<Example of imaging operation of imaging apparatus>
Prior to the photographing operation, when the operation of the system control unit 13 is started, such as when the imaging apparatus is turned on, the necessary program, control data, and correction data are volatilized from the nonvolatile memory 14 according to the setting of the sensitivity setting unit 19. Is transferred to and stored in the memory 15. These programs and data are used when the system control unit 13 controls the imaging apparatus. If necessary, additional programs and data are transferred from the nonvolatile memory 14 to the volatile memory 15, or the system control unit 13 directly reads out and uses the data in the nonvolatile memory 14.

  First, the aperture and lens of the optical system 1 are driven by a control signal from the system control unit 13, and a subject image set to an appropriate brightness is formed on the imaging surface.

  Subsequently, the solid-state imaging device 3 is driven and a photographing operation is performed. The solid-state image pickup device 3 is driven by a drive pulse based on an operation pulse generated by the timing signal generation circuit 5 controlled by the system control unit 13, and photoelectrically converts the subject image according to the accumulated charge amount. Outputs electrical signals. During this time, the mechanical shutter 2 opens and closes to control the exposure time of the solid-state imaging device 3. The solid-state imaging device 3 is controlled to accumulate generated charges for a desired accumulation time and output an electrical signal corresponding to the accumulated charge amount as an analog signal.

  The analog signal output from the solid-state imaging device 3 is converted into a digital signal by the A / D converter 4 by an operation pulse generated by the timing signal generation circuit 5 controlled by the system control unit 13.

  Next, in the signal processing circuit 7, various corrections including sensitivity correction and color ratio correction described later, image processing such as color conversion, white balance, and gamma correction, resolution conversion processing, image compression processing, etc. Is done. The image memory 8 in the signal processing circuit 7 is used for temporarily storing a digital image signal being subjected to signal processing and storing image data which is a digital image signal subjected to signal processing.

  The image data signal-processed by the signal processing circuit 7 and the image data stored in the image memory 8 are converted into data suitable for the image storage medium 9 (for example, file system data having a hierarchical structure) by the storage circuit 10. And stored in the image storage medium 9. The image data converted into the digital image signal by the A / D converter 4 is subjected to resolution conversion processing by the signal processing circuit 7 and then converted into a signal suitable for the image display device 11 by the display circuit 12. It is displayed on the image display device 11. The storage circuit 10 outputs information such as the type and free capacity of the image storage medium 9 to the system control unit 13 in response to a request from the system control unit 13.

<Configuration of solid-state image sensor>
FIG. 2 is an equivalent circuit diagram of a CMOS image sensor as an example of the solid-state imaging device 3 of FIG.

  The unit pixel 200 is a photodiode (PD) as a photoelectric conversion unit that converts incident light into electric charge and accumulates the generated signal charge, and an amplification unit that outputs an amplified signal output according to the accumulated signal charge amount. An amplifying MOS transistor (MAmp) is included. Also included is a floating diffusion portion (FD) that receives the signal charge and connects to the gate electrode of the amplification MOS transistor, and a transfer MOS transistor (MTx) as transfer means for transferring the signal charge accumulated in the PD to the FD portion. Further, a reset MOS transistor (MRes) as reset means for resetting the FD section and a selection MOS transistor (MSel) as selection means for selecting a pixel to output a signal are included. The unit pixel is configured as described above. A power supply is connected to the drain of the amplification MOS MAMP and the drain of the reset MOS MRes via a metal wiring, and a power supply potential is supplied.

  A common output line V is provided for outputting an amplification signal of a selected pixel to a group of pixels arranged in the same column among a plurality of pixels arranged in a matrix. In addition, control lines ΦTx, ΦRes, and ΦSel are provided for applying control pulses PTx, PRes, and PSel to the gate electrodes of MTx, MRes, and MSel for a plurality of pixels arranged in the same row. . Control lines ΦTx, ΦRes, and ΦSel are connected to the vertical scanning circuit VSR.

  The output line V is connected to the constant current source I and is connected to the inverting input terminal of the operational amplifier VAmp via the clamp capacitor C0.

  The non-inverting input terminal of the operational amplifier VAmp is connected to the clamp voltage VC0R (VREF). Capacitors Ch and Cl are connected between the inverting input terminal and the output terminal of the operational amplifier VAmp, and a signal voltage is supplied to the gate of the MGain corresponding to the sensitivity set in the sensitivity setting unit 19. The gain in the operational amplifier is controlled. For example, when the sensitivity setting unit 19 is set to low sensitivity, the signal is read with the first gain by controlling the MGain to be on and connecting the capacitor Cl. In addition, when the sensitivity is set to high, MGain is controlled to be in an off state, and only the capacitor Ch is used, so that the signal is read with a second gain higher than the first gain. Although two capacitors are shown in FIG. 2, at least two or more capacitors may be provided and each may be controlled with a control pulse so that the gain can be set more finely.

  The output terminal of the operational amplifier VAmp is connected to a capacitor CTn for temporarily holding a noise signal (imaging signal reference signal) via the noise signal transfer switch MTn, and to an optical signal (imaging signal) via the optical signal transfer switch MTs. ) Is temporarily connected to the capacitor CTs. The opposite terminals of the noise signal holding capacitor CTn and the optical signal holding capacitor CTs are grounded. A connection point between the noise signal transfer switch MTn and the noise signal holding capacitor CTn is connected to a differential circuit DAmp that outputs a difference signal between the optical signal and the noise signal via the horizontal transfer switch MHn. The connection point between the optical signal transfer switch MTs and the optical signal holding capacitor CTs is connected to the differential circuit DAmp via the horizontal transfer switch MHs.

  The gates of the noise signal transfer switches MTn in each column are commonly connected to a control line φTn for applying the noise signal transfer pulse PTn. The gates of the optical signal transfer switches MTs in each column are commonly connected to a control line φTs for applying the optical signal transfer pulse PTs.

  The horizontal transfer switches MHn and MHs are controlled by a column selection pulse PH supplied from the horizontal scanning circuit HSR.

  Next, the operation of the solid-state imaging device shown in FIG. 2 will be briefly described with reference to FIG.

  FIG. 3 shows a timing chart of each control pulse for operating the solid-state imaging device shown in FIG.

  Reading of signals starts from the 0th row and is performed in order of the 1st row, the 2nd row, the 3rd row, and so on. After the signal read operation for the corresponding row is completed, the next row read operation is started.

  Hereinafter, the reading operation of the nth row of the image sensor shown in FIG. 2 will be described.

  When PSel becomes low level at time T0, the selection MOS MSel in the row that was read immediately before, here, the n-1th row selection MOS MSel is turned off, and the selection of the pixel in the n-1th row is released.

  At the same time, PC0R becomes high level, and reset of the clamp capacitor C0 of each column is started. Further, PTs and PTn become high level, and resetting of the noise signal holding capacitors CTn and the optical signal holding capacitors CTs in each column is started.

When a line feed pulse PV (not shown) is input to the vertical scanning circuit at time T1, the line feed of the selected line to be read out is performed. Here, the selected row is sent from the (n−1) th row to the nth pixel row.
When PRes becomes low level at time T2, the reset of the FD is released and the reference potential of the FD is determined.
At time T3, the row feed pulse PV becomes high level, PSe1 becomes high level, the selection MOS MSe1 is turned on, and the pixel in the nth row is selected.
At time T4, PTs and PTn become low level, the resetting of the optical signal holding capacitor CTs and the noise signal holding capacitor CTn is completed, and the reference potentials of CTs and CTn are determined.
At time T5, PC0R becomes low level, and the reference potential VC0R is held in the C0 capacitor.

At time T6, PTn becomes high level, and the potential of FD is output to the noise signal holding capacitor CTn.
At time T7, PTn becomes low level, and the potential of the FD at this time is held in the noise signal holding capacitor CTn as a noise signal (reference signal of the imaging signal).
At time T8, PTs becomes high level, and the potential of FD is output to the noise signal holding capacitor CTs.
While PTs is at high level at time T9, PTx is at high level, the transfer MOS M1 is turned on, and the charge stored in the PD is transferred to the FD.
At time T10, PTx becomes low level, the transfer MOS M1 is turned off, and the transfer of charges to the FD ends.

At time T11, PTs becomes low level, and the potential of the FD at this time is held in the optical signal holding capacitor CTs as an optical signal (imaging signal).
At time T12, PRes becomes high level, and FD reset is started.

  When the input of a column feed pulse PH (not shown) is started at time T13, column transfer pulses are sequentially input to ΦH from the first column to the last column in the reading area. Therefore, the signals held in the optical signal holding capacitor CTs and the noise signal holding capacitor CTn are sequentially sent to the differential circuit DAmp for each column, and the differential amplification signal of the optical signal and the noise signal is used as an image signal. Output from the output terminal of the image sensor. For example, when the color filter shown in FIG. 14 is provided in the image sensor, the R, G, and B color component signals are sequentially output according to the color arrangement of the color filter.

  The above is the signal readout operation of the nth pixel row, and after the readout of the pixel signal of the nth row is completed, the operation moves to the readout operation of the (n + 1) th row. For the reading of the pixel rows other than the n-th row, the same reading operation as that of the n-th row is repeated, and when the signal reading of each row is completed, the reading of the next row is started.

≪Pixel structure≫
Next, FIG. 4 shows a cross-sectional structure of the PD, MTx, and FD portions per unit pixel. In this figure, 401 is an N-type semiconductor substrate, 402 is a P-type well, 403 is an N-type semiconductor region formed in the well 402, and a photodiode PD is formed by the well 402 and the region 403. Signal charges are accumulated in 1103.

  Reference numeral 404 denotes an N-type semiconductor region formed in the well 402 and serves as an FD portion. Reference numeral 405 denotes an MTx gate electrode. Regions 403 and 404 are an MTx source region and a drain region, respectively. A wiring 406 is connected to the region 404 and is connected to the gate of MAmp. The signal charges accumulated in the region 403 are all transferred to the N-type semiconductor region 404 as the FD portion during the transfer operation, and the N-type impurity concentration in the region 403 is set so that the region 403 immediately after the transfer is completely depleted. Has been. Therefore, the signal charge accumulation in the PD starts immediately after the signal charge transfer, and when the signal charge accumulation ends, the row to which the pixel belongs is selected again and the signal charge accumulated in the PD region 403 is stored in the region 404. It is time to be transferred.

≪Incomplete transfer≫
In order to improve transfer efficiency and realize complete charge transfer, the charge transfer path of the solid-state imaging device needs to have an ideally uniform structure. However, in reality, potential pockets and the like are generated on the transfer path due to slight errors such as crystal defects and contamination, the size of each part and the impurity concentration generated during manufacturing, and several levels of electrons are captured and charged. Transfer residue occurs, and transfer efficiency deteriorates.

  Since the amount of trapped electric charge is finite, the smaller the charge transferred from the PD, the greater the influence of incomplete charge transfer.

  That is, when a certain level of charge is trapped in the transfer path, if a sufficient amount of light is incident, a sufficient amount of charge is transferred relative to the trapped charge amount. The proportional relationship with the signal is generally maintained, and a signal of a desired level is output. That is, the ratio of the signal level due to charge trapping in the transfer path is small. However, when the amount of incident light is small, the ratio of the transferred charge amount to the trapped charge amount is small, so that the output signal is lower than the desired level. For this reason, the same proportional relationship between the incident light and the output signal cannot be maintained when the amount of incident light is large and small. (The linearity is bad.)

  Therefore, when the high sensitivity setting is used to capture a small amount of exposure and compensate for the small amount of accumulated charge with the subsequent gain such as the circuit gain, it is due to the remaining charge transfer compared to the low sensitivity setting where sufficient exposure can be obtained. The influence on the image signal is large.

Further, as described above, the image sensor is provided with a color filter for color separation for each unit pixel, and the ratio of the transmitted light amount to the incident light amount varies depending on the color of the color filter. For this reason, even if the pixels have the same exposure amount, the amount of light reaching the PD differs depending on the subject light source color and the color filter assigned to each pixel.
In addition, since the PD has a difference in photoelectric conversion rate depending on the wavelength of incident light, the amount of charge accumulated in the PD varies depending on the wavelength of light transmitted through each color (RGB, etc.) of the color filter. .

  Therefore, the degree of influence on the image signal due to the remaining charge transfer differs for each color of the color filter provided in each pixel. In addition, since the degree of influence on the image signal due to the residual charge transfer differs for each color, the color ratio of each color component signal of the obtained image signal differs depending on the exposure amount, and the color reproducibility of the image decreases. End up. Regarding this problem as well, when shooting with a small exposure amount and using a high sensitivity setting that compensates for the amount of accumulated charge with a subsequent gain such as a circuit gain, compared to a low sensitivity setting that provides a sufficient exposure amount. It can be said that the influence on the image signal due to the remaining charge transfer is large.

  The above problem will be specifically described with reference to FIGS. 5 and 6A-6B.

  FIG. 5 is a logarithmic graph showing an example of input-output characteristics in the solid-state imaging device.

  FIG. 5A shows the input-output characteristic at the time of low sensitivity setting (for example, ISO 100), and FIG. 5B shows the input-output characteristic at the time of high sensitivity setting (for example, ISO 51200). In addition, ideal input-output characteristics are shown as theoretical values in each graph.

  Here, during proper exposure, the signals of all colors are multiplied by a gain such that the G output becomes a uniform value regardless of ISO. Specifically, for example, the capacitors Ch and Cl inside the solid-state imaging device are controlled in accordance with the sensitivity setting so that the G output at the appropriate exposure is in the vicinity of a uniform value (here, 1000 counts).

  In addition, the sensitivity correction for the G signal is performed so that the G output at the appropriate exposure becomes a uniform value (1000 counts here) at each set sensitivity. In this correction, among the gains adjusted in advance for each sensitivity, the gain is applied to the G signal by a signal processing circuit outside the solid-state imaging device according to the set sensitivity.

  Also, by multiplying the R and B signals by a gain such that the R and B outputs at the appropriate exposure at the high sensitivity setting are equal to the R and B outputs at the appropriate exposure at the low sensitivity setting, respectively. Sensitivity correction for the R and B signals is performed. Specifically, the gain adjusted in advance for each color (R, B) so that the R output and B output at the proper exposure at ISO 51200 are equal to the R output and B output at the proper exposure at ISO 100, The signal processing circuit outside the solid-state imaging device is used.

  Therefore, the input-output relationship (see FIG. 5B) at the time of high sensitivity setting crosses the theoretical value straight line under the exposure conditions in the vicinity of the proper exposure combined by the sensitivity correction.

  FIG. 5 shows that the linearity characteristic is better as the plot of the output with respect to the input is closer to the straight line of the theoretical value, and the worse the linearity characteristic is, the further away from the straight line of the theoretical value. In addition, the lower the line is from the theoretical value, the greater the decrease in output.

  In the low sensitivity setting at which a sufficient exposure amount can be obtained, as shown in FIG. 5A, the output is not reduced, and a straight line is drawn almost as the theoretical value. However, in the high sensitivity setting with a small exposure amount, as shown in FIG. 5B, it can be seen that the output decreases as the luminance (low exposure) side decreases.

  In the example of FIG. 5, the charge amount of the G pixel is the largest, and the charge amounts of the R and B pixels are smaller.

  As described above, the influence of the incomplete transfer of the charge is increased in the portion where the charge amount is small, and therefore, the lower the charge of the color pixel, the greater the decrease in the output with respect to the input.

  Therefore, the output of the R and B pixels, which have a smaller amount of charge compared to the G pixel, is greatly reduced.

  6 (A), 6 (B), and 6 (C) show red (R) pixels with respect to the output of green (G) pixels for the outputs of the respective sensitivities shown in FIGS. 5 (A) and 5 (B). The blue (B) pixel output relationship is plotted on a logarithmic scale. FIG. 6 (A) shows a plot from the appropriate exposure -2 steps to the +1 step, FIG. 6 (B) enlarges the appropriate exposure-2 steps to the appropriate exposure, and FIG. 6 (C) enlarges the +1 step from the appropriate exposure. Showed. Again, ideal R and B outputs for G output are shown as theoretical values in each graph. Here, the output ratio of the R and B pixels to the G pixel (that is, the color ratios R / G and B / G) decreases as the plot of the R and B outputs against the G output falls below the straight line of the theoretical value. Is shown.

  At the time of low sensitivity setting, as shown in FIG. 5 (A), the relationship between input and output is maintained for each color. Therefore, as shown in FIGS. The R and B outputs have a relationship that is almost as theoretical values. On the other hand, at the time of high sensitivity setting, as shown in FIG. 5B, as the exposure decreases, the output of the R and B pixels having a small amount of charge is greatly reduced compared to the G output. Accordingly, as shown in FIGS. 6A to 6C, the lower the exposure, the lower the G output from the output level (G = 1000 counts) at the time of proper exposure. The output ratio (R / G, B / G) of the pixel decreases.

  In this embodiment, the correction method for each color of the output of each pixel is described by taking as an example the relationship of the output of other color pixels to the G pixel output at the time of high sensitivity setting shown in FIGS. 6 (A) -6 (C). A method for deriving a correction value used for the correction will be described.

In this embodiment, as a simple correction method for each pixel output, color ratio correction is performed by adjusting the color ratios R / G and B / G to target values using gain and offset correction coefficients.
Specifically, for each sensitivity set by the sensitivity setting unit 19 from an image acquired under a typical light source close to the situation in which the imaging apparatus is used (for example, a white light source having a spectral distribution close to natural light). In addition, a color ratio correction coefficient including a gain and an offset is obtained for each pixel color. This color ratio correction coefficient is stored in advance in the nonvolatile memory (ROM) 14 of the image pickup apparatus, and the correction coefficient corresponding to the sensitivity set in the image pickup apparatus by the sensitivity setting unit 19 is used as an image signal in the signal processing circuit 7. It shall be used for correction.

Here, when an image is generated, color ratio correction is not performed for pixels to which a color that contributes the most to the luminance component is assigned (that is, the gain is fixed to 1 and the offset is fixed to 0). . In the solid-state image pickup device constituted by the RGB filter given as an example, it is a G pixel. As another example, a W pixel corresponds to a solid-state imaging device using RGBW four-color filters. This is because if the offset component is added to the output signal of the color that greatly contributes to the luminance component,
Without black level = standard level,
This is because black level = reference level + offset and luminance reproducibility in the vicinity of the black level is lowered, that is, there is a problem that the black part does not appear black and the dark part is floated. . Therefore, for the G output in the present embodiment, which is the color that contributes the most to the luminance component, the color ratio correction is performed only for the other colors (R and B outputs) without performing the color ratio correction. .

  The detailed correction procedure will be described after describing the method for deriving the correction coefficient.

  Hereinafter, a method for deriving a correction coefficient used for signal correction will be described using the B output as an example.

  In the graphs of FIGS. 7A-7C, the horizontal axis represents the G pixel output (logarithm) and the vertical axis represents the B pixel output (logarithm), and the B pixel before correction with respect to the G pixel output at the time of high sensitivity setting. The pixel output (white point) and the B pixel output (black point) after the correction described in this example were plotted. The ideal relationship between G output and B output is shown as a theoretical value. FIG. 7 (A) shows a plot from proper exposure -2 steps to +1 step, FIG. 7 (B) enlarges proper exposure-2 steps to proper exposure, and FIG. 7 (C) enlarges +1 step from proper exposure. Showed. Further, FIG. 1 shows a state in which FIG. 4 is schematically emphasized.

  7 (A) -7 (C) and FIG. 8, the relationship between the G output and B output before correction deviates from the theoretical value as the output level (G = 1000 counts) at the time of proper exposure is deviated. I can read.

  Here, x is an exposure in the corresponding ISO, and an appropriate value is x = 0 [stage]. The G output at the time of exposure x is expressed as G (x), and the B output is expressed as B (x).

  For example, in FIG. 8, B (0) with respect to G (0) at the time of appropriate exposure before correction (thin solid line (1)) is on the straight line (2) of the theoretical value, whereas G at the time of appropriate +1 stage. B (+1) with respect to (+1) deviates in a direction larger than the theoretical value. Further, B (-1) with respect to G (-1) at the appropriate -1 stage deviates in a direction smaller than the theoretical value, and B (-2) with respect to G (-2) at the appropriate -2 stage is further smaller. Is dissociated.

  In other words, the color ratio between G and B is kept appropriate during proper exposure, whereas when the exposure amount is large, the ratio of the B signal increases, causing color shift, and when the exposure amount is small, the B signal The color shift is caused by the low ratio.

  The purpose of the correction shown in this embodiment is to make the B output B (x) when the G output takes a certain value G (x) close to the theoretical value B ′ (x) by a simple correction process. .

  Here, B (x) when the G output is G (x) is multiplied by a predetermined gain p, and further offset q is added to correct the color ratio so that the theoretical value B ′ (x) is obtained. The coefficients p and q are obtained.

  First, an image at a predetermined multiple exposure is acquired. As an example, here, the appropriate exposure from −2 steps to +1 step, which is a conspicuous region as the appearance of the image, is acquired in increments of one step.

  Next, from the acquired x [stage] images, an average value B (x of the outputs of B pixels in a predetermined pixel area (for example, a plurality of pixels arranged near the center of the imaging area of the solid-state imaging device) )

The average value (B (−2) to B (+1)) of the B output before correction obtained as described above is approximated as a linear function of the G output.
B (x) = a * G (x) + c (1)
The approximate expression (1) at this time is shown by a thin solid line in FIGS. 7 (A) -7 (C).
The slope of the theoretical value plot, that is, when the target color ratio B / G = d, the theoretical value is B ′ (x) = d * G (x) (2)
It can be expressed as

From equations (1) and (2)
B ′ (x) = pB (x) + q (3)
When a gain p and an offset q that satisfy
d * G (x) = p * (a * G (x) + c) + q
d * G (x) = p * a * G (x) + p * c + q
Here, when p * c + q = 0, from d = p * a, p = d / a
q = -c * d / a
From the target color ratio d (= B / G), the slope a and the intercept c of the approximate expression (1), the gain p that is a correction coefficient for correcting the approximate line so as to approach the theoretical value = D / a and offset q = -c * d / a.

  The method for deriving the correction coefficient used for correcting the B signal has been described above. The correction coefficients may be derived for other colors (here, R signal) by the same method.

  Using the gain p and the offset q, which are color ratio correction coefficients corresponding to the sensitivities obtained as described above, the imaging signals of the B and R pixels are corrected in the following procedure.

  FIG. 9 is a flowchart of the correction process according to the present embodiment. Hereinafter, correction processing according to the present embodiment will be described with reference to the flowchart.

  In step S901, the Nth pixel is selected, the signal level S (n) of the pixel is acquired, and the process proceeds to step S902.

  In step S902, it is determined whether the selected pixel is a G pixel based on a pixel address or the like. If it is determined that the pixel is a G pixel, the process proceeds to step S903, and if it is determined other than that (R pixel or B pixel), the process proceeds to step S904.

  Since color ratio correction is not performed for the G pixel, predetermined correction coefficients p = 1 and q = 0 are substituted as color ratio correction coefficients in step S903, and the process proceeds to step S907.

  In step S904, it is determined whether the selected pixel is an R pixel or a B pixel based on a pixel address or the like. If it is determined that the pixel is an R pixel, the process proceeds to step S905. If it is determined that the pixel is a B pixel, the process proceeds to step S906.

  In step S905, p = pr and q = qr are substituted as the color ratio correction coefficient of the R signal, and the process proceeds to step S907. The correction coefficient used here is read from the nonvolatile memory 14 in accordance with the sensitivity set in the imaging apparatus by the sensitivity setting unit 19. The same applies to the B signal.

  In step S906, p = pb and q = qb are substituted as the color ratio correction coefficient for the B signal, and the process proceeds to step S907.

  In step S907, the signal S (n) of the selected pixel is corrected using the color ratio correction coefficients p and q substituted in the previous step, and a correction result S '(n) is obtained. Thereafter, the process proceeds to step S908.

  In step S908, it is determined whether the selected pixel is the last pixel. If it is the last pixel, the process ends. If it is not the last pixel, the process proceeds to step S909.

  In step S909, N = N + 1, and the process returns to step S901 to continue the above processing.

  The color ratio correction coefficients p and q are set to a theoretical value B ′ (x) by applying a gain p to B (x) when the G output is G (x) and adding an offset q. This is the calculated coefficient.

  Therefore, in the correction process, the offset B is added to the actual B output after multiplying the gain p as shown in the above equation (3).

  The B signal at the high sensitivity setting corrected by such a procedure is plotted with black dots in FIGS. 7 (A) -7 (C). Moreover, it emphasized simply and showed with the thick continuous line (3) in FIG.

  For example, before correction (white dots in FIGS. 7A to 7C, thin solid line (1) in FIG. 8) with respect to G (+1) at the time of appropriate +1 stage, which deviates in a direction larger than the theoretical value. B (+1) is corrected until it is approximately on the approximate line of the theoretical value. Also, B (-1) for G (-1) at the time of the appropriate -1 stage, which was deviating in a direction smaller than the theoretical value, and B (-2) for G (-2) at the time of the appropriate -2 stage Is also corrected until it approximates the approximate line of the theoretical value.

  As described above, as shown in FIGS. 8 and 7A to 7C, it can be seen that the corrected B signal is close to the theoretical value as compared with the uncorrected B signal. . Therefore, it can be seen that the correction for the B signal is suitably performed, and the desired color ratio B / G is obtained not only at the time of proper exposure but also at the vicinity of the proper exposure.

  The B signal correction procedure has been described above. Other colors (here, R signal) may be corrected by the same method.

  As described above, the correction coefficient obtained for each sensitivity setting by the correction coefficient calculation method shown in this embodiment is used, and the image read out from the solid-state imaging device is corrected by the correction method shown in this embodiment. The color ratio of the signal can be suitably corrected, and the color reproducibility of the image can be improved.

  That is, it is possible to provide a correction unit that can reduce the deterioration of color reproducibility due to incomplete transfer, particularly the deterioration of color reproducibility with high sensitivity.

  In the first embodiment, the correction coefficient for each color determined for each sensitivity set in the imaging apparatus is uniformly used regardless of the signal level. At this time, the color reproducibility near the appropriate exposure could be improved even when the high sensitivity was set.

  Here, consider a case where an extremely dark image is taken with almost no light entering the solid-state imaging device.

  The correction coefficient for the R signal obtained by the method described in the first embodiment is the gain p_r and the offset q_r, and the correction coefficient for the B signal is the gain p_b and the offset q_b.

  For example, when light does not enter at all and the solid-state imaging device outputs a signal corresponding to a black level (0 count), all of R, G, and B are originally signals corresponding to a black level (0 count). Should be. However, by the color ratio correction described in the first embodiment, a uniform offset q_r is added to the R signal, and a uniform offset q_b is also added to the B signal. For this reason, the image signal after color ratio correction is a signal in which the R component is increased by the offset q_r and the B component is increased by the offset q_b than the signal corresponding to the black level.

  That is, if a color ratio correction coefficient that can suitably correct the color ratio at an appropriate exposure amount is used when the exposure amount is small as it is, the correction becomes overcorrected and the balance of the color ratio is lost. As a result, the color ratio of the dark part of the image is different from that of the bright part.

  In the correction method described in the first embodiment, the G component is not corrected and has no offset component, so that the luminance component decreases in the dark portion. As a result, the signal in the dark portion is not floated compared to the conventional correction in which the G signal is also offset with the same color as the other colors, so even if the color ratio in the dark portion is different from that in the bright portion, the image is extremely There is no influence that looks colored. However, it is preferable to perform color ratio correction that does not deteriorate the color reproducibility in the dark portion, because an image with better color reproducibility can be obtained.

  In this embodiment, the processing for compensating the color reproducibility in the dark portion is added to the color ratio correction processing in the first embodiment, and the color of the dark portion is corrected while suitably correcting the color ratio in the vicinity of the appropriate exposure amount. It is possible to avoid degradation of reproducibility.

  Specifically, in addition to the color ratio correction processing of the first embodiment, output signal determination means for determining the level of the output signal of each of the R and B pixels is provided to identify the dark portion. In this embodiment, the color ratio correction method is switched to another method to compensate for the color reproducibility in the dark portion.

  Accordingly, the method for obtaining the correction coefficients p and q used in the color ratio correction process of the first embodiment executed in the present embodiment is not different from that described in the first embodiment, and the description thereof is omitted.

  Hereinafter, a correction method using color ratio correction determination according to the second embodiment of the present invention will be described with reference to FIGS. 10, 11 (A), and 11 (B).

  FIG. 10 is a plot of input / output characteristics of the solid-state imaging device exemplified in Example 1 when the sensitivity is set to a lower exposure area.

  Here, if the interval (output difference) between the R and G and B and G plots of the same exposure is equal to the R and G plot interval D2 and the B and G plot intervals D1 at the appropriate exposure, respectively, It shows that a signal having the same color ratio as that at the time of exposure can be obtained.

  The R signal and B signal before color ratio correction are indicated by white dots connected by a thin solid line in FIG. Further, the result of applying the color ratio correction described in the first embodiment is shown by black dots connected by thick broken lines in FIG.

  By applying the color ratio correction described in the first embodiment to the R signal and the B signal before the color ratio correction, the exposure more than a predetermined value, in particular, the exposure more than the condition used for deriving the color ratio correction coefficient (here In the case of (appropriate -2 steps), the color ratio is corrected appropriately.

  However, for exposures less than a predetermined value, particularly exposures that do not satisfy the conditions used for deriving the color ratio correction coefficient (here, less than appropriate -2 steps), the color ratio becomes excessive and the color ratio increases as the exposure decreases. Is out of balance.

  Therefore, in order to compensate for this collapse, the color ratio correction process of the first embodiment is modified and the following correction process is added. That is, when the R signal and B signal before color ratio correction are higher than a predetermined level, the color ratio correction is performed by the correction method described in the first embodiment, and the R signal and B signal before color ratio correction are lower than the predetermined level. When it is low, the color ratio correction itself is not performed, or the color ratio correction coefficient is changed. This point can be easily understood by comparing FIG. 9 with FIGS. 11 (A) and 11 (B). Note that not performing the color ratio correction and changing the color ratio correction coefficient to p = 1 and q = 0 are equivalent.

  Specifically, the average values of the R signal level and the B signal level of the image with the lowest exposure at each sensitivity obtained at the time of deriving the color ratio correction coefficient described in the first embodiment are respectively corrected correction values for the sensitivity. Let Rj and Bj. The correction determination value is stored in advance in the nonvolatile memory (ROM) 14 of the imaging device, and the correction determination value according to the sensitivity setting of the imaging device is compared with each pixel signal before color ratio correction to determine correction. And determine how to correct the corresponding pixel based on the determination result. The setting of the correction determination value is not limited to the above method, and may be set by another method and stored in the nonvolatile memory 14.

  For example, correction determination and color ratio correction processing may be performed according to the flowcharts illustrated in FIGS. 11A and 11B.

  FIG. 11A shows a flowchart of a correction process for changing the color ratio correction coefficient when the R signal and B signal before color ratio correction are lower than the correction determination values Rj and Bj, respectively. FIG. 11B shows a flowchart of a correction process in which color ratio correction is not performed when the R signal and B signal before color ratio correction are lower than the correction determination values Rj and Bj, respectively. In the figure, the same steps as those in FIG. 9 are denoted by the same reference numerals.

  First, the color ratio correction process shown in FIG.

  In step S901, the Nth pixel is selected, the signal level S (n) of the pixel is acquired, and the process proceeds to step S902.

  In step S902, it is determined whether the selected pixel is a G pixel based on a pixel address or the like. If it is determined that the pixel is a G pixel, the process proceeds to step S903, and if it is determined other than that (R pixel or B pixel), the process proceeds to step S904.

  Since color ratio correction is not performed for the G pixel, p = 1 and q = 0 are substituted as color ratio correction coefficients in step S903, and the process proceeds to step S907.

  In step S904, it is determined whether the selected pixel is an R pixel or a B pixel based on a pixel address or the like. If it is determined that the pixel is an R pixel, the process proceeds to step S1101, and if it is determined that the pixel is a B pixel, the process proceeds to step S1102.

  In step S1101, it is determined whether the signal level S (n) of the selected R pixel is equal to or higher than the correction determination value Rj. If it is determined that it is equal to or greater than Rj, the process proceeds to step S905, and if it is determined that it is less than Rj, the process proceeds to step S1103.

  In step S905, p = p_r and q = q_r are substituted as color ratio correction coefficients, and the process proceeds to step S907.

  Since the color ratio correction is not performed when the R signal is less than Rj, p = 1 and q = 0 are substituted as color ratio correction coefficients in step S1103, and the process proceeds to step S907. That is, the color ratio correction coefficient is changed.

  In step S1102, it is determined whether the signal level S (n) of the selected B pixel is equal to or higher than the correction determination value Bj. If it is determined that it is greater than or equal to Bj, the process proceeds to step S906, and if it is determined that it is less than Bj, the process proceeds to step S1104.

  In step S906, p = p_b and q = q_b are substituted as color ratio correction coefficients, and the process proceeds to step S907.

  Since the color ratio correction is not performed when the B signal is less than Bj, p = 1 and q = 0 are substituted as color ratio correction coefficients in step S1104, and the process proceeds to step S011.

  In step S907, the signal S (n) of the selected pixel is corrected using the color ratio correction coefficients p and q substituted in the previous step, and a correction result S '(n) is obtained. Thereafter, the process proceeds to step S908.

  In step S908, it is determined whether the selected pixel is the last pixel. If it is the last pixel, the process ends. If it is not the last pixel, the process proceeds to step S909.

  In step S909, N = N + 1, and the process returns to step S901 to continue the above processing.

  Next, the color ratio correction process in FIG.

  In step S901, the Nth pixel is selected to acquire the signal level S (n) of the corresponding pixel, and the process proceeds to step S902.

  In step S902, it is determined whether the selected pixel is a G pixel based on a pixel address or the like. If it is determined that the pixel is a G pixel, the process proceeds to step S1103, and if it is determined other than that (R pixel or B pixel), the process proceeds to step S904.

  In step S904, it is determined whether the selected pixel is an R pixel or a B pixel based on a pixel address or the like. If it is determined that the pixel is an R pixel, the process proceeds to step S1101, and if it is determined that the pixel is a B pixel, the process proceeds to step S1102.

  In step S1101, it is determined whether the signal level S (n) of the selected R pixel is equal to or higher than the correction determination value Rj. If it is determined that it is greater than or equal to Rj, the process proceeds to step S905, and if it is determined that it is less than Rj, the process proceeds to step S1103.

  In step S905, p = p_r and q = q_r are substituted as color ratio correction coefficients, and the process advances to step S1105.

  In step S1102, it is determined whether the signal level S (n) of the selected B pixel is equal to or higher than the correction determination value Bj. If it is determined that it is greater than or equal to Bj, the process proceeds to step S906, and if it is determined that it is less than Bj, the process proceeds to step S1103.

  In step S906, p = p_b and q = q_b are substituted as color ratio correction coefficients, and the process proceeds to step S1105.

  Color ratio correction is not performed for the G signal, the R signal less than Rj, and the B signal less than Bj, so in step S904, the acquired signal level S (n) is corrected without performing arithmetic processing. The subsequent signal S ′ (n) is used, and the process proceeds to step S908.

  In step S1105, the signal S (n) of the selected pixel is corrected using the color ratio correction coefficients p and q substituted in the previous step to obtain a correction result S ′ (n). Thereafter, the process proceeds to step S908.

  In step S908, it is determined whether the selected pixel is the last pixel. If it is the last pixel, the process ends. If it is not the last pixel, the process proceeds to step S909.

  In step S909, N = N + 1, and the process returns to step S901 to continue the above processing.

  As described above, as a result of the comparison with the correction determination value, the R pixel whose signal level is determined to be higher than Rj and the B pixel whose signal level is determined to be higher than Bj are determined in the signal processing circuit 7. Color ratio correction is performed using the color ratio correction coefficient obtained in the first embodiment. On the other hand, as a result of the comparison, p = 1 and q = 0 are substituted as color ratio correction coefficients for the R pixel whose signal level is determined to be lower than Rj and the B pixel whose signal level is determined to be lower than Bj. Then, correction processing is performed or color ratio correction is not performed.

  The result of switching the correction according to the above flowchart is shown by the black dots connected by the thick solid line as the result after the color ratio correction (with determination) in FIG. Compared with the result after the color ratio correction (no determination) in Example 1, the R and B signals are changed so as not to have an offset. Therefore, for example, when the exposure is appropriate -4 steps, the gap D6 between G and R after correction according to the present embodiment does not change as much as the gap D4 between G and R after correction according to the first embodiment. The result is closer to the distance D2 between G and R at the proper exposure. Further, the gap D5 between G and B after correction according to the present embodiment does not change as much as the gap D3 between G and B after correction according to the first embodiment, and G and B at a more appropriate exposure. The result is close to the interval D1.

  When the appropriate exposure amount is −2 or more, a correction result equal to the result after the color ratio correction (no determination) is obtained.

  As described above, by correcting the signal read from the solid-state imaging device by the correction method using the color ratio correction determination described in the present embodiment, the predetermined exposure is performed without deteriorating the color ratio of the image signal in the dark part due to overcorrection. It is possible to suitably correct a color ratio that is greater than or equal to the amount and improve the color reproducibility of the image.

  In the first embodiment, the color ratio correction coefficient for each color obtained for each sensitivity set in the imaging apparatus is uniformly used regardless of the signal level. At this time, the color reproducibility near the appropriate exposure could be improved even when the high sensitivity was set.

  Further, in the second embodiment, when an extremely dark image is taken, if the color ratio correction coefficient obtained by the method of the first embodiment is used as it is, it is compensated that the balance of the color ratio is lost due to excessive correction. Added the process to do. Thereby, it was corrected that the color ratio of the dark part of the image is different from that of the bright part.

  By the way, as a characteristic of human photoreceptor cells, the color of the dark part is difficult to visually recognize against the bright part. Therefore, an image close to the appearance can be generated by performing processing such as attenuating the color component in the dark portion and leaving the luminance component information in the dark portion. Therefore, in this embodiment, the color ratio is suitably corrected in the vicinity of the appropriate exposure amount (first embodiment), and the process added in the second embodiment is modified in the dark portion to leave the luminance component and the color component. Correction for attenuating is performed.

  Also in the present embodiment, the method of obtaining the correction coefficients p and q used for the correction near the appropriate exposure amount is not different from that described in the first embodiment, and thus the description thereof is omitted.

  As in the second embodiment, the present embodiment includes output signal determination means for determining the level of the output signal of each of the R and B pixels for identifying the dark portion, and the correction method according to the second embodiment according to the determination result. A correction method switching process in which the switching is changed is performed.

  That is, the correction method switching process in this embodiment is characterized by the following two points. One is to use signals of pixels of different colors arranged around the selected pixel for output determination. The other is that, as a result of the determination, for the pixel determined not to perform color ratio correction, the signal of the corresponding pixel is used as it is according to the content of the determination, or the signal of the corresponding pixel is used as a signal of a predetermined level. It is to choose whether to replace with.

  The color ratio correction process according to the third embodiment of the present invention will be described below with reference to FIGS.

  FIG. 12 is a graph in which the input / output characteristics at the time of high sensitivity setting of the solid-state imaging device exemplified in Example 1 are plotted to a lower exposure range, as in FIG.

  The R signal and B signal before color ratio correction are indicated by white dots connected by a thin solid line in FIG. The result of applying the color ratio correction described in Example 1 is indicated by black dots connected by thick broken lines.

  By applying the color ratio correction described in the first embodiment to the R signal and the B signal before the color ratio correction, the exposure more than a predetermined value, in particular, the exposure more than the condition used for deriving the color ratio correction coefficient (here In (appropriate -2 steps or more), the color ratio is corrected appropriately.

  However, if the exposure is less than a predetermined value, in particular, the exposure that does not satisfy the conditions used for deriving the color ratio correction coefficient (here, less than appropriate -2 steps), the color ratio correction results in overcorrection, and the color decreases as the exposure decreases. The balance of the ratio is losing.

  In order to attenuate the color component while leaving the luminance component in the dark portion while suitably correcting the color ratio at an exposure amount in the vicinity of an appropriate amount for such a signal, a color ratio correction process according to the flowchart shown in FIG. 13 is performed. Just do it.

  The color ratio correction process according to the present embodiment will be described below with reference to the flowchart shown in FIG. In the figure, the same steps as those in FIG. 9 and FIGS. 10A and 10B are denoted by the same reference numerals.

  In step S901, the Nth pixel is selected to acquire the signal level S (n) of the corresponding pixel, and the process proceeds to step S1301.

  In step S1301, the signal level S (n) _1 of the first peripheral pixel group arranged around the Nth pixel is acquired, and the process proceeds to step S1302. The first peripheral pixel group is a pixel group composed of pixels of a single color different from the target pixel. For example, as shown in FIG. 14, in a solid-state imaging device in which pixels of R, G, and B colors are arranged in a Bayer pattern, four pixels that are adjacent to each other in the vertical and horizontal directions are selected with respect to the target pixel. It shall be. Further, the signals of a plurality of selected pixels are averaged to obtain a signal level S (n) _1 of the first peripheral pixel group.

  In step S1302, the signal level S (n) _2 of the second peripheral pixel group arranged around the Nth pixel is acquired, and the process proceeds to step S204. The second peripheral pixel group is a pixel group composed of pixels of a single color different from the target pixel and the first pixel group. For example, as shown in FIG. 14, in a solid-state imaging device in which pixels of R, G, and B colors are arranged in a Bayer array, four pixels that are adjacent to the target pixel in the tanning direction are selected. Further, the signals of a plurality of selected pixels are averaged to obtain a signal level S (n) _2 of the second peripheral pixel group.

  In step S902, it is determined whether the selected pixel is a G pixel based on a pixel address or the like. If it is determined that the pixel is a G pixel, it is determined that color ratio correction is unnecessary, and the process proceeds to step S1103. If it is determined that it is other than that (R pixel or B pixel), the process proceeds to step S1303.

  In step S1303, it is determined whether the signal level S (n) of the Nth pixel is equal to or higher than the signal level S (n) _1 of the first peripheral pixel group. Here, since it is known from the determination in step S902 that S (n) is an R or B signal, the first peripheral pixel group is always a G pixel as shown in FIG. n) _1 represents the G signal level in the vicinity of the Nth pixel. If S (n) is greater than or equal to S (n) _1, that is, if it is determined that the signal of the pixel of interest is greater than or equal to the neighboring G signal, the pixel of interest has an output with a sufficient ratio relative to the neighboring G signal. In step S1103, it is determined that color ratio correction is unnecessary. If it is determined that S (n) is less than S (n) _1, the process proceeds to step S904.

  In step S904, it is determined whether the Nth pixel is an R pixel or a B pixel based on a pixel address or the like. If it is determined that the pixel is an R pixel, the process proceeds to step S1304. If it is determined that the pixel is a B pixel, the process proceeds to step S1305.

  In step S1304, according to the determination that the Nth pixel is an R pixel, p = p_r and q = q_r are substituted as color ratio correction coefficients for the Nth pixel. Further, since the second peripheral pixel group is a B pixel, p2 = p_b and q2 = q_b are substituted as the color ratio correction coefficients for the second peripheral pixel group. Thereafter, the process proceeds to step S907.

  In step S1305, according to the determination that the Nth pixel is a B pixel, p = p_b and q = q_b are substituted as color ratio correction coefficients for the Nth pixel. Since the remaining second peripheral pixel group is an R pixel, p2 = p_r and q2 = q_r are substituted as color ratio correction coefficients for the second peripheral pixel group. Thereafter, the process proceeds to step S907.

  In step S907, the signal S (n) of the Nth pixel is corrected using the color ratio correction coefficients p and q substituted in the previous step to obtain a correction result S ′ (n). Thereafter, the process proceeds to step S1306.

  In step S1306, it is determined whether the correction result S ′ (n) of the Nth pixel signal is greater than or equal to S (n) _1 that is a neighboring G pixel signal. If it is determined that it is greater than or equal to S (n) _1, the process proceeds to step S1310, and if it is determined that it is less than S (n) _1, the process proceeds to step S1307.

  In step S1307, it is determined whether the signal S (n) _2 of the second peripheral pixel group is equal to or higher than the signal level S (n) _1 of the first peripheral pixel group. S (n) _1 is the G signal level in the vicinity of the Nth pixel. If it is determined that S (n) _2 is equal to or greater than S (n) _1, that is, the signal of the second peripheral pixel group is equal to or greater than the neighboring G signal, it is determined that the color ratio correction is overcorrected. The process proceeds to step S1310. If it is determined that S (n) is less than S (n) _1, the process proceeds to step S1308.

  In step S1308, the signal S (n) _2 of the second peripheral pixel group is corrected using the color ratio correction coefficients p_2 and q_2 substituted in the previous step to obtain a correction result S ′ (n) _2. Thereafter, the process proceeds to step S1309.

  In step S1309, it is determined whether or not the pixel signal correction result S ′ (n) _2 of the second peripheral pixel group is equal to or greater than S (n) _1 that is a neighboring G pixel signal. If it is determined that it is greater than or equal to S (n) _1, it is determined that the color ratio correction is overcorrected, and the process proceeds to step S1310. If it is determined that it is less than S (n) _1, it is determined that the color ratio correction is appropriate, and the process proceeds to step S1311.

  In step S1311, a signal S ′ (n) obtained by correcting the color ratio of the signal of the Nth pixel is output as a corrected signal, and the process proceeds to step S908.

  In step S1310, the signal S ′ (n) obtained by correcting the color ratio of the signal of the Nth pixel is replaced with 0 count corresponding to a predetermined level, here, the black level. The replaced signal is output as a corrected signal, and the process proceeds to step S908.

  In step S1310, the signal S (n) of the Nth pixel not subjected to color ratio correction is output as it is, and the process proceeds to step S908.

  In step S908, it is determined whether the selected pixel is the last pixel. If it is the last pixel, the process ends. If it is not the last pixel, the process proceeds to step S909.

  In step S909, N = N + 1, and the process returns to step S901 to continue the above processing.

  As described above, as a result of the comparison with the neighboring G signal, the R pixel and the B pixel determined that the signal level before the color ratio correction is higher than the G signal have a signal level equal to or higher than that of the G pixel. Therefore, it is determined that the color ratio correction is unnecessary, and the color ratio correction is not performed.

  In addition, the target pixel excluding the G pixel and another color pixel in the vicinity of the target pixel have both signals before the color ratio correction being less than the nearby G signal, and when both are corrected for the color ratio, the neighboring G signal When the value exceeds the value, it is determined that the color ratio correction is excessive. At this time, the output of the target pixel is replaced with a predetermined level (for example, a signal level corresponding to the black level). In other cases, the color ratio correction is performed using the color ratio correction coefficient obtained in the first embodiment.

  The result of switching the correction in accordance with the above flowchart is shown as a black dot connected with a thick solid line in FIG. 12 as the result after the color ratio correction (with determination) in FIG. Compared with the result after the color ratio correction (no determination) in the first embodiment, the B signal and the R signal are replaced with 0 count representing the black level when the B signal is about to exceed the G signal. On the other hand, even when the exposure is low, the G signal is maintained as it is. In other words, the color ratio correction equivalent to that of the first embodiment is applied to the bright portion, while the color component is attenuated and the luminance component information is left in the dark portion. As a result, it is possible to generate an image close to the appearance of human eyes, in which the color ratio of the bright part is suitably corrected and the dark part is not colored and only the luminance component is felt.

  9, 11A, 11B, and 13 in the above-described embodiment, the program for realizing the function of each process is read from the memory 14, and the system control unit 13 is also executed by the CPU. Is to realize.

  Note that the present invention is not limited to the above-described configuration, and all or some of the functions shown in FIGS. 9, 11A, 11B, and 13 may be realized by dedicated hardware. The above-mentioned memory can be read and written by a computer using a magneto-optical disk device, a non-volatile memory such as a flash memory, a recording medium such as a CD-ROM, a volatile memory other than a RAM, or a combination thereof. The recording medium may be a simple recording medium.

  Further, a program for realizing the functions of the processes shown in FIGS. 9, 11A, 11B and 13 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system. Each process may be performed by executing. Here, the “computer system” includes an OS and hardware such as peripheral devices. Specifically, the program read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” includes those that hold a program for a certain period of time. For example, a recording medium such as a volatile memory (RAM) inside a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line is included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  A program product such as a computer-readable recording medium in which the above program is recorded can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and program product are included in the scope of the present invention.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (14)

Sensitivity setting means for setting shooting sensitivity;
Imaging means for performing photoelectric conversion at the photographing sensitivity set by the sensitivity setting means and outputting an imaging signal of a subject;
For each photographing sensitivity set by the sensitivity setting means, a color ratio correction coefficient for each color generated using an output signal of a different color included in an imaging signal output from the imaging means corresponding to a different exposure amount. Storage means for storing;
Correction means for correcting the imaging signal output from the imaging means using the color ratio correction coefficient stored in the storage means;
In the photographing of the subject, a color ratio correction coefficient corresponding to the photographing sensitivity set by the sensitivity setting means is acquired from the storage means, the correction means is controlled, and the imaging signal of the subject output from the imaging means An image pickup apparatus comprising: control means for correcting output signals of different colors included in the image using a color ratio correction coefficient of a corresponding color among the acquired color ratio correction coefficients.   The control unit controls the correction unit to correct an output signal of a predetermined color included in the imaging signal output from the imaging unit regardless of the output level of the corresponding color ratio. Correction using a coefficient, and correcting an output signal of another color other than the predetermined color by using a predetermined correction coefficient determined separately from the color ratio correction coefficient regardless of the output level. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized.   The control means determines an output level of an output signal of a predetermined color included in the image pickup signal output from the image pickup means, and corresponds when the output level is lower than a predetermined level corresponding to a predetermined color. The imaging apparatus according to claim 2, wherein the correction using the color ratio correction coefficient is changed to another correction method.   The predetermined level is a value that is set based on an output signal of a predetermined color used to generate the color ratio correction coefficient and is stored in the storage unit. 4. The correction using a correction coefficient determined separately from the color ratio correction coefficient used for the output signal of the other color as the correction coefficient of the output signal of the other color. Imaging device.   The predetermined level is a value set from an output signal of a peripheral pixel having a predetermined positional relationship with respect to a pixel of the output signal of the predetermined color, and the other correction method is the color of the corresponding color 4. The correction according to claim 3, wherein one of the predetermined signals different depending on the level of the output signal of the predetermined color corrected using the ratio correction coefficient is output as the corrected output signal. 5. Imaging device.   The peripheral pixels in the predetermined positional relationship are pixels that are adjacent to the pixel of the output signal of the predetermined color and output the output signal of the other color, and are different predetermined outputs that are output in the other correction methods. The signal is an output signal corresponding to the output signal of the predetermined color and the black level corrected using the color ratio correction coefficient of the corresponding color, and the control unit controls the correction unit to An output signal corresponding to the black level is output as a correction signal when the level of the output signal of the predetermined color corrected using the color ratio correction coefficient of the color to be exceeded exceeds the predetermined level. The imaging device according to claim 5.   The color ratio correction coefficient is determined by approximating a ratio of an output signal of a predetermined color included in an imaging signal output from the imaging unit corresponding to a different exposure amount with a linear function of the output signal of the other color. And a gain and an offset value for correcting an output signal of a predetermined color, wherein the correction coefficient determined separately from the color ratio correction coefficient has a gain of 1 and an offset of 0. The imaging device according to any one of claims 2 to 6.   The imaging apparatus according to claim 1, wherein the output signal of the other color is an output signal of a color that contributes most to a luminance component of the imaging signal. Processing the imaging signal output from an imaging apparatus comprising: sensitivity setting means for setting imaging sensitivity; and imaging means for performing photoelectric conversion at the imaging sensitivity set by the sensitivity setting means and outputting an imaging signal of a subject. An image processing device,
For each photographing sensitivity set by the sensitivity setting means, a color ratio correction coefficient for each color generated using an output signal of a different color included in an imaging signal output from the imaging means corresponding to a different exposure amount. Storage means for storing;
Correction means for correcting the imaging signal output from the imaging means using the color ratio correction coefficient stored in the storage means;
In the photographing of the subject, a color ratio correction coefficient corresponding to the photographing sensitivity set by the sensitivity setting means is acquired from the storage means, the correction means is controlled, and the imaging signal of the subject output from the imaging means An image processing apparatus comprising: a control unit that corrects output signals of different colors included in the image using a color ratio correction coefficient of a corresponding color among the acquired color ratio correction coefficients. Processing the imaging signal output from an imaging apparatus comprising: sensitivity setting means for setting imaging sensitivity; and imaging means for performing photoelectric conversion at the imaging sensitivity set by the sensitivity setting means and outputting an imaging signal of a subject. An image processing method,
For each photographing sensitivity set by the sensitivity setting means, a color ratio correction coefficient for each color generated using an output signal of a different color included in an imaging signal output from the imaging means corresponding to a different exposure amount. A storage step of storing in the storage means;
A correction step of correcting the imaging signal output from the imaging unit using the color ratio correction coefficient stored in the storage unit;
In the shooting of the subject, a color ratio correction coefficient corresponding to the shooting sensitivity set by the sensitivity setting unit is acquired from the storage unit, the correction step is controlled, and the imaging signal of the subject output from the imaging unit And a control step of correcting output signals of different colors included in the image using a color ratio correction coefficient of a corresponding color among the acquired color ratio correction coefficients. Computer
In a control method of an imaging apparatus, comprising: sensitivity setting means for setting shooting sensitivity; and imaging means for performing photoelectric conversion at the shooting sensitivity set by the sensitivity setting means and outputting an imaging signal of a subject.
For each photographing sensitivity set by the sensitivity setting means, a color ratio correction coefficient for each color generated using an output signal of a different color included in an imaging signal output from the imaging means corresponding to a different exposure amount. Storage means for storing;
Correction means for correcting the imaging signal output from the imaging means using the color ratio correction coefficient stored in the storage means;
In the photographing of the subject, a color ratio correction coefficient corresponding to the photographing sensitivity set by the sensitivity setting means is acquired from the storage means, the correction means is controlled, and the imaging signal of the subject output from the imaging means A program that functions as control means for correcting the output signals of different colors included in the image using the color ratio correction coefficients of the corresponding colors among the acquired color ratio correction coefficients.   A computer-readable storage medium storing the program according to claim 11.   A program that causes a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 8.   A storage medium storing a program that causes a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 8.

Images