JP2008131580A - Imaging apparatus, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create an image in which both luminance noise and color noise are improved while suppressing deterioration in resolution even during high-sensitivity photography. <P>SOLUTION: An image in a full pixel read-out mode and an image in a pixel summation read-out mode which are substantially simultaneously obtained are separated into a luminance element and a color difference element to adaptively perform synthetic processing. Full pixel image data are characterized by high resolution, and pixel summation image data are characterized by a good luminance / color noise characteristic. The luminance element of the full pixel image data and the luminance element of the pixel summation image data are subjected to weighted average according to the luminance in order to reduce the luminance noise while suppressing deterioration in a resolution feeling. Accordingly, the luminance element image excellent in the noise characteristic and resolution is created. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, and more particularly, to an imaging apparatus capable of reducing noise in image data and an image processing method thereof.

  High-sensitivity shooting is used as a shooting method in a low-light environment in all imaging apparatuses including a digital still camera. The high-sensitivity imaging here is a technique for obtaining an image signal having a target intensity in a short time by amplifying an image signal output from the imaging element by an amplifier. In such high-sensitivity shooting, there is an advantage that even if an image with the same exposure is captured, it is possible to finish shooting with a short exposure compared to low-sensitivity shooting, while the amount of charge accumulated in the image sensor In addition to the signal component being buried in the noise component generated in the subsequent stage due to the decrease in the image quality, there is a problem that the image quality is significantly deteriorated due to excessive signal amplification by the amplifier.

For this reason, for example, Patent Document 1 proposes a technique for suppressing noise without performing unnecessary gain increase by an amplifier by using an image pickup device including pixel addition means. In this method, a pixel addition means adds a plurality of pixel values (charges) adjacent to the image sensor to obtain a pixel value of one pixel, thereby obtaining a signal component and suppressing amplification by an amplifier as much as possible to increase sensitivity. It is.
JP 2005-303519 A

  Here, since pixel addition replaces a plurality of adjacent pixels with one pixel, the image size is inevitably reduced. Therefore, even if noise is reduced, the resolution of the image is lowered.

  In other words, all pixel image data acquired by all pixel readout has high resolution, while high sensitivity increases the noise level. Conversely, pixel addition image data acquired by pixel addition readout has color noise and luminance noise. Both of them have good characteristics, but the image is reduced by the pixel addition process and the resolution is lowered.

  The present invention has been made in view of the above circumstances, and an imaging apparatus capable of generating an image with improved noise while suppressing a decrease in resolution during high-sensitivity imaging, and an image in such an imaging apparatus An object is to provide a processing method.

  In order to achieve the above object, the imaging device according to the first aspect of the present invention includes at least a pixel addition reading mode in which charges of a plurality of adjacent pixels are added and read out, and all pixels that read out charges corresponding to all the pixels. An image sensor having a plurality of drive modes consisting of a readout mode, an operation of driving the image sensor in the pixel addition readout mode to acquire pixel addition image data, and driving the image sensor in the all pixel readout mode A simultaneous photographing unit that performs the operation of acquiring all pixel image data simultaneously or continuously, and the pixel addition image data and the all pixel image data obtained by the simultaneous photographing unit are converted into a luminance component and a color difference component, respectively. A luminance / chrominance separation unit that separates the luminance component of the pixel-added image data and the luminance component of the all-pixel image data to synthesize luminance image data A luminance component image synthesizing unit to be generated, characterized by comprising the said combined luminance image data and the luminance and color difference synthesis unit for synthesizing one color image by synthesizing the color difference component of the pixel addition image data.

  In order to achieve the above object, the image processing method according to the second aspect of the present invention includes at least a pixel addition read mode in which charges of a plurality of adjacent pixels are read and a charge corresponding to all pixels. An image processing method for processing output from an image sensor having a plurality of drive modes consisting of a read all-pixel read mode, wherein the image sensor is driven in the pixel addition read mode to obtain pixel added image data And the operation of acquiring the all-pixel image data by driving the image sensor in the all-pixel readout mode simultaneously or continuously, and the luminance addition component and the color-difference component respectively for the pixel addition image data and the all-pixel image data. The luminance component of the pixel-added image data and the luminance component of the all-pixel image data are separated from the luminance component of the pixel-added image data and the luminance component of the pixel-added image data. A composite luminance image data is generated by weighted averaging at a ratio based on any one of the luminance components of the pixel image data, and one color image is synthesized by synthesizing the synthetic luminance image data and the color difference component of the pixel addition image data. It is characterized by doing.

  According to these first and second aspects, by combining high-resolution all-pixel image data and pixel-added image data with good noise characteristics, noise reduction is achieved while suppressing a decrease in resolution during high-sensitivity shooting. An image with improved can be generated.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing method in such an imaging device which can generate | occur | produce the image which improved the noise, suppressing the fall of a resolving power at the time of high sensitivity imaging | photography, and such an imaging device can be provided.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, embodiment described below is a part of many embodiment, and does not limit this invention.
[First Embodiment]
FIG. 1 is a block diagram showing the main configuration of the imaging apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the imaging apparatus of the present embodiment includes a lens 1, an imaging device 2, an analog signal processing unit 3, an A / D conversion unit 4, a CPU 5, a flash memory 6, and a memory 7. An image processing unit 8, a JPEG processing unit 9, an external storage interface (I / F) 10, an external storage device 11, an LCD driver 12, an LCD 13, an operation unit 14, and a bus 15. . In addition, an imaging unit 16 is configured by the imaging device 2, the analog signal processing unit 3, and the A / D conversion unit 4.

  The lens 1 collects light from a subject (not shown). The imaging device 2 receives the light collected by the lens 1 and converts it into a charge corresponding to the amount of light received, and outputs this charge as a video signal. Here, the image sensor 2 may be a CCD system or a CMOS system as long as it is an image sensor having an all-pixel readout mode and a pixel addition readout mode. The analog signal processing unit 3 includes a correlated double sampling (CDS) circuit and an auto gain control (AGC) circuit, and performs analog processing on the video signal obtained from the image sensor 2. The A / D converter 4 digitizes the video signal obtained by the analog signal processor 3 to generate digital image data.

  Here, digital image data (hereinafter referred to as all pixel image data and pixel addition image data) obtained by driving the image pickup device 2 in the all pixel readout mode and the pixel addition readout mode is obtained from the SDRAM or the like via the bus 15. Stored in the memory 7. The image processing unit 8 reads out all pixel image data and pixel added image data from the memory 7 and performs image processing. The JPEG processing unit 9 reads out all pixel image data and pixel addition image data processed by the image processing unit 8 and stored in the memory 7, and performs JPEG compression processing. The compressed image data obtained by compression by the JPEG processing unit 9 is stored in the external storage device 11 via the external storage I / F 10. The JPEG processing unit 9 also performs processing for reading compressed image data stored in the external storage device 11 and decompressing it. The LCD driver 12 reads out the image data processed by the image processing unit 8 from the memory 7, converts it into a video signal, and displays an image on the LCD 13 based on the video signal.

  The CPU 5 serving as a simultaneous photographing unit controls each unit of the imaging apparatus illustrated in FIG. The flash memory 6 stores various programs executed by the imaging apparatus and parameters for executing various processes. The operation unit 14 is an operation unit for an operator to perform various operation inputs.

  FIG. 2 is a flowchart showing a program executed in the CPU 5. The program shown in FIG. 2 is stored in the flash memory 6.

  When the release button included in the operation unit 14 is half-pressed and the 1st release switch is turned on (step S1), the CPU 5 starts an AE (Auto Exposure) operation to determine the exposure condition at the time of shooting and AF ( (Auto Focus) operation is started to determine the focal position, and the lens 1 is driven in accordance with the determined focal position (step S2). Thereafter, when the release button is fully pressed and the 2nd release switch is turned on (step S3), the CPU 5 controls a shutter and a diaphragm (not shown) according to the exposure conditions, and drives the image sensor 2 under predetermined photographing conditions. Charge accumulation is started (step S4). After the charge accumulation is completed, charges are read from the image sensor 2 in the all-pixel reading mode (step S5). The read charges are digitized through the analog signal processing unit 3 and the A / D conversion unit 4. All pixel image data obtained in this way is stored in the memory 7 via the bus 15 (step S6).

  Continuing with step S6, the CPU 5 starts the accumulation of electric charges by driving the image pickup device 2 under photographing conditions that take into account the increase in sensitivity due to pixel addition (step S7). After the charge accumulation is completed, the charge is read from the image sensor 2 in the pixel addition reading mode (step S8). The read charges are digitized through the analog signal processing unit 3 and the A / D conversion unit 4. The pixel addition image data obtained in this way is stored in the memory 7 via the bus 15 (step S9). Here, as shown in FIG. 1, when only one image sensor is provided, the all-pixel image and the pixel-added image cannot be acquired unless they are taken with a time difference. Therefore, when a fast-moving subject is photographed, spatial misalignment may occur between the two images, but the apparatus can be simplified because the number of image sensors can be reduced. In the configuration of FIG. 1, it is necessary to acquire all pixel images and pixel addition images with a time difference, but the acquisition order is not limited to the order shown in FIG.

  In addition, when capturing image data in step S6 and step S9, the image signal obtained in the pixel addition reading mode is compared with the image signal obtained in the all-pixel reading mode, and the gain increase performed in the analog signal processing unit 3 is performed. Since the degree of is small, an image with a small amount of noise is obtained. On the other hand, the pixel-added image data is smaller in size than the all-pixel image data.

  Since all pixel image data and pixel addition image data obtained in steps S6 and S9 are image data in which each pixel has single-color information of R, G, or B, one pixel contains information on three colors. As shown, the image processing unit 8 performs synchronization processing on each image data. Further, the image processing unit 8 performs white balance processing on each image data subjected to the synchronization processing. Thereafter, each image data is stored in the memory 7 (step S10).

  Further, in order to perform luminance image synthesis processing for noise reduction processing to be performed later, the image processing unit 8 converts all pixel image data in which each pixel is composed of three components of R, G, and B to luminance (Y ) And color difference (C) component images. Thereafter, the color difference component is discarded, and only the luminance component YF of all pixel image data is stored in the memory 7 (step S11). Next, the image processing unit 8 separates the pixel-added image data into luminance and color difference component images. Thereafter, the luminance component YM and the color difference components CrM and CbM are respectively stored in the memory 7 (step S12).

  As described above, since the pixel-added image data is smaller in size than the all-pixel image data, the image processing unit 8 performs YM, CrM so that the pixel-added image data and the all-pixel image data have the same image size. The resize conversion process (pixel value interpolation process) is performed on CbM, and then stored in the memory 7 (step S13). Thereafter, the resized pixel addition image data is used.

  Next, the image processing unit 8 performs luminance image synthesis processing for reducing luminance noise using the luminance components YF and YM (step S14). The process performed here is a process of acquiring pixel values at the same position in YF and YM, and setting the weighted average value of the acquired pixel values as the pixel value of the combined luminance image data. And when performing a weighted average process, the weighted average ratio of YF and YM is changed according to the change of a brightness | luminance. That is, a plurality of luminance thresholds and a weighted average ratio corresponding to the luminance thresholds are set in advance, one of YF and YM is set as the reference luminance image data, the other is the reference luminance image data, and the reference luminance image data The pixel value is compared with the luminance threshold, and a weighted average with the pixel value of the reference luminance image data is obtained with a weighted average ratio corresponding to the comparison result. FIG. 3 shows a flowchart detailing the process of step S14.

  FIG. 3 specifically shows the process of step S14. First, for the sake of simplicity, the luminance region is divided into three steps of a low luminance region, a medium luminance region, and a high luminance region, and the low luminance-medium luminance luminance threshold is T1, and the medium luminance-high luminance luminance threshold is T2. . Correspondingly, the weighted average ratios in the low luminance region, medium luminance region, and high luminance region are R1, R2, and R3, respectively. In general, noise is conspicuous in the dark part (low luminance region). Therefore, the ratio R1 corresponding to the low luminance region has a large weight for YM with little noise, and the ratio R2 corresponding to the medium luminance region is about the same for each of YM and YF. The ratio R3 corresponding to the high-luminance area emphasizes resolution, so by giving YF a large weight, the luminance noise level in the dark part can be kept low, and noise is not noticeable from the middle / high-luminance area It is possible to suppress a decrease in resolution at a low level. Based on these, the flowchart of FIG. 3 will be described.

  The image processing unit 8 first determines reference luminance image data for synthesis (step S21). Here, YM (the luminance component of the pixel-added image data) with less noise is used as the reference luminance image data. Next, the image processing unit 8 raster scans YM determined as the reference luminance image data, and obtains a pixel value (step S22). Then, the image processing unit 8 determines whether or not the pixel value of the pixel (referred to as the target pixel) whose pixel value has been acquired by raster scanning is equal to or less than the luminance threshold T1 (step S23). In the determination in step S23, if the pixel value of the pixel of interest is equal to or less than the luminance threshold T1, that is, if the pixel value of the pixel of interest belongs to the low luminance region, the image processing unit 8 calculates the YM pixel value acquired in step S22, A YF pixel value existing at the same position is weighted and averaged at a weighted average ratio R1 (step S24). Thereafter, the pixel value obtained by the weighted average process is output as the luminance value of the synthesized luminance image data Y (step S25). After that, the image processing unit 8 determines whether or not the current pixel of interest is a terminal pixel (for example, when raster scanning is started from the upper left pixel, the lower right pixel is the final pixel). (Step S26). If it is determined in step S26 that the current pixel of interest is a terminal pixel, the series of processing ends, and if not, the process returns to step S22 to restart raster scanning.

  If it is determined in step S23 that the pixel value of the target pixel exceeds the luminance threshold T1, the image processing unit 8 determines whether or not the pixel value of the target pixel is equal to or lower than the luminance threshold T2 ( Step S27). In the determination in step S27, if the pixel value of the pixel of interest is equal to or less than the luminance threshold T2, that is, if the pixel value of the pixel of interest belongs to the medium luminance region, the image processing unit 8 determines the pixel value of YM acquired in step S22, The YF pixel value existing at the same position is weighted and averaged by the weighted average ratio R2 (step S28), and the obtained pixel value is output as the luminance value of the synthesized luminance image data Y. If the pixel value of the pixel of interest exceeds the luminance threshold T2 in the determination in step S27, that is, if the pixel value of the pixel of interest belongs to the high luminance area, the image processing unit 8 acquires in step S22. The YM pixel value and the YF pixel value existing at the same position are weighted and averaged at a weighted average ratio R3 (step S29), and the obtained pixel value is used as the luminance value of the synthesized luminance image data Y. Output. By performing these processes, the luminance image data Y synthesized according to the luminance is finally generated. This synthesized luminance image data Y is an image in which luminance noise that is conspicuous in the low luminance region is greatly suppressed, and a decrease in resolution in the middle and high luminance regions is suppressed.

  Here, the description returns to the flowchart of FIG. After performing the luminance image data combining process in step S14, the image processing unit 8 combines the combined luminance image data Y and the color difference components CrM and CbM of the pixel addition image data to generate an image of R, G, and B components. The image is reconstructed by conversion to data and stored in the memory 7 (step S15). Since the color difference component of pixel-added image data is low noise compared to that of all pixel image data, using this as the color difference component of reconstructed image data significantly reduces not only luminance noise but also color noise. It becomes possible to make it. Next, the JPEG processing unit 9 reads the color image data composed of the R, G, and B components reconstructed from the memory 7 and performs compression processing (step S16), and then the compressed image data is transmitted via the external storage I / F 10. Is written in the external storage device 11 such as a memory card (step S17).

  With the series of processing shown in FIGS. 2 and 3, it is possible to obtain an image in which luminance noise and color noise can be significantly reduced and the degree of resolution degradation is suppressed to a low level. Further, when determining the weighted average ratio in the synthesis of luminance image data and when reconstructing the image data, the noise component can be further reduced by using the luminance component and color difference component of the pixel-added image data with less noise. It is possible to increase.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. FIG. 4 is a block diagram showing a main configuration of an imaging apparatus according to the second embodiment of the present invention. Here, in FIG. 4, the same reference numerals as those in FIG. 1 are the same as those in FIG. The imaging apparatus shown in FIG. 4 is provided with an optical path branching prism 20. The optical path branching prism 20 branches the light collected by the lens 1, makes one branched light incident on the imaging unit 27, and makes the other light incident on the imaging unit 28. With such a configuration, light from the subject can be simultaneously incident on the two imaging units.

  That is, the light condensed by the lens 1 is divided into two systems by the optical path branching prism 20, and each light is received by the image sensor 21 and the image sensor 22. Here, the image sensor 21 has an all-pixel readout mode, and the image sensor 22 has a pixel addition readout mode. The video signal obtained from the image sensor 21 is converted into digital image data (all-pixel image data) via an analog signal processing unit 23 and an A / D conversion unit 25 including later correlated double sampling (CDS) and auto gain control (AGC). ) Similarly, a video signal obtained from the image sensor 22 becomes digital image data (pixel-added image data) through the analog signal processing unit 24 and the A / D conversion unit 26.

  Digital image data obtained in the all-pixel readout mode and the pixel addition readout mode is stored in the memory 7 via the bus 15. Thereafter, all pixel image data and pixel addition image data read from the memory 7 are subjected to image processing by the image processing unit 8, and then subjected to JPEG compression processing by the JPEG processing unit 9, via the external storage I / F 10. It is stored in the external storage device 11. The image data processed by the image processing unit 8 is converted into a video signal through the LCD driver 12, and an image is displayed on the LCD 13 based on the video signal.

  FIG. 5 is a flowchart of a program executed by the CPU 5. The program shown in FIG. 5 is stored in the flash memory 6.

  When the release button included in the operation unit 14 is pressed halfway and the 1st release switch is turned on (step S41), the CPU 5 starts the AE operation and the AF operation (step S42). Thereafter, when the release button is fully pressed and the 2nd release switch is turned on (step S43), the CPU 5 drives the image pickup device 21 and the image pickup device 22 under predetermined shooting conditions to start accumulation of charges (step S43). S44). Charges are read out from the image sensor 21 in the all-pixel reading mode (step S45), digitized through the analog signal processing unit 23 and the A / D conversion unit 25, and then digitized to obtain all pixel images. Data is stored in the memory 7 via the bus 15 (step S46).

  Similarly, the charge is read from the image sensor 22 in the pixel addition reading mode (step S47), digitized through the analog signal processing unit 24 and the A / D conversion unit 26, and then obtained by digitization. Pixel-added image data is stored in the memory 7 via the bus 15 (step S48).

  Subsequent steps S49 to S56 are the same as steps S10 to S17 of the flowchart shown in FIG. 2 of the first embodiment. Further, the luminance image data synthesis processing performed in step S53 is also the processing of the flowchart shown in FIG. 3 of the first embodiment. Therefore, description of the subsequent processing is omitted.

  The series of processing described with reference to FIG. 5 makes it possible to significantly reduce luminance noise and color noise, and to acquire an image with reduced resolution degradation. In addition, although the configuration is complicated by preparing two optical paths and image pickup devices as in the second embodiment, all pixel images and pixel addition images can be taken simultaneously, and an image without time lag is obtained. It becomes possible. Accordingly, it is possible to increase the degree of coincidence of the pixel positions during the weighted average and obtain the synthesized luminance image data Y with higher accuracy.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention. For example, in the above-described embodiment, an example in which the luminance region is divided into three stages has been described. However, the luminance area may be divided into four or more stages. In this case, the luminance threshold and the composition ratio may be set according to the number of luminance areas, and the process of FIG. 3 may be performed including the added luminance area.

  Furthermore, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

1 is a main block diagram of an imaging apparatus according to a first embodiment of the present invention. It is a basic flow chart of a 1st embodiment of the present invention. It is the flowchart which described the detail of the brightness | luminance image synthetic | combination process. It is a main block diagram of the imaging device concerning a 2nd embodiment of the present invention. It is a basic flow chart of a 2nd embodiment of the present invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Lens, 2, 21, 22 ... Image pick-up element, 3, 23, 24 ... Analog signal processing part, 4, 25, 26 ... A / D conversion part, 5 ... CPU, 6 ... Flash memory, 7 ... Memory, 8 Image processing unit, 9 JPEG processing unit, 10 External storage interface (I / F), 11 External storage device, 12 LCD driver, 13 LCD, 14 operation unit, 15 bus, 20 optical path branching prism

Claims (6)

An image pickup device having a plurality of drive modes including at least a pixel addition read mode for adding and reading charges of a plurality of adjacent pixels and an all-pixel read mode for reading charges corresponding to all pixels;
The operation of acquiring the pixel addition image data by driving the image sensor in the pixel addition readout mode and the operation of acquiring the all pixel image data by driving the image sensor in the all pixel readout mode are executed simultaneously or continuously. A simultaneous shooting unit to
A luminance color difference separation unit that separates the pixel addition image data and the all pixel image data obtained by the simultaneous photographing unit into a luminance component and a color difference component, respectively;
A luminance component image combining unit that combines the luminance component of the pixel-added image data and the luminance component of the all-pixel image data to generate combined luminance image data;
A luminance / color difference combining unit that combines the combined luminance image data and the color difference component of the pixel-added image data to combine one color image;
An imaging apparatus comprising:   The luminance component image composition unit calculates a weighted average value of pixel values corresponding to the luminance component of the pixel-added image data and the luminance component of the all-pixel image data, and the pixel of the corresponding pixel in the synthetic luminance image data. The imaging apparatus according to claim 1, wherein the combined luminance image data is generated by using a value.   The luminance component image composition unit compares the pixel value of each pixel of any one of the luminance component of the pixel-added image data and the luminance component of the all-pixel image data with a luminance threshold, and according to the result of the comparison The imaging apparatus according to claim 2, wherein a weighted average ratio for obtaining the weighted average value is changed for each pixel.   The imaging apparatus according to claim 3, wherein the luminance component image composition unit compares the pixel value of each pixel of the luminance component of the pixel addition image data with the luminance threshold value. The luminance threshold value includes a luminance threshold value for identifying whether a pixel value of the pixel to be compared belongs to a low luminance region, a medium luminance region, or a high luminance region,
If the pixel value of the pixel belongs to the low luminance region as a result of the comparison, the luminance component image composition unit uses the weighted average ratio obtained by increasing the weight of the luminance component of the pixel added image data. A value is obtained, and when the pixel value of the pixel belongs to the medium luminance region, the weighted average ratio in which the weight of the luminance component of the pixel-added image data and the weight of the luminance component of the all-pixel image data are made equal is used. An average value is obtained, and when the pixel value of the pixel belongs to the high luminance region, the weighted average value is obtained using a weighted average ratio obtained by increasing a weight of a luminance component of the all pixel image data. The imaging device according to claim 3 or 4. Image processing for processing output from an image sensor having a plurality of drive modes including at least a pixel addition readout mode for adding and reading out charges of a plurality of adjacent pixels and an all-pixel readout mode for reading out charges corresponding to all the pixels A method,
The operation of acquiring the pixel addition image data by driving the image sensor in the pixel addition readout mode and the operation of acquiring the all pixel image data by driving the image sensor in the all pixel readout mode are executed simultaneously or continuously. And
Separating the pixel-added image data and the all-pixel image data into a luminance component and a color difference component, respectively;
The luminance component of the pixel-added image data and the luminance component of the all-pixel image data are weighted and averaged at a ratio based on either the luminance component of the pixel-added image data or the luminance component of the all-pixel image data. Generate data,
Combining the synthesized luminance image data and the color difference component of the pixel-added image data to synthesize one color image;
An image processing method.

Images