JP2020068391A - Image processing device - Google Patents

Abstract

To provide an image processing device which corrects a flicker component on an image signal and a parallax signal read from an image sensor.SOLUTION: A first flicker correction value calculation section 106 sets, as a first flicker correction value, the reciprocal of a flicker component of an image signal detected by a first flicker detector 105. A first flicker correction section 107 subtracts the first flicker correction value to remove the flicker component of the image signal. A second flicker correction value calculation section 109 sets, as a second flicker correction value, the reciprocal of a flicker component of a parallax signal detected by a second flicker detector 108. An area determination section 111 determines, from frequency information of a first flicker, whether or not a flicker of the parallax signal can be detected. A flicker correction value selection section 112 selects the second flicker correction value if detectable, and selects the first flicker correction value if undetectable. A second flicker correction section 113 outputs a parallax signal which is obtained by subtracting a selected flicker correction value to remove a flicker component.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image pickup apparatus represented by a digital camera, and particularly to a flicker correction apparatus that corrects a flicker component for a plurality of image pickup signals read from an image pickup element.

  In recent years, multifunctionalization of devices using image pickup devices has progressed, and periodical signal level fluctuations (hereinafter referred to as flicker) that occur in a captured image under a fluorescent lamp whose brightness fluctuates depending on the frequency of an AC power source There are a correction technology and an image processing technology using a parallax signal read by a pupil division method.

  Patent Document 1 discloses a technique of detecting a flicker component in the vertical direction in an image and multiplying it by an inverse gain to detect and correct the flicker component. As a result, the flicker component generated in the captured image can be removed.

  Further, in recent years, there are image pickup apparatuses that perform focus adjustment based on subject information obtained by an image pickup element. In Patent Document 2, one microlens and two photodiodes are provided for each pixel of the image pickup element, so that each photodiode receives light that has passed through different pupils of the image pickup lens. Focus detection can be performed by comparing the parallax signals from the two photodiodes, and an image signal can be generated by adding the parallax signals from the two photodiodes. However, if the parallax signals from the two photodiodes are read in the entire area, the reading time will be significantly increased. In order to suppress an increase in read time, it is possible to limit the area for focus detection and read only the added image signal from the other areas.

  Other techniques using parallax signals other than focus adjustment include reconstructing a pixel-by-pixel signal (hereinafter, pixel signal) to change the focus position after shooting, change the shooting viewpoint, and change the field of view. Image processing called refocusing such as depth adjustment is typical. By limiting the area from which the parallax signal is read in Patent Document 2, refocusing in that area is possible.

Japanese Patent No. 4795314 JP, 2016-021052, A

  However, in the conventional technique disclosed in Patent Document 1 described above, when the flicker component generated in the vertical direction is continuous, flicker detection is performed by sampling a certain area or more. On the other hand, if the area ratio is less than the threshold value, it cannot be detected. Therefore, as in the prior art disclosed in Patent Document 2, if the flicker components generated in the two photodiodes within one microlens are biased, flicker detection is also required for the parallax signals. However, it cannot be detected when the parallax signal acquisition area is less than or equal to the threshold value. If refocusing is performed with the flicker component remaining in the parallax signal, the flicker component is included in the generated image.

  Therefore, an object of the present invention is to provide an apparatus capable of appropriately performing flicker correction on a plurality of image pickup signals read from an image pickup device.

In order to achieve the above object, the image processing device according to the present invention,
A first input means for inputting a first image pickup signal;
Second input means for inputting a second image pickup signal obtained by partially and simultaneously photographing the same subject as the subject imaged with the first image pickup signal,
First flicker detection means capable of detecting a flicker component of the first image pickup signal,
Second flicker detection means capable of detecting a flicker component of the second image pickup signal,
First flicker correction value calculation means capable of calculating a first flicker correction value calculated from the flicker component of the first image pickup signal;
Second flicker correction value calculation means capable of calculating a second flicker correction value calculated from the flicker component of the second image pickup signal,
An area determination unit that determines whether the second image pickup signal can detect flicker according to the input range of the second image pickup signal;
Flicker correction value selection means for selecting either the first flicker correction value or the second flicker correction value based on the determination,
Flicker correction means for performing flicker correction on the second image pickup signal with the flicker correction value selected by the flicker correction value selection means,
It is characterized by including.

  According to the image processing apparatus of the present invention, flicker correction can be appropriately performed on a plurality of image pickup signals read from the image pickup device.

Block diagram showing a schematic configuration of an imaging apparatus 100 of an embodiment to which the present invention is applied FIG. 3 is a diagram showing a circuit configuration of a unit pixel of the image sensor 102 according to the embodiment. FIG. 3 is a block diagram showing a configuration example of a first flicker detection unit 105 and a second flicker detection unit 108 according to an embodiment. The figure which shows the output signal when the acquisition area of a parallax signal is area | region Sa> = threshold value T and the 2nd flicker detection part 108 can detect a flicker. The figure which shows the output signal when the acquisition area of a parallax signal is area | region Sb <threshold value T, and a 2nd flicker detection part 108 cannot detect a flicker. Timing chart showing input / output of the flicker correction value determination unit 110 according to the embodiment Flowchart showing the flow of parallax signal flicker correction processing according to the embodiment

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram of an image pickup apparatus according to an embodiment of the present invention.

  Hereinafter, the configuration of the image pickup apparatus 100 according to the embodiment of the present invention will be described with reference to FIG. 1.

  The image pickup apparatus 100 includes an image pickup optical system 101, an image pickup element 102, an A / D conversion unit 103, a capture unit 104, a first flicker detection unit 105, a first flicker correction value calculation unit 106, and a first flicker correction unit 107. The second flicker detection unit 108, the second flicker correction value calculation unit 109, the flicker correction value determination unit 110, the second flicker correction unit 113, the external recording unit 114, and the control unit 115.

  The configuration of the image pickup apparatus 100 according to this embodiment will be described in detail.

  An image of the real space is formed on the image pickup element 102 which converts the optical image into an electric signal from the image pickup optical system 101 including the focusing lens.

  The image sensor 102 is an image sensor having a two-dimensional array of unit pixel cells corresponding to each microlens, on the front surface of which a microlens array as a pupil dividing unit is arranged. The exposure amount is controlled by the shutter included in the imaging optical system 101. The optical image formed by the image pickup optical system 101 is photoelectrically converted and picked up, and during the read control, the charges accumulated in the divided pixels PD formed in the unit pixel are sequentially converted into a voltage, and the A / D conversion is performed. It is output to the unit 103. The single pixel cell is composed of two pupil-divided pixels (hereinafter, divided pixels a and b). Focus detection is performed by observing the phase difference between the divided pixels a and b, and a and b are added. As a result, it is used as a captured image.

  The A / D conversion unit 103 inputs an analog signal output from the image sensor 102 via a CDS circuit (not shown) or an analog signal processing unit represented by ISO sensitivity amplification, converts the analog signal into a digital signal, and captures the digital signal. It is output to the unit 104.

  The capture unit 104 determines the effective scanning period and type of the pixel data input from the A / D conversion unit 103, and uses the data obtained by adding the divided pixels a and b as an image signal, the first flicker detection unit 105, and the first flicker detection unit 105. It is output to one flicker correction unit 107. The divided pixel a is output as a parallax signal to the second flicker detection unit 108 and the second flicker correction unit 113.

  The first flicker detection unit 105 detects a flicker component (amplitude / phase of frequency) of the image signal input from the capture unit 104, and outputs it to the first flicker correction value calculation unit 106 and the area determination unit 111. .

  The first flicker correction value calculation unit 106 generates the reciprocal of the flicker component detected by the first flicker detection unit 105 as a first flicker correction value, and selects the first flicker correction unit 107 and the flicker correction value selection. It is output to the unit 112.

  The first flicker correction unit 107 subtracts the flicker component from the image signal by subtracting the image signal input from the capture unit 104 and the first flicker correction value from the first flicker correction value calculation unit 106. Output to the external recording unit 114.

  The second flicker detection unit 108 detects the flicker component (amplitude / phase of frequency) of the parallax signal input from the capture unit 104, and outputs it to the second flicker correction value calculation unit 109.

  The second flicker correction value calculation unit 109 generates the reciprocal of the flicker component detected by the second flicker detection unit 108 as a second flicker correction value, and outputs it to the flicker correction value selection unit 112.

  The flicker correction value determination unit 110 includes an area determination unit 111 and a flicker correction value selection unit 112. Since this is a characteristic configuration of the present invention, a detailed description will be given later.

  The area determination unit 111 determines whether the area where the parallax signal is read from the image sensor 102 is an area in which the flicker of the parallax signal can be detected based on the flicker frequency information from the first flicker detection unit 105. It is output to the flicker correction value selection unit 112.

  The flicker correction value selection unit 112 selects the first flicker correction value or the second flicker correction value based on the determination result of the area determination unit 111, and outputs it to the second flicker correction unit 113. If the determination result indicates that flicker of the parallax signal can be detected, the second flicker correction value from the second flicker correction value calculation unit 109 is selected, and if not, the first flicker correction value calculation unit 106. The first flicker correction value from is selected.

  The second flicker correction unit 113 subtracts the flicker component from the parallax signal input from the capture unit 104 and the flicker correction value selected by the flicker correction value selection unit 112 to remove the flicker component from the external recording unit 114. Output to.

  The external recording unit 114 records the image signal flicker corrected by the first flicker correction unit 107 and the parallax signal flicker corrected by the second flicker correction unit 113 in an external memory.

  It is assumed that the recorded image signal and the parallax signal are subjected to image processing by a PC or the like to perform refocus processing such as changing the focus position after shooting, changing the shooting viewpoint, and adjusting the depth of field.

  The control unit 115 controls the entire imaging device 100.

  The configuration of the image pickup apparatus 100 according to the present invention has been described above.

  From here, the configuration of the unit pixel of the image sensor 102 and the method of reading the output signal will be described with reference to FIG. A method of reading a parallax signal of an image pickup device having one microlens and two PDs in a unit pixel and having the same configuration as that of the present embodiment has been described in detail in Japanese Patent Laid-Open Publication No. 2004-242242, and therefore will be briefly described here. Describe.

  The optical signals incident on the PDs 201a and 201b of the divided pixels a and b described above are photoelectrically converted, and charges corresponding to the exposure amount are accumulated. When the signals txa and txb applied to the gates of the transfer gates 202a and 202b are respectively set to High level, the charges accumulated in the PDs 201a and 201b are transferred to the floating diffusion (hereinafter, FD) unit 203. The FD unit 203 is connected to the gate of the FD amplifier 204, and the charge amount transferred from the PDs 201a and 201b is converted into a voltage amount by the FD amplifier 204. By setting the signal sel applied to the gate of the pixel selection switch 206 to High level, the pixel signal converted into a voltage by the FD amplifier 204 is output to the output vout of the unit pixel.

  Before reading the pixel signal, the signal res applied to the gate of the FD reset switch 205 is set to a high level in order to reset the FD unit 203. Further, when the charge accumulation is started, the signal res and the signals txa and txb are simultaneously set to the high level in order to reset the charges of the PDs 201a and 201b. As a result, both the transfer gates 202a and 202b and the FD reset switch 205 are turned on, and the PDs 201a and 201b are reset via the FD unit 203.

  When reading out a region where the parallax signal is unnecessary, the signals txa and txb are simultaneously set to the high level, and the charges accumulated in the PDs 201a and PD201b are combined and transferred to the FD unit 203 to be divided pixels a and b. The composite signal of is output.

  On the other hand, when reading out a region requiring a parallax signal, first, by keeping the signal txa at the high level and the signal txb at the low level, only the charges accumulated in the PD 201a are transferred to the FD unit 203, and the divided pixels are divided. The pixel signal of a is output. Next, the signal txb is also set to the high level, so that the charges accumulated in the PD 202b are combined with the charges accumulated in the PD 201a in the FD unit 203, and the combined signal of the divided pixels a and b is output.

  In the present embodiment, the refocusing unit is assumed to be limited to the refocusable region by limiting the parallax signal acquisition region. As in Japanese Patent Application Laid-Open No. 2004-242242, a focus detection technique represented by autofocus may be applied to a region designated as a target of focus detection using a parallax signal read from an image sensor.

  Next, a flicker detection method will be described from a configuration example of the first flicker detection unit 105 and the second flicker detection unit 108 with reference to FIG. The detailed configuration of the flicker detection is described in detail in Japanese Patent Laid-Open No. 2004-187, so it will be briefly described here.

  In FIG. 3, the block division unit 301 sets an area for detecting a flicker component (hereinafter, flicker detection frame) in a horizontal direction and a vertical direction with respect to an image signal for one screen. Here, flicker detection frames of M blocks in the horizontal direction and N blocks in the vertical direction are set. The averaging unit 302 calculates the average value of the luminance levels of the pixels for each of the flicker detection frames set by the block dividing unit 301. Further, the subtractor 303, the multiplier 304, the adder 305, and the memory 306 form a so-called cyclic low-pass filter 307 by performing the following calculation.

  Here, ave indicates the output of the averaging unit 302, and mout indicates the output from the memory 306. mem indicates the output from the adder 305, and indicates the value newly stored in the memory 306. Also, k is a filter coefficient of the cyclic low-pass filter 307.

  The divider 308 calculates and outputs a flicker component for each flicker detection frame by dividing the output of the averaging unit 302 and the output from the memory 306. As a method of converting the flicker component into the frequency domain, in the present embodiment, the data sampled for each flicker detection frame is converted into the frequency domain using the discrete Fourier transform.

  Here, in the present invention, a case in which the second flicker detection unit 108 can detect flicker will be described with reference to FIG. In FIG. 4, in order to simplify the description, the read time of one line is the same in the area where only the parallax signal is read from the image sensor 102 and in other areas.

  FIG. 4A shows the main subject 401 and the subject 402 in the captured image.

  FIG. 4B shows a state of the image signal when the image sensor 102 is read out in a certain accumulation time. It is shown that the flicker generated under a fluorescent lamp whose brightness fluctuates depending on the frequency of the AC power source appears in the form of light and dark in the vertical direction.

  FIG. 4C shows a state of the parallax signal when the image sensor 102 is read out at a certain storage time as in FIG. 4B. The read area Sa of the parallax signal for the captured image is used. This area Sa is a refocusable area in the image, and the subject 402 can be refocused.

  FIG. 4D shows flicker on each vertical line. The horizontal coordinate indicates the flicker signal level for the parallax signal. Further, the vertical coordinate indicates a read vertical line of the parallax signal for the captured image. In the image signal and the parallax signal, flicker of the same level is distributed on the corresponding vertical lines.

  FIG. 4E shows the threshold value T of the area where the parallax signal can be detected by the second flicker detection unit 108 and the flicker detection result. The threshold value T is determined by the flicker frequency of the image signal from the first flicker detection unit 105. For example, when the flicker frequency of the image signal is a sine wave and 50 Hz, the number of flicker-detectable samples requires at least the number of vertical lines in the first quadrant of the sine wave.

  If the reading time of one vertical line is Ys,

Indicates that the threshold value T is the area where the second flicker detection unit 108 can correct flicker. In the present embodiment, the above calculation formula is given as an example, but the present invention is not limited to this, and the threshold value T of the flicker component may be determined based on the control unit 115. In this case, the threshold value T may be set as a fixed value in a register (not shown) in the area determination unit 111 based on the control unit 115. In FIG. 4E, since the region Sa ≧ the threshold value T, the second flicker detection unit 108 satisfies the required number of vertical lines and flicker detection can be performed.

  Next, a case where the second flicker detection unit 108 cannot detect flicker will be described with reference to FIG.

  5 (a), 5 (b), and 5 (d) are similar to FIGS. 4 (a), 4 (b), and 4 (d), respectively.

  FIG. 5C shows a state of the parallax signal when the image sensor 102 is read out at a certain storage time as in FIG. 5B. An area Sb is a reading area of the parallax signal for the captured image. This area Sb is outside the refocusable area in the image, and cannot refocus the subject 402.

  Similar to FIG. 4E, FIG. 5E illustrates the threshold value T of the area where the parallax signal can be detected by the second flicker detection unit 108 and the flicker detection result. Since the region Sb <threshold T, the number of vertical lines required by the second flicker detection unit 108 is not satisfied, and a frequency different from the flicker frequency is erroneously detected, and flicker detection cannot be performed.

  Here, the operation of the flicker correction value determination unit 110 will be described with reference to the timing chart of FIG.

  FIG. 6 shows a vertical synchronization signal, input signals 1 and 2 of the flicker correction value selection unit 112, inputs from the control unit 115 of the area determination unit 111, output signals of the area determination unit 111, and output signals of the flicker correction value selection unit 112. Is shown. The horizontal direction shows time, and shows a timing chart of 2V of the vertical synchronizing signal.

  As the input signal 1 of the flicker correction value selection unit 112, the first flicker correction value from the first flicker correction value calculation unit 106 is shown. Also, as the input signal 2, the second flicker correction value from the second flicker correction value calculation unit 109 is shown.

  It is shown that the number of vertical lines of the parallax signal read from the image sensor 102 is input as an input signal from the control unit 115 of the area determination unit 111. The first line indicates that the number of vertical lines of the parallax signals in the region Sa and the second line that is in the region Sb are input.

  As the output signal of the area determination unit 111, it is indicated that the determination result based on whether or not the number of vertical lines of the parallax signal read from the image sensor 102 is equal to or larger than the threshold value T is output. If the determination result is greater than or equal to the threshold T, the signal level is High, and if less than the threshold T, the signal level is Low.

  As the output signal of the flicker correction value selection unit 112, the flicker correction value selection unit 112 selects the second flicker correction value based on the signal from the area determination unit 111 if the input signal level is High. Is shown. If the signal level is Low, it means that the first flicker correction value is selected.

  FIG. 6 shows a period in which the parallax signal is not read from the image sensor 102 as an invalid period and a period in which the parallax signal is read as a valid period. At the 1V, since the region Sa is equal to or larger than the threshold value T, the parallax signal is within the valid period. Is flicker corrected with the second flicker correction value. Further, at the 2V, since the area Sb is less than the threshold value T, the parallax signal is flicker corrected with the first flicker correction value within the effective period. Accordingly, the appropriate flicker correction value input to the second flicker correction unit 113 can be determined by the flicker correction value determination unit 110 according to the number of vertical lines of the parallax signals read from the image sensor 102.

  FIG. 7 is a flowchart showing a processing flow of flicker correction of a parallax signal of the image pickup apparatus 100 under the control of the control unit 115.

  In step S700, flicker detection is performed by the first flicker detection unit 105 and the second flicker detection unit 108 based on the image signal and the parallax signal from the capture unit 104.

  In step S701, the flicker correction value is calculated by the first flicker correction value calculation unit 106 and the second flicker correction value calculation unit 109 from the detected flicker component. The reciprocal of each detected flicker component is calculated as a flicker correction value.

  In step S <b> 702, with the number of vertical lines from which the parallax signal is read as the parallax signal region, the region determination unit 111 determines whether or not the region is equal to or greater than a threshold T corresponding to the frequency of flicker detected from the image signal.

  If the area is equal to or larger than the threshold value T in S703 (YES in S703), the process proceeds to S704. When the area is less than the threshold value T (NO in S703), the process proceeds to S705.

  When the process proceeds to S704, the second flicker correction value is selected as the input signal of the flicker correction value selection unit 112 in S704, and the process proceeds to S706.

Further, when the process proceeds to S705, the first flicker correction value is selected as the input signal of the flicker correction value selection unit 112 in S705, and the process proceeds to S706.
In step S706, the flicker correction value selected by the flicker correction value selection unit 112 is used to perform the flicker correction processing of the parallax signal, and the processing flow ends.

  According to the above-described embodiment, the configuration of the flicker correction value determination unit 110 allows appropriate flicker correction for a plurality of image pickup signals read from the image pickup device.

  In the embodiment, two types of image pickup signals, that is, an image signal and a parallax signal are described, but the present invention is not limited to this and may be applied to flicker correction of plural types of image pickup signals. For example, in an image processing technique using four types of parallax signals, each unit pixel having one microlens and four photodiodes arranged in a square shape, the flicker correction of the present invention is applied to each parallax signal. May be adapted.

  In the embodiment, the reading time is set to the same timing for the image signal and the line designated by the parallax signal, but the timing is not limited to this and may be different timings. In this case, the parallax signal may be flicker-corrected with the flicker correction value obtained by shifting the first flicker correction value by the phase difference required for the reading time.

  Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

100 image pickup apparatus, 101 image pickup optical system, 102 image pickup element, 103 A / D converter,
104 capture unit, 105 first flicker detection unit,
106 first flicker correction value calculation unit, 107 first flicker correction unit,
108 second flicker detection unit, 109 second flicker correction value calculation unit,
110 flicker correction value determination unit, 111 region determination unit,
112 flicker correction value selection unit, 113 second flicker correction unit,
114 external recording unit, 115 control unit

Claims (3)

A first input means for inputting a first image pickup signal;
Second input means for inputting a second image pickup signal obtained by partially and simultaneously photographing the same subject as the subject imaged with the first image pickup signal,
First flicker detection means capable of detecting a flicker component of the first image pickup signal,
Second flicker detection means capable of detecting a flicker component of the second image pickup signal,
First flicker correction value calculation means capable of calculating a first flicker correction value calculated from the flicker component of the first image pickup signal;
Second flicker correction value calculation means capable of calculating a second flicker correction value calculated from the flicker component of the second image pickup signal,
An area determination unit that determines whether the second image pickup signal can detect flicker according to the input range of the second image pickup signal;
Flicker correction value selection means for selecting either the first flicker correction value or the second flicker correction value based on the determination,
Flicker correction means for performing flicker correction on the second image pickup signal with the flicker correction value selected by the flicker correction value selection means,
It is characterized by including.
Flicker correction value selection means for selecting either the first flicker correction value or the second flicker correction value based on the determination,
A flicker correction unit that performs flicker correction on the second image pickup signal with the flicker correction value selected by the flicker correction value selection unit;
An image processing apparatus comprising:   The image according to claim 1, wherein the area determination unit determines whether the flicker component of the second image pickup signal can be corrected based on the flicker component detected from the first image pickup signal. Processing equipment.   The flicker correction value selection means inputs the flicker correction value calculated by the first image pickup signal when there is a difference in input time between the input of the first image pickup signal and the input of the second image pickup signal. The image processing apparatus according to claim 1, wherein flicker correction can be performed on the second image pickup signal with a phase corresponding to a time difference.