JP2018019355A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

Kind Code: A1 An interpolation method is provided for a photoelectric conversion unit that outputs an abnormal signal in an imaging device having a plurality of pixels having a plurality of photoelectric conversion units for one microlens. SOLUTION: An imaging device has a first photoelectric conversion region including a photoelectric conversion unit that outputs an abnormal signal and a second photoelectric conversion region that includes a photoelectric conversion unit that outputs a positive signal. Interpolating means for interpolating the target pixel, the interpolation means determining the position of the first photoelectric conversion region in the target pixel for each of a plurality of peripheral pixels positioned around the target pixel. and a first value obtained by weighting and averaging the signals of the third photoelectric conversion region corresponding to and the second photoelectric conversion regions in the pixel to be processed of each of the plurality of peripheral pixels located around the pixel to be processed. A ratio between the signal of the fourth photoelectric conversion region corresponding to the position and a second value obtained by weighted averaging is obtained, and interpolation is performed using this ratio. [Selection drawing] Fig. 8

Description

  The present invention relates to a technique for interpolating a signal read from a photoelectric conversion unit having a defect in an imaging device including a plurality of pixels each having a plurality of photoelectric conversion units.

  An image sensor having a plurality of pixels each having a plurality of photoelectric conversion units is known for one microlens. In each pixel of the image sensor, signals are read independently from some photoelectric conversion units of the plurality of photoelectric conversion units and other photoelectric conversion units, and the phase difference of the independently read signals is obtained, Distance information to the subject can be obtained. When some of the photoelectric conversion units of the plurality of photoelectric conversion units are defective, or when a circuit connected to the partial photoelectric conversion unit is defective, a signal read from the photoelectric conversion unit Cannot be used for phase difference detection as it is.

  Therefore, in Patent Document 1, in an imaging device including a plurality of pixels each having a plurality of photoelectric conversion units for one microlens, the signals of the photoelectric conversion units of the peripheral pixels are detected with respect to signals of the photoelectric conversion units that are not normal. A method of interpolating using signals is disclosed.

Japanese Patent Laying-Open No. 2015-56710

  However, in the interpolation method described in Patent Document 1, a signal from a photoelectric conversion unit included in a pixel corresponding to the same color filter as a pixel including a photoelectric conversion unit that outputs an abnormal signal (hereinafter referred to as a defective pixel). Therefore, the degree of freedom of pixels that can be used for interpolation is low. For example, in the case of an image sensor configured in a Bayer array, there are only every other pixel corresponding to the same color filter in the horizontal and vertical directions. For this reason, the interpolation method described in Patent Document 1 cannot use signals read from the photoelectric conversion units of pixels adjacent in the horizontal and vertical directions when performing interpolation.

  As the signal used for interpolation, the reproducibility of the high-frequency component at the time of interpolation is improved when a signal read from a spatially close position is used. Therefore, although depending on the pattern of the subject, in order to further improve the reproducibility of the high-frequency component during interpolation, the position of the pixel including the photoelectric conversion unit that refers to the signal at the time of interpolation is determined by the photoelectric conversion to be interpolated. It is desirable that the position of the pixel of the same color as the pixel including the portion is not limited.

  Furthermore, the case where the signal of the photoelectric conversion unit itself that reads out the signal used for the interpolation is not normal or saturated may be considered. Alternatively, depending on the pattern of the subject, even a photoelectric conversion unit included in a nearby pixel may output a signal related to a subject other than the photoelectric conversion unit to be interpolated.

  Therefore, in order to further improve the accuracy of the signal obtained by interpolation, the degree of freedom of the position of the pixel including the photoelectric conversion unit that refers to the signal at the time of interpolation is improved, and the photoelectric conversion unit that is referred to according to the situation It is desirable to provide an appropriate priority for the signal read from the.

  The present invention includes a plurality of pixels having a plurality of photoelectric conversion regions with respect to one microlens, and each of the plurality of photoelectric conversion regions includes one or more photoelectric conversion units; A pixel to be processed having a first photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as a signal and a second photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as normal; Interpolating means for performing first interpolation on the first photoelectric conversion region, and the interpolating means includes a plurality of peripheral pixels positioned around the pixel to be processed, A first value obtained by performing weighted averaging using a coefficient for a signal in the third photoelectric conversion region corresponding to the position of the first photoelectric conversion region in the pixel, and the periphery of the pixel to be processed Multiple peripherals located in The second value obtained by weighted averaging the signals of the fourth photoelectric conversion regions corresponding to the positions of the second photoelectric conversion regions in the pixels to be processed using the coefficients. The imaging device is characterized in that the first interpolation is performed using the signal of the second photoelectric conversion region and the ratio.

  According to the present invention, in an image pickup apparatus having an image pickup device including a plurality of pixels each having a plurality of photoelectric conversion units for one microlens, a signal with high accuracy is obtained with respect to a photoelectric conversion unit that outputs an abnormal signal. Can be interpolated.

It is a block diagram which shows the structure of the imaging device in embodiment of this invention. It is a figure for demonstrating the structure of the image pick-up element in embodiment of this invention. It is an equivalent circuit diagram which shows a part of pixel array in embodiment of this invention. It is a figure which shows the timing of the reading of the signal in embodiment of this invention. It is a figure which shows the interpolation of the signal of the pixel which has two PD and the color filter of G hue in embodiment of this invention. It is a figure which shows the interpolation of the signal of the pixel which has two color filters and color filter of the hue of R in embodiment of this invention. It is a figure which shows the interpolation of the signal of the pixel which has four PD and the color filter of the hue of G in embodiment of this invention. It is a flowchart for demonstrating embodiment of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the following embodiment is only an example for implementing this invention, and this invention is not limited to the structure shown in figure.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. In the figure, an image sensor 100 photoelectrically converts an optical image of a subject formed by an imaging optical system into an electrical signal. The image sensor 100 is controlled by a CPU (Central Processing Unit) 103, which will be described later, and captures a still image or a moving image. As will be described later, the image sensor is covered with a color filter composed of a plurality of colors. An AFE (analog front end) 101 performs digital conversion on an analog image signal output from the image sensor 100 in accordance with gain adjustment or a predetermined quantization bit. A TG (timing generator) 102 controls the drive timing of the image sensor 100 and the AFE 101. In the present embodiment, the AFE 101 and the TG 102 are arranged outside the image sensor 100, but they may be configured to be built in the image sensor.

  As described above, the CPU 103 executes a program for controlling each unit of the image sensor. The operation unit 104 sets an imaging command, imaging conditions, and the like for the CPU 103. The display unit 105 is a display that displays captured still images, moving images, menus, and the like. The RAM 106 has a function of an image data storage unit that stores image data digitally converted by the AFE 101 and image data processed by an image processing unit 108 described later, and a function of a work memory when the CPU 103 operates. . In this embodiment, these functions are performed using the RAM 106, but other memories may be applied as long as the access speed is sufficiently high and there is no problem in operation. Is possible. The ROM 107 stores a program that the CPU 103 loads and executes in order to control the operation of each unit. Information such as a photoelectric conversion unit and a signal saturation level, which are defects related to the image sensor 100, is also stored in the ROM 107 from the time of factory shipment. In addition, the photoelectric conversion part which is a defect here includes not only a part having a defect in the configuration itself of the photoelectric conversion part but also a part having a defect in a circuit connected to the photoelectric conversion part. That is, a photoelectric conversion unit that is regarded as a signal that is output as a result is regarded as a defective photoelectric conversion unit. Here, in the present embodiment, a flash memory is shown, but this is only an example, and other memories can be applied as long as the access speed is sufficiently high and there is no problem in operation. . The image processing unit 108 performs processing such as correction and compression of a captured still image or moving image. In addition, a function for separating A image data and B image data, a function for correcting images, and a function for generating still images and moving images, which will be described later, are provided.

  The AF calculation unit 109 calculates the driving amount of the focus lens using the correlation calculation result output from the correlation calculation unit 120. The flash memory 110 is a detachable flash memory for recording still image data and moving image data. In this embodiment, a flash memory is applied as a recording medium, but other data writable nonvolatile memory may be used. Moreover, the form which incorporated these memories may be sufficient.

  The focal plane shutter 111 adjusts the exposure time when capturing a still image. In this embodiment, the exposure time of the image sensor 100 is adjusted by the focal plane shutter 111. However, the present invention is not limited to this. The image sensor 100 has an electronic shutter function, and the exposure time is controlled by a control pulse. The structure which adjusts time may be sufficient. The focus drive circuit 112 changes the focal position of the optical system, controls the drive of the focus actuator 114 based on the focus detection result of the AF calculation unit 109, and drives the third lens 119 forward and backward in the optical axis direction. Adjust the focus. The aperture driving circuit 113 controls the aperture diameter of the aperture 117 by controlling the driving of the aperture actuator 115. The first lens 116 is disposed at the tip of the imaging optical system (common optical system) and is held so as to be able to advance and retreat in the optical axis direction. The diaphragm 117 adjusts the light amount at the time of imaging by adjusting the aperture diameter. The diaphragm 117 and the second lens 118 integrally move forward and backward in the optical axis direction, and realize a zooming function (zoom function) in conjunction with the forward and backward movement of the first lens 116. The third lens 119 adjusts the focal point of the imaging optical system by advancing and retreating in the optical axis direction. The correlation calculation unit 120 performs correlation calculation using the pixel signal output from the image sensor 100.

  FIG. 2 is a diagram for explaining the configuration of the image sensor 100. FIG. 2A shows the configuration of the image sensor 100. In FIG. 2A, the imaging device includes a pixel array 100a in which pixels are two-dimensionally arranged, a vertical scanning circuit 100d that selects a row of pixels in the pixel array 100a, and a horizontal that selects a column of pixels in the pixel array 100a. It has a scanning circuit 100c. The image sensor 100 further includes a readout circuit 100b for reading out signals of pixels selected by the vertical scanning circuit 100d and the horizontal scanning circuit 100c among the pixels of the pixel array 100a. The vertical scanning circuit 100d selects a row of the pixel array 100a, and validates a readout pulse output from the TG 102 based on the horizontal synchronization signal output from the CPU 103 in the selected row. The readout circuit 100b includes an amplifier and a memory provided for each column, and stores the pixel signal of the scanning row in the memory via the amplifier. The pixel signals for one row stored in the memory are sequentially selected in the column direction by the horizontal scanning circuit 100c and output to the outside through the output circuit 100e. This operation is repeated to output all pixel signals to the outside.

  A pixel array 100a of the image sensor 100 is shown in FIG. In FIG. 2 (b), the microlens array includes microlenses 100f arranged in h in the horizontal direction and v in the vertical direction. PDs (photodiodes) 100h and 100g constitute an A image photoelectric conversion unit and a B image photoelectric conversion unit, which will be described later, as photoelectric conversion units that perform photoelectric conversion. Each pixel has a configuration in which one microlens 100f is disposed on the top of two PDs. That is, each pixel includes a plurality of photoelectric conversion units for one microlens. When one pixel is constituted by two PDs sharing the microlens 100f, the pixel array 100a includes h pixels in the horizontal direction and v pixels in the vertical direction. Signals accumulated in the PDs 100h and 100g are converted into voltages simultaneously or independently by a pixel transfer operation described later, and output to the outside by the above-described readout operation. The PD 100h and the PD 100g have a pupil division configuration, and separate images having a phase difference from each other are incident. Therefore, the signals of the PDs 100h and PD100g can be read independently, the correlation calculation unit 120 can perform correlation calculation processing, and the AF calculation unit 109 can calculate the driving amount of the focus lens using the result. Here, PD100h is an A image photoelectric conversion unit, and PD100g is a B image photoelectric conversion unit. In FIG. 2B, two PDs are arranged for one microlens, but the present invention is not limited to this. The present invention can be applied to a configuration in which a plurality of PDs are arranged vertically or horizontally with respect to one microlens, or the pupil can be divided into four or more. For example, FIG. 2C shows an example in which four PDs are arranged. The pixel 100i includes a PD 100j, a PD 100k, a PD 100m, and a PD 100n. Images having different phases are incident on the PD 100j, PD 100k, PD 100m, and PD 100n, and signals can be read out separately. Alternatively, depending on the situation, the signals of PD100j and PD100k are added and read out, and the signals of PD100m and PD100n are added and read out, so that the phase difference in the horizontal direction can be obtained through the calculation in the correlation calculation unit 120. . Alternatively, the phase difference in the vertical direction can be obtained through the calculation in the correlation calculation unit 120 by adding and reading the signals of PD100j and PD100m and adding and reading the signals of PD100k and PD100n. Thus, the signal obtained from the PD can be used for phase difference detection. FIG. 2D is a diagram illustrating a color filter of a microlens. One square in FIG. 2D means a color filter corresponding to one microlens. The letters “R”, “B”, and “G” written on the microlens represent the hue of the color filter, and correspond to “red”, “blue”, and “green”, respectively. In the present embodiment, as shown in FIG. 2D, the description will be made based on a Bayer color filter, but the present invention is not limited to this. For example, four hues of yellow, magenta, cyan, and green are 2 × 2. It can also be implemented using a color filter arranged.

  In the following, the embodiment will be described using a configuration in which two PDs are arranged for one microlens. FIG. 3 shows two adjacent rows (j rows and (j + 1) rows), two columns (i columns and (i + 1) columns), and two columns (a plurality of pixels provided in the pixel array 100a). It is an equivalent circuit diagram showing a configuration of the read circuit 100b for i columns and (i + 1) columns).

  The control signal ΦTXA (j) is input to the transfer switch 302a of the pixel 301 in the j-th row, and the control signal ΦTXB (j) is input to the gate of the transfer switch 302b. The reset switch 304 is controlled by a reset signal ΦR (j), and the row selection switch 306 is controlled by a row selection signal ΦS (j). Similarly, the pixels in the (j + 1) th row are controlled by control signals ΦTXA (j + 1) and ΦTXB (j + 1), a reset signal ΦR (j + 1), and a row selection signal ΦS (j + 1). Note that the control signals ΦTXA (j) and ΦTXB (j), the reset signal ΦR (j), and the row selection signal ΦS (j) are controlled by the vertical scanning circuit 100d.

  A vertical signal line 308 is provided for each pixel column, and each vertical signal line 308 is connected to a current source 307 and transfer switches 310a and 310b of the readout circuit 100b provided in each column.

  A control signal ΦTN is input to the gate of the transfer switch 310a, and a control signal ΦTS is input to the gate of the transfer switch 310b. The control signal ΦPH (i) output from the horizontal scanning circuit 100c is input to the gates of the transfer switch 312a and the transfer switch 312b. The storage capacitor unit 311a stores the output of the vertical signal line 308 when the transfer switch 310a is on and the transfer switch 312a is off. Similarly, the storage capacitor unit 311b stores the output of the vertical signal line 308 when the transfer switch 310b is on and the transfer switch 312b is off.

  By turning on the transfer switch 312a and the transfer switch 312b of the i-th column by the column selection signal ΦPH (i) of the horizontal scanning circuit 100c, the outputs of the storage capacitor unit 311a and the storage capacitor unit 311b are different horizontal outputs. It is transferred to the output circuit 100e via the line.

  As a read operation for reading a signal from the image sensor 100 having the above configuration, an addition read operation (first read operation) and a divided read operation (second read operation) can be selectively performed. Hereinafter, the addition reading operation and the divided reading operation will be described. In the present embodiment, it is assumed that each switch is turned on when each control signal is in an H (high) state and turned off when each control signal is in an L (low) state.

<Addition read operation> (first read operation)
FIG. 4A shows the timing of an operation for reading a signal from the pixel in the j-th row of the image sensor 100 by the addition reading operation. At time T1, the reset signal ΦR (j) becomes H. Next, when the control signals ΦTXA (j) and ΦTXB (j) become H at time T2, the PDs 100h and 100g of the pixel 100f in the j-th row are reset.

  Next, when the control signals ΦTXA (j) and ΦTXB (j) become L at time T3, the PDs 100h and 100g start to accumulate charges. Subsequently, when the row selection signal ΦS (j) becomes H at time T4, the row selection switch 306 is turned on and connected to the vertical signal line 308, and the source follower amplifier 305 is in an operating state.

  Next, when the reset signal ΦR (j) is set to L at time T5 and then the control signal ΦTN is set to H at time T6, the transfer switch 310a is turned on, and the signal after reset release on the vertical signal line 308 ( Noise signal) is transferred to the storage capacitor 311a.

  Next, at time T7, the control signal ΦTN is set to L, and the noise signal is held in the storage capacitor portion 311a. Thereafter, when the control signals ΦTXA (j) and ΦTXB (j) become H at time T8, the charges of the PDs 100h and 100g are transferred to the FD region (floating diffusion region) 303. At this time, since the charges of the two PDs 100h and 100g are transferred to the same FD region 303, a signal in which the charges of the two PDs 100h and 100g are mixed (an optical signal for one pixel + noise signal) is transmitted to the vertical signal line 308. Is output.

  Subsequently, at time T9, the control signals ΦTXA (j) and ΦTXB (j) are set to L. Thereafter, when the control signal ΦTS becomes H at time T10, the transfer switch 310b is turned on, and a signal on the vertical signal line 308 (an optical signal for one pixel + a noise signal) is transferred to the storage capacitor 311b. Next, the control signal ΦTS is set to L at time T11, and the optical signal + noise signal for one pixel is held in the storage capacitor 311b, and then the row selection signal ΦS (j) is set to L at time T12.

  Thereafter, the transfer switches 312a and 312b are sequentially set to H from the first pixel column to the final pixel column by the column selection signal ΦPH of the horizontal scanning circuit 100c. As a result, the noise signal of the storage capacitor unit 311a and the light signal + noise signal for one pixel of 311b are transferred to the output circuit 100e via different horizontal output lines. The output circuit 100e calculates a difference (optical signal for one pixel) between the two horizontal output lines and outputs a signal obtained by multiplying the difference by a predetermined gain. Hereinafter, the signal obtained by the above-described addition reading operation is referred to as a “first addition signal”.

<Division read operation> (second read operation)
Next, the divided read operation will be described with reference to FIG. FIG. 4B shows the timing of an operation for reading a signal from the pixel in the j-th row of the image sensor 100 by the divided readout operation. At time T1, the reset signal ΦR (j) is set to H. Subsequently, when ΦTXA (j) and ΦTXB (j) become H at time T2, the PDs 100h and 100g of the pixels 301 in the j-th row are reset. Next, when the control signals ΦTXA (j) and ΦTXB (j) become L at time T3, the PDs 100h and 100g start to accumulate charges. Subsequently, when the row selection signal ΦS (j) becomes H at time T4, the row selection switch 306 is turned on and connected to the vertical signal line 308, and the source follower amplifier 305 is in an operating state.

  After the reset signal ΦR (j) is set to L at time T5, when the control signal ΦTN becomes H at time T6, the transfer switch 310a is turned on, and a signal (noise signal) after the reset is released on the vertical signal line 308. Is transferred to the storage capacitor unit 311a.

  Next, when the control signal ΦTN is set to L at time T7 and the noise signal is held in the storage capacitor portion 311a, and then ΦTXA (j) becomes H at time T8, the charge of the PD 100h is transferred to the FD region 303. . At this time, one of the two PDs 100h and 100g (here, PD100h) is transferred to the FD region 303, so that only a signal corresponding to the charge of the PD 100h is output to the vertical signal line 308.

  Next, after the control signal ΦTXA (j) is set to L at time T9, when the control signal ΦTS becomes H at time T10, the transfer switch 310b is turned on, and the signal on the vertical signal line 308 (for 1 PD) Light signal + noise signal) is transferred to the storage capacitor 311b. Next, the control signal ΦTS is set to L at time T11.

  Thereafter, the transfer switches 312a and 312b are sequentially set to H from the first pixel column to the final pixel column by the column selection signal ΦPH of the horizontal scanning circuit 100c. As a result, the noise signal of the storage capacitor unit 311a and the light signal + noise signal for 1PD of 311b are transferred to the output circuit 100e through separate horizontal output lines. The output circuit 100e calculates a difference (optical signal for 1 PD) between the two horizontal output lines and outputs a signal obtained by multiplying the difference by a predetermined gain. Hereinafter, a signal obtained by the above-described reading operation is referred to as a “divided signal”.

  Thereafter, at time T12, ΦTXA (j) and ΦTXB (j) become H, and in addition to the charge of the previously transferred PD100h, the charge of the PD100g and the newly generated charge of the PD100h are transferred to the FD region 303. . At this time, since the charges of the two PDs 100h and 100g are transferred to the same FD region 303, a signal in which the charges of the two PDs 100h and 100g are mixed (an optical signal for one pixel + noise signal) is transmitted to the vertical signal line 308. Is output.

  Subsequently, after the control signals ΦTXA (j) and ΦTXB (j) are set to L at time T13, when the control signal ΦTS becomes H at time T14, the transfer switch 310b is turned on. As a result, the signal on the vertical signal line 308 (an optical signal for one pixel + noise signal) is transferred to the storage capacitor 311b.

  Next, the control signal ΦTS is set to L at time T15, and the optical signal + noise signal for one pixel is held in the storage capacitor 311b, and then the row selection signal ΦS (j) is set to L at time T16.

  Thereafter, the transfer switches 312a and 312b are sequentially set to H from the first pixel column to the final pixel column by the column selection signal ΦPH of the horizontal scanning circuit 100c. As a result, the noise signals of the storage capacitors 311a and 311b and the light signal plus noise signal for one pixel are transferred to the output circuit 100e through different horizontal output lines. The output circuit 100e calculates a difference (optical signal for one pixel) between the two horizontal output lines and outputs a signal obtained by multiplying the difference by a predetermined gain. Hereinafter, the signal obtained by the read operation is referred to as a “second addition signal” in order to distinguish it from the first addition signal.

  A divided signal corresponding to the other PD 100g can be obtained by subtracting the divided signal corresponding to one PD 100h from the second addition signal read in this way. The pair of divided signals obtained in this way are called “focus detection signals”. Then, by performing a known correlation operation on the obtained focus detection signal, the phase difference between the signals can be calculated.

  The divided readout operation and the addition readout operation can be switched as necessary for the same pixel. However, in this embodiment, it is assumed that a divided readout operation is performed on a pixel to be interpolated and a peripheral pixel to be described later.

  In addition, after performing a series of operations such as reset, charge accumulation, and signal readout on the PD 100h, the same operation is performed on the PD 100g, so that two PDs 100h for one charge accumulation operation. , 100 g of signals may be read independently. The signals of PDs 100h and 100g read out in two steps in this way can be added to obtain a second addition signal. In addition, as described above, the configuration is not limited to the configuration in which two PDs are arranged for one microlens, and even in a configuration having three or more PDs, the same concept can be used multiple times. It is possible to read the signal separately.

<Interpolation method of signal read from defective photoelectric conversion unit>
Next, a method for interpolating signals read from defective photoelectric conversion units in the present embodiment will be described. FIG. 5 is a diagram illustrating a case where two PDs are arranged for one microlens and only one PD of one pixel has a defect. Here, since the PD 501 arranged on the left side of the pixel 500 has a defect and a correct signal cannot be acquired, the signal of the PD 501 becomes an interpolation target. In contrast, it is assumed that the PD 502 arranged on the right side of the same pixel 500 has no defect. Note that the characters written on the top of each pixel represent the hue of the color filter.

  The image processing unit 108 can interpolate the signal of the PD 501 by using the ratio of the result of the weighted average of the signals of the two photoelectric conversion units of each of the peripheral pixels according to the following (Equation 1).

  In the above (Equation 1), S501 means the signal value of PD501 obtained by interpolation, and w1, w2, w3, and w4 represent weighting coefficients. The image processing unit 108 includes two PD signal values S511 to S542 included in each of the peripheral pixels 510 to 540 arranged on the top, bottom, left, and right of the pixel 500 having the PD 501 and a signal from the other PD 502 included in the pixel 500. Interpolation is performed using the value S502. Specifically, in each of the peripheral pixels 510 to 540, a value obtained by multiplying the signal values S511, S521, S531, and S541 of the PD511, PD521, PD531, and PD541 on the left side in the same manner as the PD501 by applying a weighting coefficient is obtained. Further, in each of the peripheral pixels 510 to 540, a value obtained by multiplying the signal values S512, S522, S532, and S542 of the PD 512, PD 522, PD 532, and PD 542 on the right side opposite to the PD 501 by applying a weighting coefficient is obtained. . Then, a ratio between a value obtained by weighted averaging using the signal value of the left PD and a value obtained by weighted averaging using the signal value of the right PD is calculated. This ratio corresponds to an average ratio of the left PD and the right PD in a region including a plurality of pixels with the pixel 500 as the center. Then, a value obtained by multiplying the calculated ratio by the signal value of the right PD 502 of the pixel 500 that is considered to output a normal signal is set as an interpolation value of the signal of the PD 501 on the left side of the pixel 500.

  Here, the weighting factors w1 to w4 have a role of improving the accuracy of interpolation by changing the values according to the state of the pixels around the pixel 500 and the subject. For example, when a pixel having a defect is included in a peripheral pixel, if the above ratio is obtained using a PD signal value obtained from the pixel, a value deviating from an average ratio in the peripheral pixel There is a case. Therefore, the image processing unit 108 sets the weighting coefficient to be given to 0 based on the defect information stored in the ROM 107 to be given to the signal value obtained from the peripheral pixels determined to have a defect. For example, when at least one of PD 511 and PD 512 included in the pixel 500 is defective, the weighting coefficient w1 for the signal values S511 and S512 obtained from the PD 511 and PD 512 is set to zero.

  Alternatively, when the signal value read from any PD reaches the saturation level stored in advance in the ROM 107, the weighting coefficient corresponding to the signal value obtained from the pixel including this PD is set to 0. . If the signal value read from at least one PD is saturated, the signal value read from the PD is smaller than the signal value corresponding to the original subject and is not a correct value. This is because it cannot be used for calculation of a high ratio.

  Thus, according to the above-described method, first, the signal value ratio between a plurality of PDs corresponding to the same microlens is obtained using the peripheral pixels of the pixel to be processed. Then, by multiplying this by a signal value obtained from another PD corresponding to the same microlens as the defective PD, the signal value of the defective PD is interpolated. Since the ratio of signal values between PDs corresponding to the same microlens is due to the difference in the area on the subject incident on the PD, when obtaining this ratio, the hue is different from that of the pixel to be processed. Corresponding pixels can be used. Therefore, as shown in (Expression 1), it is possible to obtain the ratio of signal values between PDs using pixels adjacent to the pixel to be processed and corresponding to a hue different from that to be processed.

  Of course, as shown in (Formula 2) below, the image processing unit 108 is obtained by weighted averaging of signal values obtained from two PDs included in the same color pixel as the pixel to be processed among the peripheral pixels. It is also possible to interpolate defective PD signal values using the ratio of values.

  In the above (Expression 2), when the pixel 500 to be interpolated corresponds to the hue G, interpolation is performed using a signal of a pixel having a color filter of the same hue G as the pixel 500. Since the pixels corresponding to the hue G have a higher density than the pixels corresponding to the hues R and B, even when the same color pixels are used, the distance between the pixel to be interpolated and the surrounding pixels is close, and the accuracy is high. A high ratio can be obtained.

  Next, FIG. 6 shows that the PD 601 on the left side of the pixel having the color filter of R hue is defective. The image processing unit 108 can interpolate the signal of the PD 601 using the following (Expression 3) or (Expression 4).

  In the above (Equation 3), S601 means a PD601 signal obtained by interpolation, and w1 to w4 represent weighting coefficients. In each of the peripheral pixels 610 to 640 arranged on the top, bottom, left, and right of the pixel 600 having the PD 601, a value obtained by multiplying the signal values S 611 to S 644 of the PDs 611 to 641 on the left side like the PD 601 by applying a weighting coefficient is obtained. . Further, in each of the peripheral pixels 611 to 641, a value obtained by multiplying the signal values S612 to S642 of the PDs 612 to 642 on the right side opposite to the PD 601 by a weighting coefficient is obtained. Then, a ratio between a value obtained by weighted averaging using the signal value of the left PD and a value obtained by weighted averaging using the signal value of the right PD is calculated. Then, a value obtained by multiplying the calculated ratio by the signal S602 of the PD 602 on the right side of the pixel 600 is set as an interpolation value of the signal of the PD 601 on the left side of the pixel 600.

  Similarly, as in the above (Expression 4), the image processing unit 108 uses signal values obtained from two PDs included in the pixels 650, 660, 670, and 680 having the same color as the pixel to be processed among the peripheral pixels. It is also possible to interpolate a signal value of a defective PD using a ratio of values obtained by weighted averaging.

  Furthermore, instead of performing interpolation using either one of (Equation 3) or (Equation 4), both may be used. That is, in each of the peripheral pixels 610 to 680, a value obtained by multiplying the signal value of the PDs 611 to 681 on the left side like the PD 601 by multiplying by the weighting coefficient is obtained. Further, in each of the peripheral pixels 610 to 680, a value obtained by multiplying the signal value of the PDs 612 to 682 on the right side opposite to the PD 601 by a weighting coefficient is obtained. Then, a ratio between a value obtained by weighted averaging using the signal value of the left PD and a value obtained by weighted averaging using the signal value of the right PD is calculated. Then, a value obtained by multiplying the calculated ratio by the signal value of the PD 602 on the right side of the pixel 600 is set as an interpolation value of the signal of the PD 601 on the left side of the pixel 600.

  Note that the image processing unit 108 may be configured to select (Equation 3) and (Equation 4) depending on the situation. For example, an area of 5 pixels × 5 pixels is set as the center of the pixel to be interpolated, and each color filter is used by using a signal value read from a normal photoelectric conversion unit corresponding to each color filter. The average value of the signal values corresponding to is obtained. If the average value of the signal values corresponding to any one of the color filters is larger than the average value of the signal values corresponding to the other color filters, exceeding a preset threshold value, (Expression 4) Otherwise, (Equation 3) is used.

  This is because if only the signal value of a specific color filter is large, the accuracy is likely to deteriorate if interpolation is performed using a signal value of a pixel corresponding to a color filter different from the pixel to be corrected. Because it becomes. For example, consider a case where only the average value of the signal values corresponding to the hue R is large and the average value of the signal values corresponding to the hue G and the hue B is small. Since hue S has a large S / N ratio, the signal value ratio between the left and right PDs can be obtained almost without being affected by noise components, but hue G and hue B have a small S / N ratio. For this reason, the noise component greatly affects the signal value ratio between PDs. In such a case, in order to interpolate the signal of the photoelectric conversion unit included in the pixel corresponding to the hue R, the signal value ratio between the PDs obtained from the peripheral pixels corresponding to the hue R may be used. desirable. Conversely, if the average values of the signal values corresponding to the respective hues are similar, the degree of the influence of the noise component does not vary greatly depending on the hues, and thus obtained from the peripheral pixels corresponding to the hues G and B. A ratio of signal values between PDs may be used.

  Further, the above-described weighting coefficient may be changed not only by the presence / absence of defects and saturation, but also by changes in the brightness and hue of the subject. Peripheral pixels that are referred to during interpolation output signal values corresponding to very close regions on the subject, so if there is no change in the luminance or color of the subject, the ratio of the left and right PD signal values in the peripheral pixels is similar. Should be. On the other hand, if the ratio of the signal values of the left and right PDs in any of the peripheral pixels is significantly different from that of the other pixels, there is a high possibility that the pixel is in a region where the luminance or hue of the subject changes.

  Accordingly, the image processing unit 108 first calculates the ratio of the signal values of the two PDs of each of the peripheral pixels 510, 520, 530 and 540 for the image sensor shown in FIG. That is, the ratios R510, R520, R530, and R540 of the following (Formula 5) are calculated.

  Then, the above ratios R510, R520, R530, and R540 are compared with each other, a small weighting coefficient is set for a pixel with a large deviation, and a signal value after interpolation is calculated using (Equation 1).

  The above setting method can be further modified as follows. The ratios R510 and R520 of the signal values of the two PDs of the symmetrical pixel 510 and pixel 520 above and below the pixel 500 to be interpolated are calculated. A difference | R510−R520 | between the two ratios is calculated. If this difference is a value equal to or greater than a predetermined threshold value, it is determined that the change in luminance and hue of the subject in the vertical direction is large, and the weighting factors w1 and w2 are set to zero.

  Similarly, the ratios R530 and R540 of the signal values of the two PDs of the pixels 530 and 540 above and below the pixel 500 to be interpolated are calculated, and the ratio difference | R530−R540 | is also calculated. . If this difference is equal to or greater than a predetermined threshold, it is set to 0 together with the weighting factors w3 and w4. If | R510-R520 | and | R510-R520 | are both equal to or greater than the threshold value, the weighting factors w1 to w4 are all set to 1. Alternatively, the weighting factors w1 to w4 are set so as to be inversely proportional to the ratio of the sizes of | R510-R520 | and | R510-R520 |. That is, if | R510-R520 | is smaller, the weighting factors w1 and w2 corresponding to the pixels R510 and R520 may be larger than the weighting factors w3 and w4. On the contrary, if | R510-R520 | is larger, the weighting factors w1 and w2 may be smaller than the weighting factors w3 and w4.

  As described above, by using the weighting coefficient in the calculation of interpolation, a smaller weighting coefficient can be given to data of a signal with low reliability, and a more accurate signal of a defective pixel can be obtained by interpolation.

  The above description is based on a configuration in which two PDs are arranged for one microlens. However, the present invention is not limited to this. For example, the present embodiment can be implemented even in a configuration in which four PDs are arranged for one microlens. FIG. 7 shows a configuration in which four PDs are arranged for one microlens. Hereinafter, processing when the interpolation method according to the present embodiment is used will be described using the configuration shown in FIG. In FIG. 7, it is assumed that the PD 701 has a defect.

  In the configuration shown in FIG. 7, as described above, for example, when detecting a horizontal phase difference in the pixel 700, the signals of PD 701 and PD 702 are added and read, and the signals of PD 703 and PD 704 are added. Need to be read. Here, a region including PD 701 and PD 702 is referred to as a first photoelectric conversion region, and a region including PD 703 and PD 704 is referred to as a second photoelectric conversion region. Since the PD 701 has a defect, when the signals of the PD 701 and the PD 702 are added and read, a correct signal as the first photoelectric conversion region cannot be obtained.

  Also in the peripheral pixel 710, when detecting a horizontal phase difference, it is necessary to add and read the signals of PD711 and PD712 and read the signals of PD713 and PD714. Here, PD 711 and PD 712 at positions corresponding to the positions of the first photoelectric conversion regions in the pixel 700 are referred to as third photoelectric conversion regions. Similarly, PD713, PD714, and a fourth photoelectric conversion region at positions corresponding to the positions of the second photoelectric conversion regions in the pixel 700 are referred to. Also in the pixels 720 to 740, the photoelectric conversion unit at the position corresponding to the position of the first photoelectric conversion region in the pixel 700 is the third photoelectric conversion region, and the position corresponding to the position of the second photoelectric conversion region in the pixel 700. The photoelectric conversion unit located in is referred to as a fourth photoelectric conversion region.

  Following the element shown in FIG. 5, the image processing unit 108 can interpolate the signals from the PD 701 and the PD 702 using the following (Equation 6).

  However, A1 is a value obtained by adding S711 and S712, A2 is a value obtained by adding S721 and S722, A3 is a value obtained by adding S731 and S732, and A4 is a value obtained by adding S741 and S742. Similarly, B1 is a value obtained by adding S713 and S714, B2 is a value obtained by adding S723 and S724, B3 is a value obtained by adding S733 and S734, and B4 is a value obtained by adding S743 and S744.

  In the method shown in (Equation 6), in the pixel 700 having the defective PD 701, the signals of the PD 701 and the PD 702 are added and read. In the other pixels, similarly, the two PD signals on the left side are added and read, and the two PD signals on the right side are added and read. In each of the peripheral pixels 710 to 740, a weighted average value is obtained by multiplying the two left PD signals by a weighting coefficient. Further, in each of the peripheral pixels 710 to 740, a weighted average value is obtained by multiplying the right two PD signals by a weighting coefficient. Then, a ratio between a value obtained by weighted averaging using the signal values of the two left PDs and a value obtained by weighted averaging using the signal values of the two right PDs is calculated. Then, the sum of the signal values of PD 701 and PD 702 is obtained by multiplying the calculated ratio by the sum of the signal values of PD 703 and PD 704 on the right side of pixel 700. This value is used for detecting the phase difference in the horizontal direction.

  Similarly, when detecting the phase difference of the vertical method, it is necessary to add and read the signals of PD 701 and PD 703 and read the signals of PD 702 and PD 704 in the pixel 700. The same applies to the peripheral pixels. The image processing unit 108 can interpolate the signals of the PD 701 and the PD 703 using the following (Expression 7).

  However, C1 is a value obtained by adding S711 and S713, C2 is a value obtained by adding S721 and S723, C3 is a value obtained by adding S731 and S733, and C4 is a value obtained by adding S741 and S743. Similarly, D1 is a value obtained by adding S712 and S714, D2 is a value obtained by adding S722 and S724, D3 is a value obtained by adding S732 and S734, and D4 is a value obtained by adding S742 and S744.

  Here, if the signal values of the first to fourth photoelectric conversion regions are defined as SS1, SS2, SS3, and SS4, respectively, the above (formula 6) and (formula 7) are expressed as the following (formula 8). Can be rewritten.

  Where N is the number of peripheral pixels used for the calculation. By subtracting the defect-free PD signal value from the calculated first photoelectric conversion region signal value SS1, it is possible to obtain a defect-PD signal.

  A similar method can be used when interpolating using signals from peripheral pixels 750, 760, 770 and 780 of the same color.

  Note that, in the above-described imaging device in which one pixel has two PDs, a defective PD can be regarded as a first photoelectric conversion region, and another PD in the same pixel can be regarded as a second photoelectric conversion region. For example, in the image sensor shown in FIG. 5, the PD 501 can be regarded as a first photoelectric conversion region and the PD 502 can be regarded as a second photoelectric conversion region. Similarly, each of PDs 511, 521, 531 and 541 can be regarded as a third photoelectric conversion region, and each of PDs 512, 522, 532 and 542 can be regarded as a fourth photoelectric conversion region. Based on this interpretation, the above (Equation 8) can be applied even to an image sensor in which one pixel has two PDs.

  Further, even in the case of an image sensor having a pixel structure having more than four PDs for one microlens, interpolation can be performed following the above method.

  Interpolation of the first photoelectric conversion region including the defective photoelectric conversion unit using the signal level read from the second photoelectric conversion region in the same pixel as the defective photoelectric conversion unit described so far The method of performing is called first interpolation.

  This first interpolation cannot be used when a correct signal cannot be obtained from the second photoelectric conversion region. In such a case, it is necessary to use the second interpolation instead of the first interpolation.

  The second interpolation is an interpolation similar to the technique described in the background art.

  In step S808, the image processing unit 108 includes signals of the photoelectric conversion unit included in the plurality of peripheral pixels corresponding to the same color filter as the pixel including the defective photoelectric conversion unit and defective with respect to the microlens. A photoelectric conversion unit existing at the same position is selected.

  For example, when the image sensor shown in FIG. 6 is used, the signal value of the PD 601 can be obtained by the following (Equation 9). In (Equation 9), the interpolation value of the defective PD 601 is obtained by averaging the signal values of the PD 651 to PD 681 on the left side in the four peripheral pixels 650 to 680 peripheral pixels of the same color as the defective pixel 600, as in the PD 601. It is shown that.

  A weighted average may be used instead of the simple average of the signal values of PD651 to PD681.

  FIG. 8 is an example of a flowchart when the above interpolation method is used.

  In step S <b> 801, the CPU 103 reads information on the defective photoelectric conversion unit stored in the ROM 107. In step S802, the CPU 103 compares the defective photoelectric conversion unit read in step S801 with information on the defective photoelectric conversion unit, and determines whether the pixel being processed includes the defective photoelectric conversion unit. If the pixel being processed is not a pixel including a defective photoelectric conversion unit, the process proceeds to step S810. If the pixel being processed is a pixel including a defective photoelectric conversion unit, the process proceeds to step S803.

  In step S803, the CPU 103 further determines whether or not a normal photoelectric conversion region necessary for interpolating a defective photoelectric conversion unit signal exists in the same pixel. If there is a normal photoelectric conversion region necessary for interpolation, the process proceeds to step S804, and the CPU 103 determines to interpolate using the first interpolation. On the other hand, if there is no normal photoelectric conversion region necessary for interpolation, the process proceeds to step S807, and the CPU 103 determines to interpolate using the second interpolation.

  In step S804, the image processing unit 108 determines peripheral pixels including a photoelectric conversion region used for interpolation. As an example, as described above, the closest pixel may be a peripheral pixel, or the closest pixel of the same color may be a peripheral pixel. In step S806, the image processing unit 108 determines a weighting coefficient used for the settlement of the interpolation. The method described above is used to determine the weighting factor.

  Here, returning to step S807, in the subsequent step S808, the image processing unit 108 selects a plurality of peripheral pixels corresponding to the same color filter as the pixel including the defective photoelectric conversion region.

  In step S809, the image processing unit 108 performs interpolation calculation using a signal of the photoelectric conversion region included in the selected pixel. In step S810, the CPU 103 determines whether all the pixels have been processed. If all the pixels have not been processed, the process returns to step S802.

  According to the present embodiment, when an interpolation is performed on a pixel having a defect in a part of the photoelectric conversion unit with an imaging device including a plurality of pixels having a plurality of photoelectric conversion units for one microlens, It is possible to provide a higher degree of freedom in determining the surrounding image used for interpolation.

(Other embodiments)
The above embodiment has been described based on the implementation in the imaging apparatus, but is not limited to the imaging apparatus. For example, you may implement with the portable apparatus etc. with which the image pick-up element was incorporated.

  Note that the present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image sensor 101 AFE
102 TG
103 CPU
104 Operation unit 105 Display unit 106 RAM
107 ROM
DESCRIPTION OF SYMBOLS 108 Image processing part 109 AF calculating part 110 Flash memory 111 Focal plane shutter 112 Focus drive circuit 113 Aperture drive circuit 114 Focus actuator 115 Aperture actuator 116 1st lens 117 Aperture 118 2nd lens 119 3rd lens 120 Correlation calculation part

Claims (17)

An imaging device comprising a plurality of pixels having a plurality of photoelectric conversion regions for one microlens, each of the plurality of photoelectric conversion regions including one or more photoelectric conversion units;
A processing target having a first photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as not normal and a second photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as normal Interpolation means for performing a first interpolation on the first photoelectric conversion region of the pixel,
The interpolation means includes
For each of a plurality of peripheral pixels located around the pixel to be processed, a coefficient is calculated for a signal in the third photoelectric conversion region corresponding to the position of the first photoelectric conversion region in the pixel to be processed. Corresponding to the first value obtained by using the weighted average and the position of the second photoelectric conversion region in the processing target pixel of each of a plurality of peripheral pixels positioned around the processing target pixel The ratio of the second value obtained by performing the weighted average using the coefficient for the signal of the fourth photoelectric conversion region to be obtained,
An imaging apparatus, wherein the first interpolation is performed using the signal of the second photoelectric conversion region and the ratio.   The imaging apparatus according to claim 1, wherein the peripheral pixel used when the interpolation unit obtains the ratio is a pixel having a different color from a pixel including the first photoelectric conversion region.   The interpolation unit selects a pixel adjacent to the processing target pixel and having a color different from that of the processing target pixel when the predetermined condition is satisfied and does not satisfy the predetermined condition. The imaging apparatus according to claim 1, wherein a pixel that is not adjacent to the processing target and has the same color as the processing target pixel is selected as the peripheral pixel.   The predetermined condition is that, in an area set based on the pixel to be processed, a signal of a pixel corresponding to any one color is larger than a signal of a pixel corresponding to another color, and the difference between the signals has a threshold value. The imaging apparatus according to claim 3, wherein the number exceeds.   5. The coefficient according to claim 1, wherein the interpolation unit sets a coefficient for a photoelectric conversion region including a defective photoelectric conversion unit among the photoelectric conversion regions of the plurality of peripheral pixels to 0. 6. The imaging device according to item.   6. The interpolation means sets the coefficient in each of the peripheral pixels in accordance with a signal ratio between the third photoelectric conversion region and the fourth photoelectric conversion region. The imaging device according to any one of the above. Of each of the peripheral pixels, the ratio of the signal of the third photoelectric conversion region and the signal of the fourth photoelectric conversion region is calculated in each of two pixels located symmetrically with respect to the pixel,
The imaging apparatus according to claim 6, wherein the interpolation unit sets the coefficient according to a difference in the signal ratio.   The imaging apparatus according to claim 7, wherein the interpolation unit sets the smaller coefficient as the difference in the signal ratio is larger.   The imaging apparatus according to claim 8, wherein the interpolation unit sets the coefficient to 0 when a difference in the signal ratio is larger than a predetermined threshold value. Detecting means for detecting that the respective outputs of the photoelectric conversion units of the plurality of pixels have reached a saturation level;
The detection means sets a coefficient for a photoelectric conversion region included in a pixel in which at least a part of the output of the photoelectric conversion unit is detected to reach a saturation level to 0. The imaging device according to any one of 1 to 9.   The interpolation means performs second interpolation for performing interpolation on the signal of the first photoelectric conversion region without using the signal of the second photoelectric conversion region and the ratio. The imaging apparatus according to any one of claims 1 to 10, wherein one of interpolation and the second interpolation is selected.   When the second photoelectric conversion region includes a defective photoelectric conversion unit, the interpolation means performs the second interpolation using a signal of the third photoelectric conversion region of the peripheral pixel. The imaging apparatus according to claim 11. In the second interpolation,
The image pickup apparatus according to claim 12, wherein the image pickup apparatus performs the calculation using an average of signals in the third photoelectric conversion region of the peripheral pixels. Having storage means for storing information indicating a defective photoelectric conversion unit;
The said interpolation means makes the photoelectric conversion area | region containing the photoelectric conversion part which the information memorize | stored by the said memory | storage means shows as the object of said 1st interpolation, The one of Claim 1 thru | or 13 characterized by the above-mentioned. The imaging device described.   The image pickup apparatus according to claim 1, wherein the pixel array is a Bayer array. An imaging device comprising a plurality of pixels having a plurality of photoelectric conversion regions for one microlens, each of the plurality of photoelectric conversion regions including one or more photoelectric conversion units;
A processing target having a first photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as not normal and a second photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as normal Interpolation means for performing a first interpolation on the first photoelectric conversion region of the pixel;
In the control method of the imaging device having
For each of a plurality of peripheral pixels located around the pixel to be processed, a coefficient is calculated for a signal in the third photoelectric conversion region corresponding to the position of the first photoelectric conversion region in the pixel to be processed. Corresponding to the first value obtained by using the weighted average and the position of the second photoelectric conversion region in the processing target pixel of each of a plurality of peripheral pixels positioned around the processing target pixel Obtaining a ratio of a second value obtained by performing a weighted average using the coefficient with respect to the signal of the fourth photoelectric conversion region to be performed;
A method for controlling an imaging apparatus, comprising the step of performing the first interpolation using the signal of the second photoelectric conversion region and the ratio. An imaging device comprising a plurality of pixels having a plurality of photoelectric conversion regions for one microlens, each of the plurality of photoelectric conversion regions including one or more photoelectric conversion units;
A processing target having a first photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as not normal and a second photoelectric conversion region including a photoelectric conversion unit that outputs a signal regarded as normal Interpolation means for performing a first interpolation on the first photoelectric conversion region of the pixel;
A program for causing a computer to operate a control method of an imaging apparatus having
In the computer,
For each of a plurality of peripheral pixels located around the pixel to be processed, a coefficient is calculated for a signal in the third photoelectric conversion region corresponding to the position of the first photoelectric conversion region in the pixel to be processed. Corresponding to the first value obtained by using the weighted average and the position of the second photoelectric conversion region in the processing target pixel of each of a plurality of peripheral pixels positioned around the processing target pixel Obtaining a ratio of a second value obtained by performing a weighted average using the coefficient with respect to the signal of the fourth photoelectric conversion region,
A computer program comprising the step of performing the first interpolation using the signal of the second photoelectric conversion region and the ratio.

Images