JP2017098790A - Imaging apparatus, control method of the same, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress reduction of a quality of an output image when a pixel number of which a signal has to be read in a screen is different at each line.SOLUTION: An imaging apparatus comprises: a plurality of imaging elements to which a unit pixel having a plurality of photoelectric conversion elements is arranged; a shutter scanning for starting an exposure to the imaging element; a driving part that allows the apparatus to execute a reading out scanning for an exposure end; a gain part that applies a gain to an image signal read out from the imaging element; and a control part that controls the driving part so that the shutter scanning of a reading scanning and a present frame are prevented from an overlap when the number of photoelectric conversion element reading the image signal from the plurality of unit pixel is changed in each frame, and control the gain of the gain part so that the frames before and after and a signal level are matched in accordance with an exposure time of the present frame set by the control of driving part.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an imaging apparatus and a control method thereof.

  In recent years, an image pickup device in which a pixel for detecting a phase difference is mounted in a pixel portion that takes in light from a lens and converts it into an image signal, and is capable of focus detection by a method called an imaging surface phase difference detection method. Equipment has been commercialized by each company.

  For example, Patent Document 1 proposes a method for controlling an image sensor that can perform focus detection using an imaging surface phase difference detection method. In Patent Document 1, the imaging device includes a plurality of pairs of pixels that respectively receive a subject light flux that has passed through a pair of pupil regions (for example, left and right regions) in an exit pupil of a photographing lens (imaging optical system). A signal for phase difference detection can be generated.

  By the way, in an imaging device having pixels that perform pupil division, power consumption increases when signals are read from all pixels. Therefore, in Patent Document 1, a signal that is not combined with a signal obtained by combining the output signals of a plurality of photodiodes is read from each unit pixel only in a region where autofocus is performed, and output signals of a plurality of photodiodes are output from each unit pixel in other regions. There has been proposed a method of reducing power consumption by reading only a signal obtained by combining the signals.

  In such a reading method, the following problems occur because the number of output signals read from each unit pixel of the image sensor is different in each row. That is, if the horizontal blanking period (the period during which the pixel signal is not read out of the horizontal scanning period) is sufficiently long, the output signals of all the pixels can be read without changing the horizontal scanning period even if the number of output signals changes. Can do. However, when the horizontal blanking period is short, it is necessary to change the horizontal scanning period for each row.

JP 2014-239316 A

  In driving the image sensor, the exposure time of a pixel is from the shutter scan that resets the pixel to the read scan that reads a signal from the pixel. For this reason, it is necessary for the pixels in the same frame to have the same time from the shutter scan to the readout scan.

  However, when the area used for autofocus is changed on the screen, there are pixel rows on the screen that must be read from each unit pixel and a signal that is not combined with the combined output signal of multiple photodiodes. Change. That is, every time the area used for autofocus is changed in the screen, the pixel rows for which the horizontal scanning period has to be extended change. There is also a possibility that the readout scanning timing changes, the exposure time changes in each row, and the shutter scanning overtakes the readout scanning. In such a case, there is a possibility that a normal image may not be output in the next frame whose readout scanning timing is changed. However, when a pixel signal is used for autofocus, the video of the next frame is normally output. It is necessary to let

  The present invention has been made in view of the above-described problems, and an object of the present invention is to suppress degradation in the quality of an output image when the number of pixels from which signals must be read out in a screen is different for each row. .

  An image pickup apparatus according to the present invention includes an image pickup element in which a plurality of unit pixels each having a plurality of photoelectric conversion elements are arranged, a shutter scan for starting exposure to the image pickup element, and a readout scan for ending exposure. When changing the number of drive means for executing the above, gain means for gaining an image signal read from the image sensor, and photoelectric conversion elements for reading the image signal from each of the plurality of unit pixels, for each frame The driving means is controlled so that the readout scanning of the previous frame and the shutter scanning of the current frame do not overlap, and the preceding and succeeding frames and signal levels are set according to the exposure time of the current frame set by the control of the driving means. And control means for controlling the gain of the gain means so as to match.

  The image pickup apparatus according to the present invention includes an image pickup element in which a plurality of unit pixels each having a plurality of photoelectric conversion elements are arranged, shutter scanning for starting exposure on the image pickup element, and reading for completion of exposure. Drive means for executing scanning, gain means for applying a gain to the image signal for each line read from the image sensor, and the number of photoelectric conversion elements for reading the image signal from each of the plurality of unit pixels for each frame In the case of changing to the above, the driving means is controlled so that the shutter scan of the current frame is matched with the reading scan of the previous frame, and the exposure time for each line in the current frame set by the control of the driving means Control means for controlling the gain of the gain means so as to match the signal level with the frame.

  According to the present invention, when the number of photoelectric conversion elements that read an image signal from each of a plurality of unit pixels in a frame is different for each row, it is possible to suppress degradation in the quality of an output image.

1 is a block diagram showing a configuration of an imaging apparatus according to a first embodiment of the present invention. FIG. 3 is a diagram illustrating a configuration of an image sensor according to the first embodiment. FIG. 6 is a diagram illustrating timing of shutter scanning and readout scanning of the image sensor. FIG. 6 is a diagram illustrating timing of shutter scanning and readout scanning of the image sensor. FIG. 6 is a diagram illustrating timing of shutter scanning and readout scanning of the image sensor. 6 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to the first embodiment of the present invention. In FIG. 1, an optical lens 101 captures light of a subject and forms a subject image on the imaging surface of the image sensor 102. A focusing mechanism for adjusting the focus, a diaphragm mechanism for adjusting the light amount and depth of field, a zoom mechanism for changing the focal length, and the like are provided. However, it does not necessarily have all of these configurations depending on the type of lens. For example, if it is a single focus lens, it does not have a zoom mechanism, and if it is a pan focus lens, the focus is only one point at infinity and no focus mechanism. In order to reduce the cost of the lens, there is a case in which the aperture position is set to one point, and the light amount adjustment is replaced with an ND filter instead of the aperture. In the present embodiment, the optical lens 101 includes all types of lenses having a function of forming light on an image sensor even if the configuration is different.

  The image sensor 102 receives incident light from the optical lens 101, converts it into an electrical signal, and outputs it. Representative examples include a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, and the like. In these imaging devices 102, there are an image signal to be output that is an analog image signal and a digital image data. A device that outputs digital image data performs AD (analog-digital) conversion processing inside the image sensor, and outputs digital data using an interface such as LVDS (Low voltage differential signaling).

  FIG. 2A is a block diagram showing the configuration of the image sensor in the present embodiment. In FIG. 2A, a timing generator (hereinafter referred to as TG) 201 controls the driving of the entire image sensor 102. The pixel unit 202 includes a plurality of unit pixels each having a photodiode or a floating diffusion amplifier for converting light into an electric signal, and a column AD converter (hereinafter referred to as a column ADC) in the subsequent stage converts the signal of each pixel into a column unit. It outputs to 203. The column ADC 203 converts an analog image signal of each pixel output from the pixel unit 202 into a digital signal. A horizontal shift register (hereinafter, HSR) 204 transfers the digital signal of each pixel converted by the column ADC 203 to a parallel / serial conversion unit (hereinafter, P / S) 205 for each row. The P / S 205 converts it into a serial signal compatible with LVDS, which is often used in recent years as a method for outputting a digital signal. The LVDS unit 206 is a drive circuit for outputting the serial signal converted by the P / S 205.

  Among these configurations, the configuration of each unit pixel of the pixel unit 202 will be described with reference to FIG. FIG. 2B is a diagram schematically illustrating a cross section of a plurality of unit pixels arranged in the pixel portion 202 of the image sensor 102. The microlens 301 is a lens for efficiently condensing the light incident on the image sensor onto the photodiode, and increases the sensitivity of the image sensor by improving the light collection rate. The color filter 302 splits incident light into three or four sets of colors including R, G, and B. A typical structure is an array structure called a Bayer array. The inner lens 303 is also called an in-layer lens, and is provided between the microlens and the photodiode. Usually, the arrangement of the inner lens is effective in reducing the pixel size, and the sensitivity can be improved even with respect to an incident light beam having a small F value and a large inclination.

  The photodiodes (photoelectric conversion elements) 304A and 304B perform photoelectric conversion that converts incident light into electric charges. Usually, one photodiode is arranged for one microlens or color filter. However, the imaging apparatus according to the present embodiment includes a plurality of photodiodes (two photodiodes 304A and 304B in the present embodiment). The plurality of photodiodes receive light that has passed through different areas of the exit pupil of the optical lens 101 and perform pupil division. That is, two or more circuits for reading out photodiode signals are required for one microlens. This is one method for realizing the imaging surface phase difference detection method, and the phase difference for calculating the defocus amount by comparing the image signals read from the two photodiodes 304A and 304B. Is detected.

  In this embodiment, the image sensor 102 reads a signal so that the phase difference can be detected from the two photodiodes 304A and 304B, and a method of reading the signal without mixing the phase difference by mixing the signals of the two photodiodes. Can be changed in each row (each line). A method of reading so that the phase difference can be detected is called independent reading. For independent reading, the signals of the two photodiodes are output separately as an A signal and a B signal. Alternatively, an A signal (or B signal) that is an output of one photodiode and an A + B signal obtained by mixing the A signal and the B signal are output. On the other hand, a read method that does not detect the phase difference and reads only the A signal (B signal) which is the output of one of the two photodiodes or only the A + B signal obtained by mixing the A signal and the B signal is called mixed reading. At this time, the line for performing independent reading is twice as long as the pixel signal reading time for the line for performing mixed reading, so the horizontal scanning period is also twice as long.

  Returning to the description of FIG. 1, the image processing unit 103 made of an LSI includes an analog front end unit and a digital processing unit. The analog front end unit converts the analog image signal output from the image sensor 102 into a digital signal. The digital processing unit performs each process on the digitized image signal. In addition, when AD conversion is performed inside the image sensor 102 and a digital signal such as LVDS is output, the analog front end unit is omitted. For example, when the image sensor 102 is a CMOS image sensor, the image processing unit 103 performs removal of fixed pattern noise unique to the CMOS image sensor, black level clamp processing, and the like. In addition, various image processing such as pixel mixing processing, noise reduction processing, gamma correction processing, knee correction processing, digital gain processing, and scratch correction processing, which are typical image processing in the imaging apparatus, is performed. The image processing unit 103 also includes a storage circuit that stores setting values necessary for each correction and image processing.

  Typical examples of the display unit 104 that displays an input subject image include an external monitor, a liquid crystal monitor attached to the imaging apparatus, and a viewfinder. The user of the imaging device confirms the angle of view, exposure, and the like through the display unit 104. In recent years, there are many image pickup apparatuses in which a touch panel is mounted on the display unit 104.

  The signal recording unit 105 records an image signal reflecting each image processing received from the image processing unit 103 in a storage device or a storage medium. The exposure control unit 108 calculates an appropriate exposure amount from the image signal information received from the image processing unit 103. Further, a defocus amount based on the phase difference is calculated from the signal output from the image sensor 102, and an appropriate focus position is determined. Then, the information is transmitted to the image sensor control unit 107 that controls the image sensor 102 and the lens control unit 106 that controls the optical lens 101.

  Next, with reference to FIG. 3, problems that occur in shutter scanning and readout scanning when the number of signal readout pixels is changed for each region in the screen of the image sensor will be described. In the image sensor, shutter scanning is performed for the start of exposure, and readout scanning is performed for the end of exposure. FIG. 3A is a diagram showing the state of image output, with time on the horizontal axis and the number of vertical lines on the vertical axis. On the horizontal axis, a vertical synchronization signal (VD) is output every predetermined time, and one frame of image data is output every 1 VD. The imaging element 102 includes a region where independent reading is performed and a region where mixed reading is performed, and FIG. 3 shows the timing of shutter scanning and reading scanning when the region where independent reading is performed is changed.

  If the image sensor is a CMOS image sensor, the rolling shutter method is generally used because the signal of each pixel is sequentially read out line by line. That is, the image signal is read out line by line from the upper direction to the lower direction of the vertical line. The image sensor 102 performs readout scanning, which is the end of exposure, at the fall of VD, and then performs shutter scanning, which is the start of exposure, to reset each pixel. In frame 1, the exposure period from the shutter scan to the readout scan is time t for all lines. In the independent readout region, the number of pixel signals to be read increases because the signals of the two photodiodes are output separately as the A signal and the B signal, and the readout scanning takes time. That is, in FIG. 3, this independent readout region is represented by a portion where the inclination of shutter scanning and readout scanning is smaller than that of the mixed readout region.

  In response to an instruction to change the independent readout area, readout scanning is changed from frame 2. In this case, when the shutter scan is performed along the reading scan of the frame 1, an exposure change region 1 in which the relationship of the exposure time t is broken in the screen is generated. Further, when the independent reading area is changed also in the next frame 3, the exposure change area 2 is similarly generated. When an exposure change area occurs, the exposure time changes by that line, resulting in a low quality image.

  FIG. 3B is a diagram showing a case where the shutter scan is also changed in advance in accordance with the readout scan after the readout area is changed. In this case, the shutter scanning immediately after the readout area of frame 1 is matched with the independent readout area in frame 2. In this case, the readout scan of frame 1 and the shutter scan in frame 2 overlap, and the exposure time of the line of frame 1 in the overlapped area changes greatly, which also results in a low-quality image.

  Next, the operation of the imaging apparatus according to the present embodiment will be described with reference to FIGS. FIG. 4A shows the state of image output, with time on the horizontal axis and the number of vertical lines on the vertical axis, as in FIG. On the horizontal axis, a vertical synchronization period (VD) is output every predetermined time, and one frame of image data is output every 1 VD. The image sensor 102 includes both independent readout areas and mixed readout areas. FIG. 4A shows the timing of shutter scanning and readout scanning when the independent readout area is changed. Also, the timing at which each step in FIG. 6 is performed is indicated by a step number in FIG.

  When outputting an image in which the independent readout area is changed from frame 2, the imaging apparatus determines whether or not the exposure time t to frame 1 is longer than time t2. Here, t2 is expressed by the following mathematical formula, for example.

t2 = t1-nα
Here, the time t1 [ms] is the maximum exposure time when the shutter scanning does not overtake the readout scanning for reading the signal of the previous frame, and the time α [ms] is one horizontal scanning period (HD) at the time of independent readout. The number of lines n [lines] is the number of lines for performing independent reading.

  When the time t is longer than the time t2, changing the independent readout area may cause the shutter scanning to pass over the readout scanning of the frame 1. Therefore, in the frame 2 where the readout scanning is changed, the timing of performing the shutter scanning is changed so that the exposure time is the time t2, and the shutter scanning is changed so as to match the readout scanning of the frame 2. It is not necessary to change the exposure time t in a frame in which the area to be read is not changed. By controlling the exposure time and shutter scanning of the image sensor at such timing, it is possible to output an image whose exposure time does not change over the entire screen without overtaking the readout scanning.

  Next, the operation of the imaging apparatus of the present embodiment will be described with reference to the flowchart of FIG. First, after receiving a read area change instruction to change the independent read area in step S1, the exposure period t of frame 1 is compared with the above time t2 in step S2. As a result of the comparison, if the exposure time t is longer than the time t2, the image sensor 102 is driven and controlled to change the exposure time of the frame 2 to the time t2 in step S3. When the exposure time t is less than the time t2, it is not necessary to change the exposure time.

  Next, in step S4, the image sensor 102 is driven and controlled to change the shutter scanning of frame 2 to match the reading scanning of frame 2, and the shutter scanning of frame 2 is changed in step S5. After that, in step S6, the image sensor 102 is driven and controlled to change the reading scan of frame 2, so that the reading scan of frame 2 is changed in step S7.

  When the exposure time t is changed to the time t2, the exposure time of the frame is different from that of other frames. Therefore, the image processing unit 103 adjusts the signal level by applying digital gain to the entire frame image. In the digital gain section, the continuity of image brightness with other frames can be maintained by applying a gain of t / t2.

  By the method described above, even if the readout area is changed while changing the horizontal scanning period of the image sensor, the readout area can be changed seamlessly without degrading the image quality.

  In the present embodiment, an image sensor having two photoelectric conversion elements in which a unit pixel is divided into pupils is used to mix independent readout for reading the signals of the two photoelectric conversion elements and the signals of the two photoelectric conversion elements. An example was given in which the horizontal scanning period was changed for mixed readout to be read. However, the method of this embodiment may be applied when the horizontal scanning period is changed using, for example, a driving method of reading out signals by thinning out pixels. In other words, this embodiment is established when the horizontal scanning period of the image sensor performs different driving for each line.

(Second Embodiment)
In the first embodiment, a method of adjusting the exposure time by combining shutter scanning and readout scanning for every frame has been described. However, the quality of the image may be maintained by combining the readout scanning of the previous frame and the shutter scanning of the current frame.

  This embodiment will be described with reference to FIG. FIG. 4B is a diagram illustrating a case where the readout area is changed in frame 2 and frame 3. By combining the shutter scan of frame 2 with the readout scan corresponding to the readout area of frame 1, the shutter scan does not overtake the readout scan. However, since the shutter scanning and readout scanning of frame 2 do not match, in frame 2, the exposure time differs in the portion of the line where independent readout is performed in the preceding and succeeding frames.

Therefore, the digital gain unit of the image processing unit 103 needs to apply a gain by changing the value of the digital gain for each line, not the entire frame image. In this case, the digital gain value of each line is
t / (frame readout scanning timing−shutter scanning timing)
It becomes. Therefore, the digital gain value can be determined by calculating the scanning timing from the horizontal scanning period of each line.

  FIG. 5 is a diagram illustrating an example of a change in the digital gain value of each line when the shaded portion is an exposure change region in relation to the previous and subsequent frames. At the top of the frame, the exposure time is time t, and after the exposure change region is reached, the exposure time is gradually shortened. When the shutter scanning and readout scanning periods are the shortest, the period is t3. Thereafter, the exposure time returns to time t. As shown in the figure, the digital gain value is changed from 1 × at the upper part of the screen to t / t3 times at the line with the shortest exposure time and to 1 × as it goes to the lower part of the screen.

  By using the method described above, the position of the readout area can be seamlessly changed without degrading the quality of the image even when the position of the readout area is changed while changing the horizontal scanning period of the image sensor. It becomes possible to change.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101: Optical lens, 102: Image sensor, 103: Image processing part, 104: Display part, 105: Signal recording part, 106: Lens control part, 107: Image sensor control part, 108: Exposure control part

Claims (16)

An image sensor in which a plurality of unit pixels each having a plurality of photoelectric conversion elements are arranged;
Driving means for causing the image sensor to perform shutter scanning for starting exposure and reading scanning for ending exposure;
Gain means for applying a gain to the image signal read from the image sensor;
When the number of photoelectric conversion elements that read image signals from each of the plurality of unit pixels is changed for each frame, the driving unit is controlled so that the reading scan of the previous frame and the shutter scanning of the current frame do not overlap. Control means for controlling the gain of the gain means so as to match the signal level with the preceding and succeeding frames according to the exposure time of the current frame set by the control of the driving means;
An imaging apparatus comprising:   The image pickup apparatus according to claim 1, wherein the control unit controls shutter scanning of the current frame so as to match the reading scanning of the current frame.   The control means controls the readout scan of the previous frame and the shutter scan of the current frame so as not to overlap by delaying the timing of the shutter scan of the current frame with respect to the readout scan of the previous frame. The imaging apparatus according to claim 2.   The imaging apparatus according to claim 3, wherein an exposure time of the current frame is set shorter than an exposure time of the previous frame. The control means includes
Current frame gain = previous frame gain
X (exposure time of previous frame) / (exposure time of current frame)
The imaging apparatus according to claim 4, wherein the gain of the gain unit is controlled so that   The imaging apparatus according to claim 1, wherein the gain unit applies a digital gain to the image signal.   The imaging device according to claim 1, wherein each of the unit pixels includes a plurality of photoelectric conversion elements arranged for one microlens.   The image pickup device calculates a defocus amount using output signals of a plurality of photoelectric conversion elements arranged for the one microlens and a case where the defocus amount is not calculated in a horizontal scanning period. The imaging device according to claim 7, wherein the two are different. An image sensor in which a plurality of unit pixels each having a plurality of photoelectric conversion elements are arranged;
Driving means for causing the image sensor to perform shutter scanning for starting exposure and reading scanning for ending exposure;
Gain means for applying a gain to the image signal for each line read from the image sensor;
When changing the number of photoelectric conversion elements that read an image signal from each of the plurality of unit pixels for each frame, the driving unit is controlled to match the shutter scan of the current frame with the read scan of the previous frame, and Control means for controlling the gain of the gain means so as to match the signal level with the preceding and following frames according to the exposure time for each line in the current frame set by the control of the driving means;
An imaging apparatus comprising:   The imaging apparatus according to claim 9, wherein the gain unit applies a digital gain to the image signal.   11. The image pickup apparatus according to claim 9, wherein each of the unit pixels includes a plurality of photoelectric conversion elements arranged with respect to one microlens.   The image pickup device calculates a defocus amount using output signals of a plurality of photoelectric conversion elements arranged for the one microlens and a case where the defocus amount is not calculated in a horizontal scanning period. The imaging apparatus according to claim 11, wherein the two are different from each other. A method for controlling an imaging apparatus including an imaging element in which a plurality of unit pixels each including a plurality of photoelectric conversion elements are arranged,
A driving process for causing the image sensor to perform shutter scanning for starting exposure and reading scanning for ending exposure;
A gain step of applying a gain to the image signal read from the image sensor;
When the number of photoelectric conversion elements that read image signals from each of the plurality of unit pixels is changed for each frame, the driving process is controlled so that the reading scan of the previous frame and the shutter scanning of the current frame do not overlap. A control step of controlling the gain of the gain step so as to match the signal level with the preceding and following frames according to the exposure time of the current frame set by the control of the driving step;
A method for controlling an imaging apparatus, comprising: A method for controlling an imaging apparatus including an imaging element in which a plurality of unit pixels each including a plurality of photoelectric conversion elements are arranged,
A driving process for causing the image sensor to perform shutter scanning for starting exposure and reading scanning for ending exposure;
A gain step of applying a gain to the image signal for each line read from the image sensor;
When changing the number of photoelectric conversion elements that read image signals from each of the plurality of unit pixels for each frame, the driving process is controlled so that the shutter scan of the current frame is matched with the read scan of the previous frame, and A control step for controlling the gain of the gain step so as to match the signal level with the preceding and following frames according to the exposure time for each line in the current frame set by the control of the driving step;
A method for controlling an imaging apparatus, comprising:   The program for making a computer perform each process of the control method of Claim 13 or 14.   15. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 13 or 14.

Images