JP2018037740A - Imaging device having an image sensor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve obtaining an imaging apparatus which can comply with also movie shooting of a moving object or the like and allowing for shooting an image having a wide dynamic range without losing color balance of a high- luminance object.SOLUTION: The imaging apparatus includes: an imaging device 402 in which pixels each including a light receiving element 108 and change units 104, 105 for changing capacity of a charge storage unit of the light receiving element are arranged on a matrix plane; and a setting unit 406 for setting, using information of the imaging device, a region in which the pixel reaches a threshold as a high-luminance region. The change units 104, 105 change the capacity of the charge storage unit of the light receiving element 108 in the imaging device 402 set to be the high-luminance region. The setting unit 406, when there is an interval region separating the high-luminance region and the interval region separates the high-luminance region at a fixed interval in a vertical-axis direction or a horizontal-axis direction of the matrix plane, sets also the interval region as the high-luminance region.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus having an image sensor.

  In an image pickup apparatus having an image sensor with a narrow dynamic range, in order to prevent a blackout phenomenon and a whiteout phenomenon, the capacitance is increased with respect to a technique for high dynamic range composition (HDR composition) and a pixel determined to have high luminance. Technology to do so is widespread.

  For example, Patent Document 1 describes a method of capturing images with different exposure times with a time difference and synthesizing two images. Patent Document 2 describes a method of increasing the capacitance of a floating diffusion (FD) that accumulates charges output from a photodiode (PD) when high luminance is detected.

JP 2003-163831 A JP 2013-192058 A

  However, for example, when a moving image of a moving object or the like is shot in Patent Document 1, the positions of the high-luminance portion and the low-luminance portion are shifted in the two images to be combined, and the natural image is not displayed. Deteriorate. Further, in Patent Document 2, since the mounting positions of the photometric sensor for determining high brightness and the image sensor are actually different in the imaging apparatus, the position of the pixel determined to be high brightness and the position of the high brightness pixel on the image sensor are different. It is out of place and cannot be strictly handled.

  Further, in the high luminance determination in Patent Document 2, the pixel is determined as a high luminance pixel when the sensor output level exceeds a preset threshold value. Here, with reference to FIG. 7, a general high brightness determination in the Bayer array of the color image sensor will be described. Here, the vertical axis represents the pixel output level, and the horizontal axis represents the exposure time.

  As shown in FIG. 8, in the Bayer array, there are twice as many G pixels 601 indicating green as R pixels 602 indicating red. Therefore, when a white subject is photographed and the luminance of the subject increases, the G pixel 601 is saturated before the R pixel 602. As a result, only the G pixel 601 is determined to be a high luminance pixel, and if only the G pixel 601 is suppressed from whiteout, the color balance with the R pixel 602 and the B pixel is lost, and the color of the high luminance region becomes uneven.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an imaging apparatus capable of shooting moving images of moving objects and the like and capable of shooting images with a wide dynamic range without impairing the color balance of high-luminance subjects. Is to provide.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention is an imaging device in which pixels including a light receiving element and a changing unit that changes a capacity of a charge storage unit of the light receiving element are arranged in a matrix plane. And a setting unit that sets a region where the pixel has reached a threshold from the information of the imaging device as a high luminance region, and the changing unit is configured to set the region of the imaging device set to the high luminance region. The capacitance of the charge accumulating unit of the light receiving element is changed, and the setting unit includes a spacing region that separates the high luminance region, and the spacing region is in a vertical axis direction or a horizontal axis direction of the matrix plane. Thus, when the high-luminance region is separated at a constant interval, the interval region is also set as the high-luminance region.

  According to the present invention, it is possible to obtain an imaging apparatus capable of shooting moving images of moving objects and the like and capable of shooting images with a wide dynamic range without impairing the color balance of a high-luminance subject.

The equivalent circuit schematic of the pixel of embodiment. The figure explaining the case where the pixel which does not reach the high-intensity level of embodiment is included. The first half of the flowchart in which the high brightness area setting unit in the embodiment sets the high brightness area. The latter half of the flowchart in which the high-intensity area setting unit in the embodiment sets the high-intensity area. 1 is a block diagram of an entire imaging apparatus according to an embodiment. The figure explaining the sensitivity difference of the ON state of a high sensitivity mode, and an OFF state. The figure explaining the sensitivity of G pixel and R pixel. Explanatory drawing of a Bayer arrangement | sequence.

  Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment. Moreover, you may comprise combining suitably one part of each embodiment mentioned later.

(Embodiment)
First, the configuration of the first embodiment will be described with reference to FIG. The imaging apparatus according to the first embodiment includes an optical system 401, an image sensor 402, an AD converter 403, an image processing unit 404, a video output unit 405, a high brightness area setting unit 406, and a timing generator (TG) 407. The image sensor 402 is a color image sensor and is a CMOS image sensor, but is simply referred to as an image sensor below.

  Furthermore, the optical system 401 includes a zoom mechanism unit and a focus mechanism unit (not shown). The light incident from the optical system 401 forms an image on the light receiving surface of the image sensor 402 which is an image sensor.

  Here, the pixel arrangement in the image sensor 402 will be described with reference to FIG. In the image sensor 402, green (G) pixels, red (R) pixels, and blue (B) pixels are arranged in a matrix in an array called a Bayer array. Light that forms an image on the light receiving surface of the image sensor 402 during the exposure period is converted into an RGB analog signal by an intensity of voltage corresponding to the luminance level as an RGB color signal in each pixel.

  The RGB analog signal is converted into an RGB digital signal by the AD conversion unit 403 and sent to the image processing unit 404. The image processing unit 404 performs correction processing such as optical black (OB) clamp processing and noise reduction processing on the RGB digital signal, and then performs color adjustment and brightness adjustment such as white balance and gamma processing, and the image of the YPbPr signal. Generate a signal.

  The image processing unit 404 is a detection unit that checks the output level of each pixel and constantly detects pixels that reach a certain threshold during the exposure period as saturated pixels, that is, high-luminance pixels. When the saturated pixel is detected, the coordinates on the image sensor are transmitted to the setting unit 406 in the high luminance area. The high brightness area setting unit 406 sets a range in which the high brightness area should be set by calculation based on the coordinate information of the saturated pixels. The setting of the high luminance area will be described in detail in a later paragraph.

  In addition, the TG 407 controls the pixels in the high luminance area of the image sensor set by the high luminance area setting unit 406 to shift to the high saturation mode, and suppresses the saturation of the pixels of the image sensor. . The high saturation mode will be described in detail in a later paragraph.

  On the other hand, the video signal processed by the image processing 404 is sent to the video output unit 405, converted into a high definition-serial digital interface (HD-SDI) signal, and output to the outside.

  Next, an equivalent circuit of one pixel in the image sensor 402 will be described with reference to FIG. The equivalent circuit of the image sensor 402 includes a mode (MODE) terminal 101, a reset terminal 102, a MOS FET (MOSFET), a floating diffusion (FD) 104, an extended FD 105, a photodiode (PD) 108, a terminal 109, and a line select. It comprises a terminal 110, an output amplifier 114, and the like. There are 103, 106, 107, 111, 112, and 113 in the MOS MOST (MOSFET).

  MOS MOS (MOSFET) 103, 106, 107, 111, 112, 113 is a kind of field effect transistor. The PD 108 is a light receiving element that performs photoelectric conversion for converting light during an exposure period into electricity and accumulates charges in the PD 108. When the terminal 109 becomes HIGH, the MOSFET 111 is turned on, and the charge accumulated in the PD 108 is output to the FD 104. The FD 104 is a standard FD. The charge output from the PD 108 is converted into a voltage by the standard FD 104 and output as a pixel voltage signal via the MOSFET 107.

  A reset terminal 102 resets the standard FD 104 and the extended FD 105. When the reset terminal 102 becomes HIGH, the MOSFETs 103 and 106 are turned on, and the charges accumulated in the standard FD 104 and the extended FD 105 are reset.

  As many line select terminals 110 as the number of lines are arranged. When the line select terminal 110 becomes HIGH, the MOSFET 112 is turned on, and the voltage signal from the MOSFET 107 is output and is output after being amplified by the output amplifier 114. The MODE terminal 101 is connected to the line select terminal 110 by an AND circuit. When both are HIGH, the MOSFET 106 is turned on, and the extended FD 105 is connected to the standard FD 104 in parallel. In this way, a changing unit that changes the capacity of the charge storage unit is formed.

(High saturation mode)
When the FD capacity (capacitance) increases,
Q = CV (Q: charge [C], C: capacitance [F], V: voltage [V])
Thus, even if the charge Q output from the photodiode is constant, the voltage output V is low. In this case, the mode in which the FD capacity is increased is referred to as a high saturation mode.

  With reference to FIG. 6, the output level when the high saturation mode is ON and when it is OFF will be described. The horizontal axis is the received light intensity, and the vertical axis is the output level. A graph 501 is when the high saturation mode is OFF, and a graph 502 is when the high saturation mode is ON.

  In this embodiment, since the capacitance component of the FD is set to be doubled when the high saturation mode is turned on, the slope of the graph when the high saturation mode is on is a graph of the OFF state. It is ½ compared to the inclination of. That is, when the high saturation mode is ON, the output level decreases.

  As shown in FIG. 6, by turning on the high saturation mode, it is possible to suppress saturation in the standard FD 104 and the extended FD 105 and saturation of the output amplifier 114 in the subsequent stage.

  The MODE terminal 101 is arranged for each column, and by using it together with the line select terminal 110, the extended FDs 105 of all the pixels in the CMOS image sensor can be individually controlled.

  Next, a case where pixels that do not reach the high luminance level are included will be described with reference to FIG. 201 is a scene in which the user takes a picture of a person in the room and a car outside the window. When the brightness level is adjusted to a person in the room, the outside of the window is very bright, so almost all pixels of the image sensor corresponding to the outside of the window are saturated. However, a portion of the subject to be photographed outside the window that has a dark color such as black is low in reflectance, so that some of the image sensors that correspond to outside the window may not be saturated.

  Reference numeral 202 is an enlarged view of the image sensor at that time. The pixel of the image sensor does not reach the high luminance level in the portion of the material having low reflectance such as the tire 203. In this situation, when only the pixels that have reached the high luminance level are set to the high saturation mode, the pixels other than the dark tire are in the high saturation mode, and the output level is lowered and appears dark. However, on the other hand, since the output level is maintained in the normal mode, the pixels in the tire portion appear relatively bright, and the overall scenery outside the window looks uncomfortable.

(High brightness area setting)
Therefore, the high brightness area setting unit 406 sets the high brightness area by assuming that the pixels surrounded by the pixels determined to have high brightness in the upper, lower, left, and right directions are high brightness even if they do not reach the high brightness level. . Then, information is sent to the timing generator (TG) so as to change the high luminance region to the saturation mode.

  The manner in which the high brightness area setting unit 406 sets the high brightness area will be described with reference to the flowcharts of FIGS. The coordinates of the pixel determined in step S300 (hereinafter referred to as a determination pixel) are initialized to (0, 0). In step S301, it is determined whether the determination pixel has reached a high luminance level. If step S301 is YES, the mode is changed to the high saturation mode in step S318. When step S301 is NO, the process proceeds to process blocks 323, 324, 325, and 326. Here, the processing blocks 323, 324, 325, and 326 are high luminance level confirmation processing blocks on the left side, right side, upper side, and lower side of the determination pixel, respectively.

  First, the flow of the processing block 323 will be described. In step S302, the coordinates of the investigation pixel are initialized. The investigation pixels are pixels on the upper, lower, left, and right sides of the determination pixels, and it is determined whether or not the determination pixels are to be in a high saturation mode depending on whether these inspection pixels have reached a high luminance level.

  In step S303, the left side of the current survey pixel coordinate is set as the next survey pixel coordinate. In step S304, it is confirmed whether the investigation pixel coordinate is a certain distance away from the determination pixel or is an end pixel. If YES here, it is determined that there is no pixel that has reached the high luminance level on the left side, and the process proceeds to step S319. If the investigation pixel coordinate is not more than a certain distance from the determination pixel and is not an end pixel, it is determined as NO and the process proceeds to step S305.

  In step S305, it is determined whether the investigation pixel has reached the high luminance level. If not, the process returns to step S303. If the survey pixel has reached the high luminance level, the process proceeds to process block 324.

  Next, the flow of the processing block 324 will be described. In step S306, the coordinates of the investigation pixel are initialized.

  In step S307, the right side of the current survey pixel coordinate is set as the next survey pixel coordinate. In step S308, it is confirmed whether the investigation pixel coordinate is a certain distance or more than the determination pixel or an end pixel. If YES here, it is determined that there is no pixel that has reached the high luminance level on the right side, and the process proceeds to step S319. If the investigation pixel coordinate is not more than a certain distance from the determination pixel and is not an end pixel, it is determined as NO, and the process proceeds to step S309.

  In step S309, it is determined whether the investigation pixel has reached the high luminance level. If not, the process returns to step S307. If the survey pixel has reached the high luminance level, the process proceeds to process block 325.

  Next, the flow of the processing block 325 will be described. In step S310, the coordinates of the investigation pixel are initialized.

  In step S311, the upper side of the current survey pixel coordinate is set as the next survey pixel coordinate. In step S312, it is confirmed whether the investigation pixel coordinate is a certain distance away from the determination pixel or is an end pixel. Here, if YES, it is determined that there is no pixel that has reached the high luminance level on the upper side, and the process proceeds to step S319. If the investigation pixel coordinate is not more than a certain distance from the determination pixel and is not an end pixel, it is determined as NO and the process proceeds to step S313.

  In step S313, it is determined whether the investigation pixel has reached the high luminance level. If not, the process returns to step S311. If the survey pixel has reached the high luminance level, the process proceeds to process block 326.

  Next, the flow of the processing block 326 will be described. In step S314, the coordinates of the investigation pixel are initialized.

  In step S315, the lower side of the current survey pixel coordinate is set as the next survey pixel coordinate. In step S316, it is confirmed whether the investigation pixel coordinate is a certain distance or more than the determination pixel or an end pixel. Here, if YES, it is determined that there is no pixel that has reached the high luminance level on the lower side, and the process proceeds to step S319. If the survey pixel coordinate is not more than a certain distance from the determination pixel and is not an end pixel, it is determined as NO, and the process proceeds to step S317.

  In step S317, it is determined whether the investigation pixel has reached the high luminance level. If not, the process returns to step S315. If the survey pixel has reached the high luminance level, the process proceeds to step S318.

  In step S318, after the determination pixel is changed to the high saturation mode, the process proceeds to step S319. In step S319, the determination pixel coordinate is set to one right.

  Next, in step S320, it is confirmed whether the determination pixel is the right end. If it is the right end, the process proceeds to step S321. If it is not the right end, the process proceeds to step S301, and the determination is started again.

  In step S321, the determination pixel coordinate is set to the pixel at the left end and one row below. In step S322, it is confirmed whether the determination pixel is the lower end. If it is the lower end, the flow is terminated. If it is not the lower end, the process proceeds to step S301, and the determination is started again.

  In steps S304, 308, 312, and 316, the number of pixels within a certain interval from the determination pixel is set as a threshold value. In this embodiment, for example, the number of pixels of 200 pixels or more from the determination pixel is set as the threshold value.

  This threshold value can be changed from the outside, and can be changed according to the situation and application.

  According to the processing steps described above, in the high brightness area setting unit 406, a pixel sandwiched between the top and bottom and the left and right by a pixel that has reached the high brightness level is a high brightness pixel even if the pixel has not reached the high brightness level. Simulated and changed to high saturation mode. As a result, when a high brightness area is set and an indoor person and a vehicle outside the window are photographed, even if a low reflectance portion such as a tire does not reach the high brightness level, the mode is changed to the high saturation mode. . As a result, the overall scenery outside the window looks uncomfortable.

  In the processing steps described above, it has been described that a pixel sandwiched between the upper, lower, left, and right sides of a pixel that has reached a high luminance level is regarded as a high luminance pixel and is changed to a high saturation mode. As a method of sandwiching, a pixel that is sandwiched between the upper and lower sides or the left and right of pixels with a high luminance level may be changed to the high saturation mode.

  In addition to the processing steps shown above, there are the following threshold setting methods.

  In the pixels arranged in a matrix in the image sensor 402, it is assumed that there is an interval region in which pixels that separate the pixels from the high luminance region in which pixels that have reached a high luminance level are aggregated. When this space region separates the high brightness region at a constant interval with respect to the vertical or horizontal axis direction of the matrix plane, the pixels in this space region are changed to the high saturation mode even if they do not reach the high brightness level. To do. The interval area may be separated from the high brightness area in the vertical axis direction and the horizontal axis direction.

  Here, the constant interval is the value of R1 in Equation 1 when the width in the vertical axis direction or the horizontal axis direction in the high luminance region is X and the width corresponding to the coaxial direction in the interval region is X ′. Is 1 or less. At this time, the interval region is regarded as a high luminance region. R1 is preferably 0.5, and more preferably 0.2.

  As a result, when a high-intensity area is set and an indoor person and a car outside the window are photographed, even if the low-reflectance part such as a tire does not reach the high-intensity level, it is quickly changed to the high-saturation mode. Is done. As a result, the overall scenery outside the window looks uncomfortable.

  In addition, when the space area is formed in a donut shape and all of the top, bottom, left, and right are surrounded by a high brightness area, the sum of the top and bottom or left and right widths of the high brightness area is X, and the space area corresponds to the same direction. Let X ′ be the width to be performed. When the value of R1 in Equation 1 is 1 or less, the interval region may be regarded as a high luminance region.

  Further, when the interval region is surrounded by the high luminance region in all directions, the area of the high luminance region is Y, and the area of the interval region is Y ′. When the value of R2 in Expression 2 is 1 or less, the interval region may be regarded as a high luminance region.

  Thereby, determination to a high saturation mode can be performed by area ratio. For example, when shooting indoors and the scenery outside the window, there is a long gap between the buildings in the scenery outside the window. If the value of R2 is 1 or less, it can be regarded as a high luminance region even if it is narrower than the width of. In addition, when taking a picture of a person outside the window with an indoor person, if the value of R2 is larger than 1, even if only a window frame that is narrower and narrower than the width of the scenery outside the window shines, the scenery outside the window appears. The change to the high saturation mode can be prevented.

  In the present embodiment, the scene in which the user photographs a person inside the room and the car outside the window is exemplified. However, unlike Patent Document 1, it is not necessary to photograph images with different exposure times with a time difference. Needless to say, it can be used for video recording.

  As described above, according to the present embodiment, it is possible to capture moving images of moving objects and the like, and it is possible to capture images with a wide dynamic range without impairing the color balance of a high-luminance subject.

104 Floating diffusion (FD)
105 Extended FD
108 Photodiode (PD)

Claims (8)

An imaging device,
An image sensor in which pixels including a light receiving element and a changing unit that changes a capacitance of a charge storage unit of the light receiving element are arranged in a matrix plane;
A setting unit that sets a region in which the pixel has reached a threshold value from the information of the image sensor as a high luminance region;
With
The changing unit changes a capacity of a charge storage unit of the light receiving element of the imaging element set in the high luminance region,
The setting unit has a case in which there is an interval region separating the high-intensity region, and the interval region separates the high-intensity region at a constant interval with respect to a vertical axis direction or a horizontal axis direction of the matrix plane. In this case, the interval area is also set as the high brightness area.
An imaging apparatus characterized by that. The setting unit sets the spacing area as the high brightness area when the spacing area separates the high brightness area at a constant interval with respect to a vertical axis direction and a horizontal axis direction of the matrix plane. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized. The constant interval is R1 in Formula 1 when the width of the high luminance region in the vertical axis direction or the horizontal axis direction is X and the width corresponding to the coaxial direction in the interval region is X ′. The imaging apparatus according to claim 1, wherein when the value of is less than or equal to 1, the interval area is also set to the high-luminance area.

  The imaging apparatus according to claim 3, wherein when the value of R1 is 0.5, the interval area is also set as the high-luminance area.   The imaging apparatus according to claim 3, wherein when the value of R1 is 0.2, the interval area is also set to the high luminance area. The setting unit sets the spacing area as the high brightness area when the spacing area is surrounded by the high brightness area in all directions.
The imaging apparatus according to claim 1, wherein: The setting unit is a case where the interval region is surrounded by the high luminance region in all directions, and when the area of the high luminance region is Y and the area of the interval region is Y ′ When the value of R2 of 2 is 1 or less, the interval region is also set as the high luminance region.
The imaging apparatus according to claim 1, wherein:

The setting unit is a case where the spacing area is surrounded by the high brightness area in all directions, and the sum of the top and bottom or left and right widths of the high brightness area is X, and the spacing area is the same. When the width corresponding to the direction is X ′, when the value of R1 in Equation 1 is 1 or less, the interval region is also set as the high luminance region.
The imaging apparatus according to claim 1, wherein:

Images