JP2019220945A - Image pick-up device and imaging apparatus - Google Patents

Abstract

An object of the present invention is to acquire a highly effective high dynamic range image while suppressing deterioration of image quality due to afterimages. A first pixel group included in a pixel portion is alternately exposed with a first exposure time and a second exposure time shorter than the first exposure time in a predetermined first cycle. , the first image signal and the second image signal are read out through the first output channel, and the second pixel group included in the pixel portion is read out with the second period longer than the first period. and reading out the third image signal through the second output channel. At that time, reading of the third image signal and reading of the first or second image signal are performed in parallel. [Selection drawing] Fig. 7

Description

  The present invention relates to an imaging device and an imaging device.

  2. Description of the Related Art In recent years, a technique called high dynamic range (HDR) synthesis has been proposed as a technique for obtaining an image with a wider dynamic range using an image sensor formed of a CCD or a CMOS. In this HDR combination, an image with less whiteout and an image with less blackout are obtained by photographing a plurality of images under different exposure conditions, and by combining the properly exposed portions in each image, the whiteout is reduced. Black crushing is suppressed, and an image having good gradation from a dark portion to a bright portion can be obtained.

  In addition, when performing moving image HDR shooting in which HDR synthesis is applied to moving image shooting, it can be realized, for example, by alternately repeating shooting with different exposure for each frame of the moving image.

  FIG. 13A shows that the over-exposure image and the under-exposure image (L) are alternately acquired by switching over-exposure and under-exposure for each frame, and the two images are combined to obtain the HDR effect. This shows the timing of shooting a moving image with a shadow.

  However, photographing a moving subject using this method may cause image quality degradation such as an afterimage in a composite image due to the accumulation time difference between frames.

  On the other hand, Patent Literature 1 proposes a technique of generating an HDR image from the same frame by obtaining image signals of two different exposure times for each line of an image sensor.

  FIG. 13B shows a case in which the overexposed image (H) and the underexposed image (L) are sorted and output in line units. As a result, images with different exposures can be obtained in the same frame, so that the image quality does not deteriorate due to the afterimage.

JP 2010-279016 A

  However, according to the method described in Patent Document 1, the exposure step between the overexposed image and the underexposed image can be made only within one frame period (16.6 ms at 60 fps), and the HDR effect is weakened. In order to further enhance the HDR effect, it is desirable to combine more images with different exposures.

  For example, it is possible to enhance the HDR effect by combining three images by adding an appropriately exposed image, rather than combining only two of the underexposed image and the overexposed image.

  However, when trying to acquire images with three different exposure times using the method described in Patent Document 1, the number of lines constituting one image decreases, and the image quality deteriorates. is there.

  The present invention has been made in view of the above problems, and has as its object to suppress deterioration of image quality due to an afterimage and obtain a higher-quality high dynamic range image.

  In order to achieve the above object, an image sensor according to the present invention includes a pixel including a first pixel group including a plurality of pixels and a second pixel group including a plurality of pixels different from the first pixel group. Unit, a first output channel for outputting an image signal obtained from the first pixel group, a second output channel for outputting an image signal obtained from the second pixel group, and the first pixel The group is exposed alternately for a first exposure time and a second exposure time shorter than the first exposure time at a predetermined first cycle, and the first group is exposed via the first output channel. And the second image signal is read out, and the second pixel group is exposed at a second cycle longer than the first cycle and at a third exposure time longer than the first exposure time. Reading out a third image signal through the second output channel, A 3 and the image signal of the read, and a driving means for driving the first driving method performed in parallel with reading of the first or second image signal.

  According to the present invention, it is possible to suppress deterioration of image quality due to an afterimage and obtain a high-quality high dynamic range image.

FIG. 1 is a diagram illustrating a configuration of an image sensor according to an embodiment of the present invention. FIG. 4 is a timing chart showing output timing of a sensor according to the first embodiment. FIG. 4 is a diagram illustrating an example of an addition thinning pattern according to the first embodiment. FIG. 3 is a diagram illustrating a configuration example of a pixel and a column ADC block according to the embodiment. FIG. 1 is a block diagram illustrating a schematic configuration of an imaging device according to an embodiment. 5 is a flowchart illustrating shooting control in a moving image HDR mode according to the first embodiment. FIG. 4 is a timing chart showing exposure and readout timings according to the first embodiment. 9 is a flowchart illustrating shooting control in a moving image HDR mode according to the second embodiment. FIG. 9 is a timing chart showing exposure and readout timings according to the second embodiment. 13 is a flowchart illustrating shooting control in a moving image HDR mode according to the third embodiment. FIG. 10 is a timing chart showing exposure and readout timings according to the third embodiment. FIG. 13 is a diagram illustrating an exposure control pattern according to the third embodiment. 7 is a timing chart for explaining conventional HDR moving image shooting.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, and the like of the components exemplified in the present embodiment should be appropriately changed depending on the configuration of the apparatus to which the present invention is applied and various conditions. However, the present invention is not limited to the example.

  FIG. 1A is a block diagram illustrating a configuration of an image sensor 506 according to an embodiment of the present invention, and FIG. 1B is a schematic diagram illustrating an appearance of the image sensor 506. As shown in FIG. 1B, the image pickup device 506 is formed of a first semiconductor chip 10 (image pickup layer) and a second semiconductor chip 11 (circuit layer), and the second semiconductor chip 11 and the first semiconductor chip 11 are connected to each other. The semiconductor chip 10 and the semiconductor chip 10 are stacked.

  In the first semiconductor chip 10, a pixel portion including a plurality of pixels 101 arranged in a matrix is arranged on a light incident side, that is, on a light receiving side of an optical image. Pixels 101 arranged in a matrix on the first semiconductor chip 10 are connected to a transfer signal line 103, a reset signal line 104, and a row selection signal line 105 in a row unit in the horizontal direction (row direction). . On the other hand, in the vertical direction (column direction), it is connected to the column output line 102a or 102b. The pixels 101 connected to the column output line 102a are referred to as a first pixel group, and the pixels 101 connected to the column output line 102b are referred to as a second pixel group.

  On the second semiconductor chip 11, column ADC blocks 111a and 111b provided in each column, and pixel driving circuits such as a row scanning circuit 112, column scanning circuits 113a and 113b, and a timing control circuit 114 are formed. . Further, a changeover switch 116, a frame memory 117, an in-element operation unit 118, a resize conversion unit 119, and a parallel / serial (P / S) conversion unit 120 are also formed.

  As described above, by forming the pixel 101 on the first semiconductor chip 10 and forming the pixel driving circuit, the memory circuit, the arithmetic circuit, and the like on the second semiconductor chip 11, the imaging layer and the circuit layer of the imaging element 506 can be formed. Can separate the manufacturing process. As a result, it is possible to increase the speed, reduce the size, and enhance the function of the circuit layer by thinning and increasing the density of the wiring.

  In the pixel 101 of the image sensor 506, charge accumulation and readout are controlled by a control signal from the row scanning circuit 112 via the transfer signal line 103, the reset signal line 104, and the row selection signal line 105. Then, a signal is read from the pixels 101 in the row selected by the row scanning circuit 112. In the present embodiment, there are two output channels as output channels.

  The first output channel includes a column ADC block 111a, a column scanning circuit 113a, and a signal line 115a, and the second output channel includes a column ADC block 111b, a column scanning circuit 113b, and a signal line 115b. Thus, signals can be read from the pixels 101 for two rows in parallel.

  The signal read from the pixel 101 of the first pixel group is sent to the column ADC block 111a of the first output channel via the column output line 102a of each column, and is subjected to A / D conversion. The signal read from the pixel 101 of the second pixel group is sent to the column ADC block 111b of the second output channel via the column output line 102b of each column, and is subjected to A / D conversion.

  Thereafter, the columns to be read are sequentially selected by the column scanning circuit 113a or 113b, and the A / D-converted image signal is output to the changeover switch 116 via the signal line 115a or 115b. Note that it is also possible to use one of the first and second output channels to read a signal from the pixel 101 every other row.

  The timing control circuit 114 sends a timing signal to the row scanning circuit 112, the column ADC blocks 111a and 111b, and the column scanning circuits 113a and 113b based on the control from the overall control calculation unit 509.

  The changeover switch 116 is a switch for selectively outputting the image signals output from the signal lines 115a and 115b to the frame memory 117 sequentially. The frame memory 117 temporarily stores the output image signal as image data.

  The in-element operation unit 118 performs an operation such as a compression process of image data according to the drive mode. The resize conversion unit 119 resizes the image data stored in the frame memory 117 to a required angle of view based on the result calculated by the in-element operation unit 118, and sends the image data to the P / S conversion unit 120.

  When the resizing process and the compression process are unnecessary, the transfer from the changeover switch 116 to the P / S converter 120 is performed directly. The P / S converter 120 performs parallel-to-serial conversion on the image data, and sends the image data to an image signal processing circuit 507 outside the image sensor 506.

  The image sensor 506 and the image signal processing circuit 507 are connected by a plurality of lanes. For example, the main stream 121 is composed of four Lanes 0-3, and the sub-stream 122 is composed of four Lanes 4-7. Shall be. Then, signals of different pixels or signals of the same pixel are distributed to the main stream 121 and the sub-stream 122 or output only from the main stream 121 according to the drive mode.

  FIG. 2 is a timing chart showing the drive timing of the image sensor 506. State A is a state in which only four Lanes 0 to 3 of the main stream 121 operate and one frame data is output from the four lanes. State B is a state in which the main stream 121 and the sub-stream 122 of the image sensor 506 operate simultaneously.

  When the drive mode of the image sensor 506 is set to the “moving image HDR mode”, a vertical synchronization signal XVS is input to the image sensor 506 from the overall control calculation unit 509 described below, and the state A and the state B are changed for each frame ( It operates while alternately switching (at the first cycle). Then, the image sensor 506 sorts and outputs the three types of images obtained by shooting with proper exposure in addition to overexposure and underexposure, into the main stream 121 and the substream 122. The data output here is data for a moving image, and a signal is output from the added / decimated pixel 101.

FIG. 3 is a diagram illustrating an example of an addition pattern. When the pixel 101 is covered by the filter of the Bayer array, for four pixels in the horizontal and vertical directions,
R ′ = (3 × R1 + R2 + 1.5 × R3 + 0.5 × R4) / 6
Gr ′ = (Gr1 + 3 × Gr2 + 0.5 × Gr3 + 1.5 × Gr4) / 6
Gb ′ = (1.5 × Gb1 + 0.5 × Gb2 + 3 × Gb3 + Gb4) / 6
B ′ = (0.5 × B1 + 1.5 × B2 + B3 + 3 × B4) / 6
The signal added at the ratio is output.

  For example, in the case of 5472 horizontal pixels and 3648 vertical pixels, the number of pixels in the horizontal direction is reduced to ２７, 2736 pixels, the number of pixels in the vertical direction is reduced to 且 つ, and upper and lower pixels are cut. The output signal of 1538 pixels is output.

  FIG. 4 is a diagram illustrating a configuration of one pixel 101 of the image sensor 506 and details of one column ADC block 111 according to the present embodiment. The outline of the operation of the image sensor 506 according to the embodiment of the present invention will be described with reference to FIGS. Since the column ADC block 111a and the column ADC block 111b have the same configuration, they are described as the column ADC block 111 in FIG.

  In FIG. 4, a photodiode (PD) 201 included in a pixel 101 photoelectrically converts received light into photocharges (here, electrons) having a charge amount corresponding to the light amount. The transfer transistor 202 is connected between the cathode of the PD 201 and the gate of the amplification transistor 204, and is turned on when a transfer pulse φTRG is applied to the gate via the transfer signal line 103. A node that is electrically connected to the gate of the amplification transistor 204 forms a floating diffusion (FD) unit 206. When the transfer transistor 202 is turned on by the transfer pulse φTRG, the photocharge photoelectrically converted by the PD 201 is transferred to the FD unit 206.

  The reset transistor 203 is turned on when the drain is connected to the pixel power supply Vdd and the source is connected to the FD unit 206, and the gate is supplied with a reset pulse φRST via the reset signal line 104. Then, prior to the transfer of the photocharge from the PD 201 to the FD unit 206, the charge of the FD unit 206 is reset to the pixel power supply Vdd by turning on the reset transistor 203.

  The amplification transistor 204 has a gate connected to the FD unit 206 and a drain connected to the pixel power supply Vdd. The selection transistor 205 is turned on when the drain is connected to the source of the amplification transistor 204 and the source is connected to the column output line 102, and the selection pulse φSEL is applied to the gate via the row selection signal line 105. .

  While the selection transistor 205 is on, first, the potential of the FD section 206 after being reset by the reset transistor 203 is output to the column output line 102 as a reset level. Further, by turning on the transfer transistor 202, the potential of the FD section 206 after the transfer of the photocharge is output to the column output line 102 as a signal level. In this embodiment, N-channel MOS transistors are used as the transistors 202 to 205.

  Note that the configuration of the pixel 101 is not limited to the above configuration. For example, a circuit configuration in which the selection transistor 205 is connected between the pixel power supply Vdd and the drain of the amplification transistor 204 may be employed. Further, the present invention is not limited to the above-described four-transistor configuration, and may be, for example, a three-transistor configuration in which the amplifying transistor 204 and the selection transistor 205 are also used.

  A signal output from the pixel 101 via the column output line 102 is transmitted to the column ADC block 111. The column ADC block 111 includes a comparator 211, an up / down counter (U / D CNT) 212, a memory 213, and a DA converter (DAC) 214.

  In the comparator 211, one of the pair of input terminals is connected to the column output line 102, and the other is connected to the DAC 214. The DAC 214 outputs a ramp signal whose signal level changes in a ramp shape with the passage of time based on the reference signal input from the timing control circuit 114.

  The comparator 211 compares the level of the ramp signal input from the DAC 214 with the level of the signal input from the column output line 102. The timing control circuit 114 outputs a reference signal to the DAC 214 based on a command from the overall control calculation unit 509 described below.

  For example, the comparator 211 outputs a high-level comparison signal when the level of the image signal is lower than the level of the ramp signal, and outputs a low-level comparison signal when the level of the image signal is higher than the level of the ramp signal. Output. The up / down counter 212 is connected to the comparator 211, and counts, for example, a period in which the comparison signal is at a high level or a period in which the comparison signal is at a low level.

  By this counting process, the output signal of each pixel 101 is converted into a digital value. Note that an AND circuit may be provided between the comparator 211 and the up / down counter 212, a pulse signal may be input to the AND circuit, and the number of the pulse signals may be counted by the up / down counter 212.

  The memory 213 is connected to the up / down counter 212 and stores the count value counted by the up / down counter 212. The column ADC block 111 counts the count value corresponding to the reset level of the pixel 101, then counts the count value corresponding to the signal level after a predetermined imaging time, and stores the difference value in the memory 213. May be. Thereafter, the count value stored in the memory 213 is transmitted as a digital image signal to the changeover switch 116 via the signal line 115a or 115b in synchronization with the signal from the column scanning circuit 113.

  Note that the column ADC block 111 is not limited to the above-described configuration, and it goes without saying that a known column ADC may be used.

  FIG. 5 is a block diagram illustrating a schematic configuration of an imaging apparatus having the imaging element 506 described with reference to FIGS. In FIG. 5, a subject image that has passed through a lens unit 501 and a mechanical shutter (hereinafter, referred to as a “shutter”) 503 is adjusted to an appropriate amount of light by an aperture 504, and is formed on an imaging surface of an imaging element 506. .

  The subject image formed on the imaging surface of the image sensor 506 is photoelectrically converted by the PD 201 as described above, and further subjected to analog-to-digital conversion (A / D conversion) for converting an analog signal to a digital signal, thereby obtaining an image signal. The data is sent to the processing circuit 507.

  The imaging signal processing circuit 507 performs various image signal processing such as low-pass filter processing, shading processing, and WB processing for reducing noise, various correction processing, and compression processing of image data.

  The driving of the lens unit 501 is controlled by the lens driving unit 502 to perform zooming, focusing, and the like. The shutter 503 and the aperture 504 are driven and controlled by a shutter / aperture drive unit 505. The overall control calculation unit 509 controls the entire imaging device and performs various calculations. The first memory unit 508 temporarily stores image data.

  A recording medium control interface (I / F) unit 510 records or reads image data on or from a recording medium 512. The display unit 511 displays image data. The recording medium 512 is a removable storage medium such as a semiconductor memory, and stores the written image data. The external interface (I / F) unit 513 is an interface for communicating with an external computer or the like.

  The second memory unit 514 stores the calculation result of the overall control calculation unit 509. Information about the driving conditions of the imaging device set by the user on the operation unit 515 is sent to the overall control calculation unit 509, and the entire imaging device is controlled based on the information. Note that the lens unit 501 may be detachable from the imaging device.

  Hereinafter, control in the imaging device having the above configuration will be described.

<First embodiment>
The drive control of the imaging device according to the first embodiment of the present invention will be described.

  FIG. 6 is a flowchart illustrating shooting control in the moving image HDR mode according to the first embodiment, and FIG. 7 is a timing chart illustrating exposure and readout timings according to the first embodiment. Hereinafter, the shooting control in the moving image HDR mode according to the first embodiment will be described with reference to FIGS. 6 and 7.

  When the moving image HDR mode is selected by the imaging device, in S101, the drive mode of the image sensor 506 is switched to the moving image HDR mode. Next, in S102, a frame number N for counting the number of frames is initialized (N = 0). After the frame number N is initialized, it is determined in S103 whether the current frame number N is an even number. If the frame number N is even, the process proceeds to S104. Here, 0 is treated as an even number.

  In step S <b> 104, under the control of the overall control calculation unit 509, an appropriately exposed image (M image) 901 is obtained from the first pixel group of the image sensor 506, and is output to the image signal processing circuit 507 via the main stream 121. Predetermined image signal processing is performed and stored in the first memory unit 508. This frame corresponds to the state A of the image sensor 506 in the moving image HDR mode.

  Next, in step S105, under the control of the overall control calculation unit 509, the appropriate exposure time is set as the exposure time of the image acquired from the first pixel group of the image sensor 506 and output via the main stream 121 in the next frame. (M) is set to the image sensor 506.

  Further, in step S106, under the control of the overall control calculation unit 509, the overexposure time (overexposure time) is set as the exposure time of the image acquired from the second pixel group of the image sensor 506 and output via the substream 122 in the next frame. H) is set to the image sensor 506.

  On the other hand, if the current frame number N is an odd number in S103, the process proceeds to S107.

  In step S107, under the control of the overall control calculation unit 509, an underexposed image (L image) 902 is obtained from the first pixel group of the image sensor 506, and output to the image signal processing circuit 507 via the main stream 121. A predetermined image signal processing is performed, and in S108, under the control of the overall control arithmetic unit 509, an overexposed image (H image) 903 is obtained from the second pixel group of the image sensor 506, and the acquired image is transmitted via the sub-stream 122. The image data is output to the imaging signal processing circuit 507 and subjected to predetermined image signal processing, and is stored in the first memory unit 508. This control of the frame corresponds to the state B of the image sensor 506 in the moving image HDR mode.

  Next, in S109, under the control of the overall control calculation unit 509, the under exposure time is set as the exposure time of the image acquired from the first pixel group of the image sensor 506 and output via the main stream 121 in the next frame. (L) is set to the image sensor 506. Note that the images output from the main stream 121 and the sub-stream 122 may be exchanged.

  Next, in S110, the overall control calculation unit 509 determines whether or not three images (M image 901, L image 902, and H image 903) having different exposures have been collected in the first memory unit 508, and if so, they are aligned. If so, proceed to S111. If not, the process proceeds to S113.

  In the first embodiment, the case where the exposure time is switched has been described. However, in addition to the exposure time, the ISO sensitivity and the aperture may be changed together.

  In S111, HDR synthesis processing is performed. The overall control calculation unit 509 determines, for example, a pixel level predetermined for each of an underexposed image (L image), a properly exposed image (M image), and an overexposed image (H image) from pixel data constituting each image. Pixel data corresponding to the range (L, M, H) is extracted and combined. As a result, the HDR image data (synthesized image) 904 with an expanded dynamic range is generated.

  Note that the synthesis method is not limited to this, and the synthesis may be performed using a known method. The overall control calculation unit 509 stores the generated HDR image data 904 in the first memory unit 508. At this time, the underexposed image (L image), the properly exposed image (M image), and the overexposed image (H image) used for the HDR image generation are deleted from the first memory unit 508.

  Next, in S112, the HDR image data 904 is encoded as necessary, and then recorded on the recording medium 512 in a predetermined image data file format. Further, the overall control calculation unit 509 also generates image data for display, and outputs it to the display unit 511. The display unit 511 converts the HDR image data into a signal suitable for a display device and displays the signal.

  In S113, the frame number N is counted up, and in S114, it is determined whether or not the selection of the moving image HDR mode is canceled by the imaging device. If not, the process returns to S103 and repeats until the cancellation. If it has been released, the process is completed.

  As described above, according to the first embodiment, the properly exposed image and the underexposed image acquired from the first pixel group of the image sensor 506 are alternately output from the main stream for each frame. The overexposed image acquired from the two pixel groups is acquired from the substream every two frames. This makes it possible to acquire three images with different exposures in a time corresponding to two frames.

  In this way, by controlling the exposure and reading so that a plurality of different exposure times can be acquired with a smaller number of frames, an HDR moving image having a high HDR effect with an exposure step is created while suppressing the effects of afterimages. It becomes possible.

<Second embodiment>
Next, a second embodiment of the present invention will be described. In the first embodiment described above, a properly exposed image is first obtained from the first pixel group of the image sensor 506 and output from the main stream 121, and then an underexposed image is obtained and output. On the other hand, in the second embodiment, a case will be described in which the order of the properly exposed image and the underexposed image obtained from the first pixel group of the image sensor 506 and output from the main stream 121 are interchanged.

  FIG. 8 is a flowchart illustrating shooting control in the moving image HDR mode according to the second embodiment, and FIG. 9 is a timing chart illustrating exposure and readout timings according to the second embodiment. Hereinafter, shooting control in the moving image HDR mode according to the second embodiment will be described with reference to FIGS. 8 and 9. The processes from S101 to S103 are the same as those in the first embodiment, and thus the description is omitted.

  If the current frame number N is an even number in S103, the process proceeds to S107. In step S107, under the control of the overall control calculation unit 509, an underexposed image (L image) 902 is obtained from the first pixel group of the image sensor 506 and output to the image signal processing circuit 507 via the main stream 121. Then, predetermined image signal processing is performed, and the result is stored in the first memory unit 508. This frame corresponds to the state A of the image sensor 506 in the moving image HDR mode.

  Next, in S109, under the control of the overall control calculation unit 509, the under exposure time is set as the exposure time of the image acquired from the first pixel group of the image sensor 506 and output via the main stream 121 in the next frame. (L) is set to the image sensor 506.

  Further, in step S106, under the control of the overall control calculation unit 509, the overexposure time (overexposure time) is set as the exposure time of the image acquired from the second pixel group of the image sensor 506 and output via the substream 122 in the next frame. H) is set to the image sensor 506.

  On the other hand, if the current frame number N is an odd number in S103, the process proceeds to S104.

  In step S <b> 104, under the control of the overall control calculation unit 509, an appropriately exposed image (M image) 901 is obtained from the first pixel group of the image sensor 506, and is output to the image signal processing circuit 507 via the main stream 121. A predetermined image signal processing is performed, and in S108, under the control of the overall control arithmetic unit 509, an overexposed image (H image) 903 is obtained from the second pixel group of the image sensor 506, and the acquired image is transmitted via the sub-stream 122. The image data is output to the imaging signal processing circuit 507 and subjected to predetermined image signal processing, and is stored in the first memory unit 508. This control of the frame corresponds to the state B of the image sensor 506 in the moving image HDR mode.

  Next, in step S105, under the control of the overall control calculation unit 509, the appropriate exposure time is set as the exposure time of the image acquired from the first pixel group of the image sensor 506 and output via the main stream 121 in the next frame. (M) is set to the image sensor 506. Note that the images output from the main stream 121 and the sub-stream 122 may be exchanged.

  The processing after S111 is the same as the processing in the first embodiment, and a description thereof will be omitted.

  As described above, according to the second embodiment, an underexposed image is obtained from the first pixel group of the image sensor 506, a properly exposed image is obtained, and the image is alternately output from the main stream for each frame. Thus, the time difference between the underexposed image and the properly exposed image can be reduced.

  Compared to FIG. 7 in the first embodiment, in the second embodiment, as shown in FIG. 9, the time difference between the exposure timing of the underexposed image and the exposure timing of the properly exposed image is shorter than that in FIG. Has become.

  As a result, the afterimage feeling due to the combining process when capturing the moving object is further suppressed, and it is possible to create a higher-quality HDR moving image.

<Third embodiment>
Next, a third embodiment of the present invention will be described. In the first and second embodiments described above, a properly exposed image and an underexposed image are obtained from the first pixel group of the image sensor 506, output from the main stream 121, and output from the second pixel group of the image sensor 506. The example in which the overexposed image is obtained and output from the substream 122 has been described. On the other hand, in the third embodiment, a method of dynamically switching images to be acquired according to conditions to reduce power consumption will be described.

  FIG. 10 is a flowchart illustrating shooting control in the moving image HDR mode according to the third embodiment. FIG. 11 is a timing chart illustrating exposure and readout timings according to the third embodiment. Hereinafter, the shooting control in the moving image HDR mode according to the third embodiment will be described with reference to FIGS. 10 and 11. In FIG. 10, the same control as that in FIG. 6 is denoted by the same step number, and the description is omitted as appropriate.

  When the moving image HDR mode is selected by the imaging device, in S101, the drive mode of the image sensor 506 is switched to the moving image HDR mode. Next, in S303, in order to determine an exposure control method of the image sensor 506, a pattern of a histogram of luminance values of an appropriately exposed image is determined.

  In the third embodiment, it is determined whether the histogram of the luminance value of the captured image corresponds to any one of the patterns A to C in FIG. The present embodiment is not limited to this, and the exposure control method may be determined in consideration of not only the brightness value of the captured image but also the remaining battery capacity.

  Next, in S304, an exposure time is determined according to the determined pattern. The determination of the exposure time is performed using, for example, a table as shown in FIG.

  Specifically, when the distribution of the luminance value is biased toward a dark area (low luminance) as in the pattern A of FIG. 12A, the appropriate exposure time (M) is obtained from the table of FIG. Two types of overexposure time (H) are determined. In this case, since there are few areas in the shooting scene where overexposure is likely to occur, an HDR image having good tone characteristics can be obtained by suppressing black crushing from two images of proper exposure and overexposure.

  On the other hand, when the distribution of luminance values is biased toward a bright region (high luminance) as in the pattern B in FIG. 12A, the appropriate exposure time (M) and the under exposure time (M) are obtained from the table shown in FIG. L) are determined. In this case, since there are few regions in the shooting scene where black is likely to be crushed, it is possible to obtain an HDR image having good gradation characteristics by suppressing overexposure from the two images of proper exposure and underexposure.

  When the luminance distribution has a bipolar distribution in both the bright region and the dark region as in the pattern C of FIG. 12A, the appropriate exposure time (M ), Under-exposure time (L), and over-exposure time (H). As a result, it is possible to suppress blackout and whiteout, and obtain an HDR image having good gradation characteristics.

  Next, in step S305, it is determined whether an image is acquired from the first pixel group of the image sensor 506 and output via the main stream 121. If the image is acquired and output, the process proceeds to step S306. Goes to S307.

  In step S <b> 306, an image is obtained from the first pixel group of the image sensor 506 under the control of the overall control calculation unit 509, and output via the main stream 121. This control is executed based on FIG. 12B when the main stream 121 is not in use or is not settable.

  Similarly, in step S307, it is determined whether an image is acquired from the second pixel group of the image sensor 506 and output via the sub-stream 122. If the image is acquired and output, the process proceeds to step S308. Goes to S309.

  In step S308, an image is obtained from the second pixel group of the image sensor 506, output via the sub-stream 122, and the images output from the main stream 121 and the sub-stream 122 are stored in the first memory unit 508, respectively. I do.

  In step S309, it is determined whether to expose the pixels of the first pixel group of the image sensor 506 corresponding to the main stream 121 in the (N + 1) th frame. If the exposure is to be performed, the process proceeds to step S310; otherwise, the process proceeds to step S311.

  In step S310, under the control of the overall control calculation unit 509, the exposure time of an image acquired from the first pixel group of the image sensor 506 and output via the main stream 121 in the next frame is set. This control is executed when the exposure control of the (N + 1) th frame is not unused or not impossible, based on FIG. 12B. The exposure time set here is the exposure time based on the pattern determined in S304. is there.

  Similarly, in S311, it is determined whether to expose the pixels of the second pixel group of the image sensor 506 corresponding to the substream 122 in the (N + 1) th frame. If the exposure is to be performed, the process proceeds to S312, and if not, the process proceeds to S313. .

  In step S312, under the control of the overall control calculation unit 509, the exposure time of an image acquired from the second pixel group of the image sensor 506 and output via the substream 122 in the next frame is set.

  For example, in the case of the pattern A in FIG. 12B, when acquiring two images of the properly exposed image (M) and the overexposed image (H), the timing control as shown in FIG. 11A is performed. . As a result, the control of exposure and reading on the sub-stream side is performed once every two frames, and the output on the sub-stream side is stopped in frames where exposure and reading are not performed, thereby reducing power consumption.

  Similarly, in the case of the pattern B in FIG. 12B, two images of the properly exposed image (M) and the underexposed image (L) are acquired. In this case, the timing control as shown in FIG. 11B is performed, and the output of both the main stream and the substream is stopped once every two times to reduce power consumption.

  In the case of the pattern C in FIG. 12B, the timing control is the same as in the first embodiment (FIG. 11C).

  In step S313, it is determined whether a plurality of exposure images required for the HDR combining process have been obtained. In the third embodiment, the determination is made based on whether the exposure images of the Nth frame and the (N + 1) th frame in FIG. If it is determined that a plurality of necessary exposure images are prepared and can be combined, the process proceeds to S111, and if not, the process returns to S305.

  In step S111, HDR image data is generated by performing HDR synthesis processing. In step S112, the generated HDR image data is encoded as necessary, and then recorded on the recording medium 512 in a predetermined image data file format.

  In S114, it is determined whether or not the selection of the moving image HDR mode has been canceled by the imaging device. If the selection has not been canceled, the process returns to S303 and repeats until the cancellation. If it has been released, the process is completed.

  In the third embodiment, a table as shown in FIG. 12B is stored in advance in the second memory unit 514, but the present invention is not limited to this.

  As described above, according to the third embodiment, the exposure time of the image output in the main stream and the sub-stream is dynamically switched according to the distribution pattern of the signal shot with the proper exposure. As a result, the power consumption can be reduced by stopping the output of the main stream / sub-stream in the unused frames.

  In the example described above, in the case of the pattern of FIG. 11B, the properly exposed image (M) and the underexposed image (L) are read out every other frame. ) And the underexposed image (L). In that case, an HDR moving image with a high temporal resolution can be created.

  In the first to third embodiments, a case has been described in which images with three different exposure times are acquired from two streams for every two frames and HDR synthesis is performed. However, the present invention is not limited to this. There is no. For example, it is needless to say that the present invention is also applicable to a case where images of four or more different exposure times are acquired from three or more streams over three or more frames (multiple frame periods) and are synthesized.

  Further, in each of the above-described embodiments, the case where the present invention is applied to a digital camera has been described as an example, but the present invention is not limited to this example. That is, the present invention may be applied to any device with an image sensor. That is, the present invention is applicable to any device capable of capturing an image, such as a mobile phone terminal, a portable image viewer, a television having a camera, a digital photo frame, a music player, a game machine, and an electronic book reader.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist.

  10: first semiconductor chip, 11: second semiconductor chip, 101: pixel, 102a, 102b: column output line, 111a, 111b: column ADC block 111a, 112: row scanning circuit, 113a, 113b: column scanning circuit , 114: timing control circuit, 506: image sensor, 508: first memory unit, 509: overall control operation unit, 514: second memory unit

Claims (6)

A pixel unit including a first pixel group including a plurality of pixels and a second pixel group including a plurality of pixels different from the first pixel group;
A first output channel for outputting an image signal obtained from the first pixel group;
A second output channel for outputting an image signal obtained from the second pixel group;
Exposing the first pixel group alternately with a first exposure time and a second exposure time shorter than the first exposure time at a predetermined first cycle, Read the first image signal and the second image signal through the second pixel group, and set the second pixel group in a third cycle longer than the first exposure time in a second cycle longer than the first cycle. And the third image signal is read out via the second output channel. The reading of the third image signal and the reading of the first or second image signal are performed in parallel. And a driving means for driving by a first driving method.   The image sensor according to claim 1, wherein the second cycle is a cycle that is a multiple of the first cycle.   The said 1st output channel and the said 2nd output channel each have an analog-to-digital conversion part which converts the image signal obtained from the said 1st pixel group into a digital signal. An image sensor according to claim 1. A first semiconductor chip on which the pixel unit is arranged;
A second semiconductor chip on which the first output channel, the second output channel, and the driving unit are arranged;
The imaging device according to any one of claims 1 to 3, wherein the first semiconductor chip and the second semiconductor chip are electrically connected and stacked. An imaging device according to any one of claims 1 to 4,
Generating means for generating an image having an expanded dynamic range from a plurality of image signals obtained by exposure at a plurality of different exposure times output from the image sensor. A determining unit that determines a driving method based on a luminance distribution of an image signal obtained by exposing for the first exposure time;
The determining means comprises:
When the luminance of the image signal is a first pattern distributed in both high luminance and low luminance, it is determined that the image signal is driven by the first driving method,
When the distribution is a second pattern that is biased toward low luminance, the first pixel group is exposed at the third exposure time in the second cycle, and the first and second pixels are exposed. Reading the third image signal through one of the output channels, exposing the second pixel group at the first exposure time in the second cycle, and The first image signal is read through the other of the output channels, and it is determined that driving is performed by a second driving method in which reading of the third image signal and reading of the first image signal are performed in parallel. ,
When the distribution is a third pattern that is biased toward high luminance, the first pixel group is exposed at the first exposure time in the first or second cycle, and the first and second pixels are exposed. Reading the first image signal through one of the second output channels, exposing the second pixel group at the same cycle as the first pixel group for the second exposure time, The third image signal is read out via the other of the first and second output channels, and the reading of the first image signal and the reading of the second image signal are performed in parallel. Determined to drive by the driving method,
6. The apparatus according to claim 5, wherein the driving unit is driven by a driving method determined by the determining unit.

Images