JP2010074584A - Imaging apparatus and imaging method - Google Patents

Abstract

An object of the present invention is to accurately detect a movement of a subject in a captured image without greatly increasing the circuit scale.
An imaging unit 21 photoelectrically converts light from the outside in units of pixels, and accumulates electric charges obtained as a result in units of pixels. The imaging unit 21 reads a signal corresponding to the accumulated electric charge as a detection signal for detecting a motion of a subject in a captured image that is captured immediately after the imaging time of one frame, non-destructively a plurality of times. . In addition, the imaging unit 21 reads out a signal corresponding to the accumulated charge as a captured image signal for each imaging time of one frame. The present invention can be applied to, for example, an imaging apparatus.
[Selection] Figure 1

Description

  The present invention relates to an image pickup apparatus and an image pickup method, and more particularly to an image pickup apparatus and an image pickup method capable of accurately detecting the movement of a subject in a picked-up image without greatly increasing the circuit scale.

  A CCD (Charge Coupled Device) image sensor or a CMOS (Coplementary Metal-Oxide Semiconductor) image sensor is used as an imaging device for cameras such as digital cameras, video cameras, cameras mounted on mobile phones, and surveillance cameras.

  In recent years, a state-of-the-art miniaturization process has been used in the manufacturing process of these image sensors. By this miniaturization process, the image sensor itself can be reduced in size and the cost can be reduced, or the number of pixels of the image sensor can be increased.

  However, since the pixel size is reduced, the sensitivity to light for photographing is lowered. Therefore, when shooting in a dark place, blur due to camera movement or subject movement (hereinafter referred to as motion blur) within the charge accumulation time of the image sensor appears prominently.

  For example, in a digital still camera, motion blur appears prominently during long-exposure shooting in a night view or in a dark room, and the captured image is greatly degraded by motion blur. Although it depends on the size and weight of the digital still camera itself, this motion blur is noticeable when the exposure time is 1/15 second or longer.

  Also, when shooting a distant view using a telephoto lens, due to the reduction in pixel size, a fine camera movement appears as a large movement on the imaging surface, and the motion blur becomes noticeable.

  Furthermore, due to the reduction in pixel size, the image formed on the imaging surface of the image sensor moves to the surrounding pixels even if the digital still camera moves insignificant, which has not been a problem in normal shooting. And motion blur appears prominently. As a result, the resolution and sharpness of the captured image are reduced. In addition, when the pixel size is reduced to increase the number of pixels, the motion blur becomes more remarkable, and the effect of increasing the number of pixels at the folding angle is halved.

  Also in the video camera, motion blur occurs due to the motion of the video camera between frames. Therefore, when the number of pixels is increased from full HD (High Definition) to 2K × 4K or 4K × 8K, the influence on the peripheral pixels due to the slight movement of the video camera cannot be ignored as in the case of the digital still camera.

  As described above, due to the reduction in the pixel size, the motion blur appears remarkably regardless of the shooting conditions. Therefore, prevention and reduction of motion blur are indispensable when pixel miniaturization proceeds.

  Incidentally, methods for reducing motion blur have been widely studied.

  For example, as a first method, there is a method of correcting a motion by a camera optically and mechanically. In this method, the camera is composed of, for example, a motion sensor such as an acceleration sensor or a velocity sensor and an optical system such as a lens having a movable part. Then, the camera corrects the motion of the stationary object as the subject on the imaging surface by moving the optical system in accordance with the unnecessary motion of the camera detected by the motion sensor.

  This first method exhibits excellent performance if the camera operates properly, but requires a motion sensor and new optical components for motion detection, and is not suitable for downsizing and cost reduction. Also, due to optical and mechanical operations, the performance of following high-speed movement is not good.

  As a second method, there is a method of analyzing motion of a camera to estimate motion blur by analyzing image data for one frame read from the image sensor, and reducing motion blur in terms of image processing. is there. This second method can be easily realized because motion blur can be reduced only by signal processing.

  However, it is impossible to accurately estimate the original motion of the camera from the image data of one frame, and the second method is limited to the extent that motion blur is approximately reduced. In the second method, since real-time performance is not required, processing can be performed offline with a personal computer or the like after shooting.

  Furthermore, as a third method, there is a method in which a camera for detecting the motion of the camera itself is separately provided to enhance the motion blur detection capability (for example, Non-Patent Document 1). In this third method, for example, a total of two cameras, a camera for shooting and a camera for motion detection, are provided, and the motion of the camera is detected by the camera for motion detection, and from the captured image based on the motion. Image processing is performed to reduce motion blur.

  The third method is more effective in reducing motion blur because the motion itself can be detected more accurately than the second method. However, it is necessary to use two cameras, and it is difficult to put it to practical use as an actual product.

  On the other hand, in recent years, a circuit capable of nondestructive reading that reads an image signal corresponding to the charge without destructing the charge accumulated in the image sensor has been devised (for example, see Non-Patent Document 2).

Moshe Ben-Ezra, Shree K. Nayar “Motion-Based Motion Deblurring”, IEEE Transactions of Pattern Analysis and Machine Intelligence, June 2004, Vol. 26, No. 6, p.689698 T. Nakamura, K. Matsumoto, R. Hyuga and A. Yusa “A NEW MOS IMAGE SENSOR OPERATING IN A NON-DESTRUCTIVE READOUT_MODE”, Electron Devices Meeting, 1986 International, p.353-356

  As described above, various methods have been devised for reducing motion blur. However, the motion of a subject in a captured image can be accurately detected and the motion blur can be increased without significantly increasing the circuit scale. A method of reducing the accuracy was not considered.

  The present invention has been made in view of such a situation, and makes it possible to accurately detect the movement of a subject in a captured image without greatly increasing the circuit scale.

  One aspect of the present invention is a photoelectric conversion means for photoelectrically converting light from the outside in units of pixels, an accumulation means for storing charges obtained by the photoelectric conversion means in units of pixels, and within an imaging time of one frame, A detection signal reading unit that reads a signal corresponding to the electric charge accumulated by the accumulation unit as a detection signal for detecting a motion of a subject in a captured image that is imaged immediately thereafter, and a frame of non-destructive reading The imaging apparatus includes: a captured image signal reading unit that reads out a signal corresponding to the charge accumulated by the accumulation unit as a captured image signal every imaging time.

  According to one aspect of the present invention, an imaging device performs photoelectric conversion steps in which external light is photoelectrically converted in units of pixels, an accumulation step in which charges obtained by the processing of the photoelectric conversion steps are stored in units of pixels, and one frame In the imaging time, a signal corresponding to the charge accumulated by the processing of the accumulation step is detected as a detection signal for detecting the motion of the subject in the captured image that is imaged immediately after the detection, and is read out a plurality of times. The imaging method includes a signal readout step and a captured image signal readout step of reading out a signal corresponding to the charge accumulated by the processing of the accumulation step as a captured image signal every imaging time of one frame.

  In one aspect of the present invention, light from the outside is photoelectrically converted in units of pixels, the charges are accumulated in units of pixels, and a signal corresponding to the accumulated charges is immediately after the imaging time of one frame. As a detection signal for detecting the movement of the subject in the captured image to be captured, a signal corresponding to the accumulated charge that is read out non-destructively a plurality of times and stored for each imaging time of one frame is a captured image signal. Is read as

  As described above, according to one aspect of the present invention, it is possible to accurately detect the movement of a subject in a captured image without significantly increasing the circuit scale.

  FIG. 1 is a block diagram showing a configuration example of an embodiment of an imaging apparatus to which the present invention is applied.

  The imaging device 10 in FIG. 1 includes an image sensor 11 that is a non-destructive readable CMOS image sensor and a signal processing unit 12, and a subject in a captured image that is generated when the imaging device 10 or the subject moves within the imaging time. Motion is detected, and motion blur of the captured image is reduced.

  In the image sensor 11, the imaging unit 21 photoelectrically converts light from the outside that has passed through the color filter 21 </ b> A arranged on the mosaic in units of pixels, and accumulates electric charges obtained as a result. The imaging unit 21 is provided with two readout paths, ie, an readout path for imaging and a readout path for motion detection.

  Specifically, the imaging unit 21 outputs a signal corresponding to the charge accumulated in units of pixels within the imaging time of one frame corresponding to the control line 23 selected by the image vertical scanning circuit 22 of the captured image. It reads out as an image signal (hereinafter referred to as a captured image signal). As a result, the captured image signal is read for each scanning line at a predetermined frame rate, and is supplied to the image column parallel ADC 25 via the reading line 24.

  In addition, the imaging unit 21 immediately captures a signal corresponding to the charge accumulated in units of pixels within a time shorter than the imaging time of one frame corresponding to the control line 30 selected by the detection vertical scanning circuit 29. As a detection signal for detecting the movement of the subject in the captured image, the non-destructive readout is performed. As a result, the detection signal is read for each scanning line at a frame rate faster than the frame rate of the captured image, and is supplied to the detection column parallel ADC 32 via the reading line 31.

  The image vertical scanning circuit 22 controls the readout of the captured image signal by the imaging unit 21 by sequentially selecting the control line 23 corresponding to each scanning line. The image column parallel ADC 25 performs A / D (Analog / Digital) conversion on the captured image signal supplied from the imaging unit 21, and reads out the captured image signal obtained as a result via the signal line 26. To supply.

  The readout circuit 27 is a digital circuit, and outputs the captured image signal supplied from the image column parallel ADC 25 to the signal processing unit 12 via the signal line 28.

  The detection vertical scanning circuit 29 controls the reading of the detection signal by the imaging unit 21 by selecting the control line 30 corresponding to the predetermined scanning line. The detection column parallel ADC 32 performs A / D conversion on the detection signal supplied from the imaging unit 21, and supplies the detection signal obtained as a result to the readout circuit 34 via the signal line 33.

  The readout circuit 34 is a digital circuit, and outputs the detection signal supplied from the detection column parallel ADC 32 to the signal processing unit 12 via the signal line 35.

  In the signal processing unit 12, the camera signal processing unit 41 performs various corrections, level adjustments, demosaic processing (details will be described later), etc. on the captured image signal supplied from the readout circuit 27 via the signal line 28. Do. Then, the signal processing unit 12 supplies the captured image signal obtained as a result to the buffer memory 43 via the signal line 42.

  Note that demosaic processing is performed by interpolating R, G, and B data of each pixel on the entire screen from R, G, and B data (hereinafter referred to as RAW data) as a captured image signal. This is a process for obtaining data (bitmap data).

  Further, the processing in the camera signal processing unit 41 is not limited to the processing described above. For example, the demosaic processing may not be performed in the camera signal processing unit 41, the subsequent processing may be performed using RAW data as the captured image signal, and the demosaic processing may be performed last.

  In this case, since motion blur, which will be described later, is estimated using RAW data, motion blur closer to the actual motion blur can be estimated, and motion blur can be reduced more accurately. In addition, since the data amount of the RAW data is 1/3 of the data amount of the bitmap data, it is possible to suppress the calculation amount of the subsequent processing. Further, there may be a case where the efficiency of calculation can be realized by performing the demosaic process at the time of the motion blur reduction process.

  The buffer memory 43 temporarily holds the captured image signal supplied from the camera signal processing unit 41 for timing adjustment with subsequent processing.

  The frame difference unit 44 obtains a difference for each pixel of the detection signals of two consecutive frames supplied from the readout circuit 34 via the signal line 35 and obtains an image signal of a motion detection image (hereinafter referred to as a motion detection image signal). Said). Then, the frame difference unit 44 supplies the motion detection image signal to the noise reduction unit 46 via the signal line 45.

  The noise reduction unit 46 reduces the noise level of the motion detection image signal supplied from the frame difference unit 44 in order to stabilize the subsequent processing. As a noise level reduction method, there are generally a median filter, a viral filter, an averaging process, and the like. The noise reduction unit 46 supplies the motion detection image signal with the reduced noise level to the motion detection unit 48 via the signal line 47. Note that the noise reduction unit 46 may not be provided.

  Based on the motion detection image signal supplied from the noise reduction unit 46, the motion detection unit 48 detects a motion vector in a motion detection image corresponding to the motion detection image signal. As a method for detecting this motion vector, there are a block matching method, an optical flow method, a corner tracking method, and the like.

  The block matching method is a method for obtaining a difference value between blocks of an image of the current frame and the previous frame and obtaining a vector between blocks having the smallest difference value as a motion vector.

  The optical flow method is a method typified by the Lucas-Kanade method, in which an image is regarded as a light field and a motion vector of the image is detected by calculation of the flow.

  Further, the corner tracking method is a method of obtaining a motion vector by detecting a corner of a subject such as an object or a landscape as a feature point (Feature Point, Interest Point) and tracking the feature point. This corner tracking method can be realized by using a corner detection function represented by a Harris Detector and a tracking function such as a particle filter.

  Since it is difficult to detect motion vectors at all points on the screen, the motion detection unit 48 detects motion vectors only at discrete sampling points. The motion vector detected by the motion detector 48 is supplied to the motion vector field generator 50 via the signal line 49.

  Based on the motion vector of each sampling point supplied from the motion detector 48, the motion vector field generator 50 deduces and interpolates the motion vectors of all points in the screen. For example, when a moving object is not included as a subject and the movement of the imaging device 10 does not include a rotation component, the magnitude of the movement in the screen is uniform. However, since this uniformity exists only on scanning lines having the same accumulation time, it is necessary to perform interpolation in the vertical direction of the screen.

  When the movement of the imaging device 10 includes a rotation component, the magnitude of the movement in the screen varies depending on the distance from the rotation center, and the direction of movement varies depending on the position. Therefore, it is necessary to perform interpolation in consideration of them. Therefore, the motion vector field generation unit 50 obtains, for example, the sum of motion vectors weighted by distance as a motion vector after interpolation.

  Further, the motion vector field generation unit 50 supplies the motion vectors of all points in the screen obtained as a result of the interpolation to the trajectory interpolation unit 52 via the signal line 51.

  The trajectory interpolation unit 52 converts the motion detection image into a motion detection image using a spline function or the like based on the motion vectors of all the points in the predetermined number of continuous motion detection images supplied from the motion vector field generation unit 50. The locus of movement of the subject in the corresponding captured image is obtained. Then, the trajectory interpolation unit 52 supplies the obtained motion trajectory to a PSF (Point Spread Function) generation unit 54 via the signal line 53.

  The PSF generation unit 54 uses a PSF (hereinafter referred to as motion blur PSF) as a function for reducing motion blur based on the motion vector trajectory from the trajectory interpolation unit 52 and the optical PSF data from the optical PSF data memory 55. The motion blur of each pixel is estimated. The optical PSF data is data representing the PSF of an optical system (not shown) that collects light from the outside and passes through the color filter 21A, and represents the optical blur of the imaging unit 21. It is.

  In addition, the PSF generation unit 54 supplies motion blur PSF data representing the obtained motion blur PSF to the motion blur reduction unit 59 via the signal line 57.

  The optical PSF data memory 55 stores optical PSF data in advance. The optical PSF data is read out as necessary and supplied to the PSF generation unit 54 via the signal line 56.

  The motion blur reduction unit 59 reads the captured image signal from the buffer memory 43 via the signal line 58. The motion blur reduction unit 59 uses the motion blur PSF data supplied from the PSF generation unit 54 to reduce motion blur of the captured image signal read from the buffer memory 43 and outputs the signal to the outside via the signal line 60. To do.

  The buffer memory 61 holds data output on the path from the frame difference unit 44 to the motion blur reduction unit 59, a calculation intermediate result, and the like. The detection control unit 62 transmits a selection signal indicating a detection signal reading method to the detection vertical scanning circuit 29 of the image sensor 11 via the signal line 63. Based on this selection signal, the detection vertical scanning circuit 29 selects the control line 30.

  As described above, in the imaging device 10, the image sensor 11 not only reads a captured image signal at a predetermined frame rate, but also reads a detection signal at a frame rate faster than the captured image signal. Therefore, the imaging apparatus 10 can acquire a captured image and acquire a plurality of frames of motion detection images within the imaging time of the captured image.

  As a result, the imaging apparatus 10 can accurately detect the movement of the subject in the captured image using the image sensor 11 that acquires the captured image signal. That is, the imaging apparatus 10 can accurately detect the movement of the subject in the captured image without significantly increasing the circuit scale.

  Next, FIG. 2 shows a detailed configuration example of the imaging unit 21.

  In FIG. 2, for convenience of explanation, the configuration of one pixel in the imaging unit 21, that is, the configuration of the pixel unit that captures one pixel is illustrated. This also applies to FIG. 3 described later.

  As shown in FIG. 2, the pixel unit 80 that captures an image of one pixel is provided with a photodiode 81 that performs photoelectric conversion. The photodiode 81 is connected to a captured image reading unit 82 that reads a captured image signal and a detection image reading unit 83 that reads a detection signal.

  The captured image reading unit 82 includes four MOS transistors 91 to 94 and a power source 95. Specifically, in the captured image reading unit 82, the source of the MOS transistor 91 is connected to the photodiode 81 and the back gate 103 (described later) of the detection image reading unit 83. The drain of the MOS transistor 91 is connected to the source of the MOS transistor 92 and the gate of the MOS transistor 93, and the gate is connected to the transfer signal control line 97 of the control line 23.

  The power supply 95 is connected to the drains of the MOS transistors 92 and 93. The gate of the MOS transistor 92 is connected to the control line 96 for the reset signal in the control line 23, and the gate of the MOS transistor 94 is connected to the control line 98 for the scanning line selection signal in the control line 23. Yes. Further, the drain of the MOS transistor 94 is connected to the source of the MOS transistor 93, and the source is connected to the readout line 24.

  In the captured image reading unit 82 configured as described above, when the control line 97 is selected by the image vertical scanning circuit 22 and the transfer signal is supplied to the gate of the MOS transistor 91 as the transfer gate, the MOS transistor 91 is turned on. As a result, the charge accumulated in the back gate 103 of the detection image reading unit 83 is transferred to the gate of the MOS transistor 93. Thereby, the electric charge accumulated in the back gate 103 is destroyed.

  When the image vertical scanning circuit 22 selects the control line 98, a scanning line selection signal is supplied to the gate of the MOS transistor 94, and the MOS transistor 94 is turned on. As a result, the charge transferred by the MOS transistor 91 is amplified by the MOS transistor 93 connected in series to the MOS transistor 94, and is output via the readout line 24 as a captured image signal.

  As described above, in the pixel unit 80, the charge stored in the back gate 103 is transferred to read the captured image signal, and the stored charge is destroyed by the transfer of the stored charge. It can be said that reading of is destructive reading.

  On the other hand, when the image vertical scanning circuit 22 selects the control line 96, a reset signal is supplied to the MOS transistor 92. As a result, the charge accumulated in the back gate 103 is reset by the MOS transistor 92.

  The detection image reading unit 83 includes a MOS transistor 101 and a power source 102. Specifically, the source of the MOS transistor 101 is connected to the power supply 102, the drain is connected to the readout line 31, and the gate is connected to the control line 30. The back gate 103 provided immediately below the channel region of the MOS transistor 101 is connected to the photodiode 81.

  In the detection image reading unit 83 configured as described above, charges generated in the photodiode 81 are accumulated in the back gate 103. When the control line 30 is selected by the detection vertical scanning circuit 29 and a read signal is supplied to the gate of the MOS transistor 101 via the control line 30, the MOS transistor 101 is turned on. As a result, a signal depending on the amount of charge accumulated in the back gate 103 is output as a detection signal via the readout line 31.

  As described above, the detection image reading unit 83 reads the detection signal depending on the charge amount while the charge accumulated in the back gate 103 is maintained. Therefore, the detection signal is read by non-destructive reading. It can be said.

  Next, FIG. 3 shows another detailed configuration example of the imaging unit 21.

  3, the pixel unit 120 includes a photodiode 121, and a captured image reading unit 122 and a detection image reading unit 123 configured in the same manner as the detection image reading unit 83 in FIG. Similarly to the detection signal, nondestructive reading is performed.

  Specifically, the captured image reading unit 122 includes a MOS transistor 131 and a power source 132, and the detection image reading unit 123 includes a MOS transistor 141 and a power source 142. The back gate 133 of the MOS transistor 131 and the back gate 143 of the MOS transistor 141 are connected to the photodiode 121.

  The drain of the MOS transistor 131 is connected to the power supply 132, the source is connected to the readout line 24, and the gate is connected to the control line 23. On the other hand, the source of the MOS transistor 141 is connected to the power supply 142, the drain is connected to the readout line 31, and the gate is connected to the control line 30.

  In the pixel unit 120 configured as described above, charges generated in the photodiode 121 are accumulated in the back gate 143 of the detection image reading unit 123 when the detection signal is read. Specifically, a potential for pushing the electric charge generated in the photodiode 121 to the back gate 143 as indicated by an arrow A is applied to the gate of the MOS transistor 131 of the captured image reading unit 122 via the control line 23 as an imaging read signal. Applied.

  Then, the control line 30 is selected by the detection vertical scanning circuit 29, and a detection read signal is applied to the gate of the MOS transistor 141 via the control line 30, and the MOS transistor 141 is turned on. As a result, a signal that depends on the amount of charge accumulated in the back gate 143, that is, a signal modulated by the amount of charge, is output as a detection signal via the readout line 31.

  On the other hand, when the captured image signal is read, the charge generated by the photodiode 121 is accumulated in the back gate 133 of the captured image reading unit 122. Specifically, as indicated by an arrow B, a potential that pushes the charge accumulated in the back gate 143 of the detection image reading unit 123 to the back gate 133 of the captured image reading unit 122 is a detection read signal. And applied to the gate of the MOS transistor 141 via the control line 30.

  Then, the control line 23 is selected by the image vertical scanning circuit 22, an imaging readout signal is applied to the gate of the MOS transistor 131 via the control line 23, and the MOS transistor 131 is turned on. As a result, a signal depending on the amount of charge transferred from the back gate 143 to the back gate 133 is output as a captured image signal via the readout line 24. Thereafter, the accumulated electric charge is caused to flow, for example, to a substrate (not shown) on which the image sensor 11 is arranged, and is reset.

  As described above, the pixel unit 120 in FIG. 3 can perform nondestructive readout of both the captured image signal and the detection signal. In addition, since the accumulated charge is moved to the back gate 133 or 143, a read operation can be performed reliably. Furthermore, since the number of MOS transistors necessary for reading the captured image signal and the detection signal is two, the pixel unit 120 can be downsized as compared with the pixel unit 80 of FIG. 2 that requires five MOS transistors. .

  Note that the charges accumulated in the back gates 133 and 143 may be electrons or holes.

  Next, the readout timing of the captured image signal and the detection signal will be described with reference to FIGS.

  First, FIG. 4 shows the readout timing when the imaging apparatus 10 performs long exposure, and FIG. 5 shows the readout timing when the imaging apparatus 10 performs a relatively short exposure.

  4A and 5A, the horizontal axis represents time, and the vertical axis represents the magnitude of the motion vector of the subject in the captured image, that is, the shift amount. 4B and 5B, the horizontal axis represents time, and the vertical axis represents the position of the scanning line in the vertical direction in the screen corresponding to the captured image signal or detection signal to be read. However, the value representing the position in the vertical direction is increased toward the top. Therefore, the upper part of the vertical axis represents the upper part of the screen, and the lower part represents the lower part of the screen. The same applies to the subsequent drawings.

  In FIG. 4B, there are 16 motion detection images during the readout of the captured image signal of a frame PF (Picture Frame) 1 of the captured image and the captured image signal of the next imaging frame PF2. Detection signals of DF (Detection Frame) 1 to DF16 are read out. Then, the signal processing unit 12 detects the locus of the motion vector of the subject in the captured image of the frame PF2 from the read detection signals of the 16 detection frames DF1 to DF16, and blurs the motion based on the locus. Reduce.

  However, since readout of the captured image signal and the detection signal is performed for each scanning line, the readout time is different for each scanning line even in the same frame as shown in FIG. 4B. In the example of FIG. 4B, the readout time at the top of the screen is early and the readout time at the bottom of the screen is late. That is, the captured image signal and the detection signal are sequentially read for each scanning line from the top to the bottom of the screen.

  Therefore, as shown in FIG. 4B, the imaging time 161 for the scanning line at the top of the screen, the imaging time 162 for the scanning line at the center of the screen, and the imaging time 163 for the scanning line at the bottom of the screen are different.

  However, when long-time exposure is performed, most of the imaging times 161 to 163 overlap, and as shown in FIG. 4A, there is no great difference in the trajectory of the motion vector of the subject within the imaging times 161 to 163. Therefore, the signal processing unit 12 can reduce motion blur by using the motion vector trajectory of the subject within the imaging time common to the entire screen.

  On the other hand, as shown in FIG. 5, when the exposure time is relatively short, as shown in FIG. 5B, the interval between reading of the captured image signal of the captured frame PF11 and the captured image signal of the next frame PF12 is between. This is shorter than in the case of FIG. 4B.

  For this reason, the ratio of the overlapping portion of the imaging time 181 in the scanning line at the top of the screen, the imaging time 182 in the scanning line in the center of the screen, and the imaging time 183 in the scanning line in the bottom of the screen is small. Therefore, as shown in FIG. 5A, the trajectory of the motion vector of the subject within the imaging time 181 to 183 may be greatly different.

  Therefore, when the exposure time is relatively short, the signal processing unit 12 divides the entire screen into a plurality of areas in the scanning direction in the vertical direction, and obtains the locus of the motion vector of the subject within the imaging time of each area, Using the motion vector trajectory of each region, motion blur is reduced for each region. Thereby, the motion blur reduction accuracy can be improved.

  In addition, when the frame rate of the detection signal is high, that is, when the frame interval of the detection signal is short due to a short exposure time, a large number of detection signal frames to be read within the imaging time, etc. Must be read at the same timing.

  FIG. 6 is a diagram for explaining the read timing required when the frame rate of the detection signal is high.

  In FIG. 6, the horizontal axis represents time, and the vertical axis represents the position of the scanning line in the vertical direction in the screen corresponding to the captured image signal or detection signal to be read. In the case of FIG. 6, the electronic shutter is not used.

  In FIG. 6, after the captured image signal of the imaging frame PF21 is read, until the captured image signal of the next imaging frame PF22 is read out, that is, within the imaging time 201 of the imaging frame PF22, four images are detected. The detection signals of the frames DF21 to DF24 are read out. Note that the detection signal of the last detection frame DF24 is read out by Δ time immediately before the captured image signal of the imaging frame PF22.

  Similarly to the imaging frame PF22, the detection signals of the four detection frames DF21 to DF24 are read out within the imaging time of the imaging frame PF21 and the imaging frame PF23 next to the imaging frame PF22.

  In the example of FIG. 6, since the frame interval 202 of the detection frames DF21 to DF24 is short, it is necessary to read the detection signals of the scanning lines of the four detection frames DF21 to DF24 together with the imaging signal of one imaging frame at one time. There is.

  For example, at time 203, it is necessary to read out detection signals of four detection frames straddling the imaging frame PF21 together with the imaging signal of the scanning line at the position 210 in the vertical direction of the imaging frame PF21. Specifically, the vertical position 211 of the detection frame DF23, the vertical position 212 of the detection frame DF22, the vertical position 213 of the detection frame DF21, and the imaging frame PF21 within the imaging time of the imaging frame PF22 It is necessary to read the detection signal of the scanning line at the position 214 in the vertical direction of the detection frame DF24 within the time.

  At time 204, it is necessary to read out the detection signals of the four detection frames straddling the imaging frame PF22 together with the imaging signal of the imaging frame PF22. Specifically, at time 204, it is necessary to read out detection frames DF23 and DF24 within the imaging time 201 of the imaging frame PF22 and detection signals of the detection frames DF21 and DF22 within the imaging time of the imaging frame PF23.

  However, when the readout rate of the captured image signal and the detection signal is the same, the detection signal of 4 frames cannot be read out in a time division manner while the captured image signal of 1 frame is read out. Therefore, the readout time per frame of the detection signal needs to be ¼ of the readout time per frame of the captured image signal.

  Therefore, in the imaging apparatus 10, the reading time per frame of the detection signal is reduced to 1/4 by using any of the following first to third methods. In the reading using any one of the first to third methods, the detection control unit 62 transmits a selection signal representing the method to the detection vertical scanning circuit 29 via the signal line 63. Done.

  First, the first method will be described with reference to FIG.

  The first method is a method of thinning out scanning lines of detection signals to be read.

  Specifically, in the first method, as shown in FIG. 7A, the detection signal of one scanning line is read out for every four scanning lines. That is, the detection signal corresponding to the charge accumulated in the pixel portion 80 (120) of the pixel (black circle in the figure) of the nth, n + 4th, n + 8th,. And the detection signal corresponding to the electric charge accumulated in the pixel portion 80 (120) of the pixel (white circle in the figure) of the scanning line other than the scanning line is not read out.

  In addition, pixel detection signals represented by black circles in FIG. 7A are sequentially read from the top for each scanning line. That is, the detection signals of the respective scanning lines are read out in the order of nth, n + 4th, n + 8th,. In FIG. 7A, the horizontal axis represents the horizontal position in the screen, and the vertical axis represents the vertical position of the scanning line in the screen.

  Next, with reference to FIG. 7B, the detection timing of the detection signals of the four detection frames DF21 to DF24 around the time 203 (FIG. 6) when the first method is used will be described. In FIG. 7B, the horizontal axis represents time, and the vertical axis represents the vertical position of the scanning line in the screen. In FIG. 7B, a broken line represents a frame boundary.

  As shown in FIG. 7B, at time 203, first, the detection signal of the a-th scanning line from the top corresponding to the position 211 of the detection frame DF23 within the imaging time of the imaging frame PF22 is read out. Next, the detection signal of the b-th scanning line from the top corresponding to the position 212 of the detection frame DF22 within the imaging time of the imaging frame PF22 is read. Thereafter, the detection signal of the c-th scanning line from the top corresponding to the position 213 of the detection frame DF21 within the imaging time of the imaging frame PF22 is read. Then, the detection signal of the d-th scanning line from the top corresponding to the position 214 of the detection frame DF24 within the imaging time of the imaging frame PF21 is read.

  Next, one time after the time 203, first, the detection signal of the a + 4th scanning line from the top of the detection frame DF23 within the imaging time of the imaging frame PF22 is read out. Next, the detection signal of the (b + 4) th scanning line from the top of the detection frame DF22 within the imaging time of the imaging frame PF22 is read out. Thereafter, the detection signal of the c + 4th scanning line from the top of the detection frame DF21 within the imaging time of the imaging frame PF22 is read out. Then, the detection signal of the d + 4th scanning line from the top of the detection frame DF24 within the imaging time of the imaging frame PF21 is read out.

  Then, two hours after the time 203, the detection signal of the a + 8th scanning line from the top of the detection frame DF23 within the imaging time of the imaging frame PF22 is read out, and the detection signal is similarly read out thereafter.

  As described above, in the reading using the first method, the detection signals of the four detection frames DF21 to DF24 that need to be read at the same time are read out in time division for each detection frame, and each detection frame is read out. These detection signals are read every four scanning lines.

  In the above description, the scanning line to be read is thinned to 1/4, but the scanning line is thinned to 1/2 and the number of pixels to be read in one scanning line is halved. By thinning out and simultaneously reading out the detection signals of the two scanning lines, the read time per frame of the detection signal may be reduced to 1/4.

  The detection signal read in this case will be described with reference to FIG. In FIG. 8, the horizontal axis represents the horizontal position in the screen, and the vertical axis represents the vertical position of the scanning line in the screen. In FIG. 8, circles represent pixels.

  As shown in FIG. 8, the reading target is thinned out with a thinning block 250 composed of all the pixels of the four scanning lines as a unit.

  Specifically, for example, every other pixel of the first scanning line from the top in the thinning block 250 and every other pixel (black circle in the figure) of the third scanning line from the top, the pixel portion 80 ( 120) are simultaneously read out as detection signals via different readout lines 31.

  In the two scanning lines to be read out in the thinning block 250, detection signals are read out for every other pixel, and therefore, through the readout lines 31 provided for the number of pixels in the horizontal direction in the screen. Thus, the detection signals of the two scanning lines can be read out simultaneously.

  In addition, every other pixel of the first scanning line from the top in the thinning block 250 and every other pixel (black circle in the figure) other than every other pixel (black circle in the figure) of the third scanning line from the top. The charge accumulated in the unit 80 (120) is not read out as a detection signal.

  Next, the second method will be described with reference to FIG.

  The second method is a method of adding and detecting detection signals of neighboring pixels.

  Specifically, as shown in FIG. 9, the color filter 21 </ b> A provided in the previous stage of the imaging unit 21 is configured by arranging color filters of R, G, and B colors in a Bayer manner. Then, the light that has passed through the color filter 21A is photoelectrically converted by the photodiode 81 (121) of the imaging unit 21. As a result, as the captured image signal and the detection signal, a signal of any color of R, G, and B, that is, a color signal is read out.

  However, even if the detection signal is not a color signal but a gray signal, the movement of the subject in the captured image can be detected.

  Therefore, in the second method, detection signals of neighboring pixels of each color are added, and a pseudo gray signal obtained as a result is generated as a detection signal. In the example of FIG. 9, the detection signals of the window 270 made up of 4 × 4 pixels are added and read.

  Next, with reference to FIG. 10, a detailed configuration of the reading circuit 34 in the case where the detection signals are added and read in units of the window 270 in FIG. 9 will be described.

  In the second method, the detection vertical scanning circuit 29 in FIG. 10 simultaneously selects the four control lines 30 corresponding to the window 270 of the color filter 21A. As a result, the detection signals of the pixels on the four scanning lines at the same position in the horizontal direction constituting the window 270 in the readout line 31 are added, and a signal obtained as a result (hereinafter referred to as a four-pixel combined detection signal). Is output.

  For example, in the example of FIG. 10, detection signals of four pixels R, G, R, and B are added in the rightmost readout line 31, and G, B, and G are added in the second readout line 31 from the right. , B detection signals of the four pixels are added.

  As described above, in the second method, four scanning lines are simultaneously selected and the detection signal is read out, so that the detection time of the detection signal of the four scanning lines is the detection signal of one scanning line. Is the same as the read time of. As a result, the readout time of the detection signal per frame is reduced to 1/4.

  As shown in FIG. 10, the detection column parallel ADC 32 includes ADCs 280 having the number of pixels in the horizontal direction in the screen connected in parallel. Each ADC 280 is connected to the readout line 31 connected to the pixel portion 80 (120) of the horizontal pixel corresponding to the ADC 280.

  Therefore, in the second method, the four pixel total detection signals of the pixels in the horizontal direction corresponding to the respective ADCs 280 are simultaneously supplied to the four ADCs 280 via the readout lines 31 connected thereto. Each ADC 280 performs A / D conversion on the four-pixel sum detection signal, and supplies a digital signal obtained as a result to the readout circuit 34 via the signal line 33.

  Further, in FIG. 10, the readout circuit 34 includes an addition circuit 291, a signal line 292, and a detection readout circuit 293.

  The adder circuit 291 digitally adds the four 4-pixel sum detection signals of the window 270 supplied via the four signal lines 33. As a result, the result of adding all the detection signals of the 4 × 4 pixels composed of the 8 G pixels and the 4 R and B pixels constituting the window 270 is calculated as pseudo luminance information. Is done.

  The detection readout circuit 293 reads out pseudo luminance information obtained as a result of addition by the addition circuit 291 as a detection signal, and outputs the detection signal via the signal line 35. Therefore, the data amount of the detection signal per frame is reduced to 1/16.

  As described above, in the second method, the detection signal is added in units of the window 270, so that the detection sensitivity can be improved. That is, since the detection signal is read out at a high frame rate, the charge accumulation time in the pixel portion 80 (120) is short. Therefore, the increase amount of the accumulated charge is reduced, and the detection sensitivity of the motion detection based on the detection signal corresponding to the accumulated charge may be deteriorated. In the second method, since the detection signal is added in units of window 270, the increase in the detection signal is increased and the detection sensitivity is improved.

  In FIG. 10, the reading of the detection signal of one window 270 has been described. However, the reading of the detection signal in units of the window 270 is simultaneously performed for the windows 270 having the same vertical direction.

  The timing of reading the detection signal in units of window 270 is the same as in the case of the first method, and thus the description thereof is omitted. In the second method, the number of pixels in the vertical direction of the window 270 corresponds to the thinning-out number of the scanning lines in the first method.

  In the above description, in the second method, the detection signals of all the pixels constituting the window 270 are added, but the detection signals of all the pixels constituting the window 270 may be averaged.

  Next, the third method will be described with reference to FIG.

  The third method is a method of partially reading the detection signal.

  FIG. 11 shows an example of a region in the screen corresponding to the detection signal to be read. In FIG. 11, the horizontal direction is the horizontal direction of the screen, and the vertical direction is the vertical direction of the screen.

  In the third method, for example, on the screen 300 in FIG. 11, the detection signal of the area 301 that is hatched in the drawing is read, and the detection signals of other areas are not read. In the example of FIG. 11, four areas 301 are provided at positions Y1 to Y4 in the vertical direction of the screen 300, and seven areas 301 are provided at positions X1 to X7 in the horizontal direction. In other words, a total of 28 areas 301 are provided on the screen 300.

  Note that since the detection signal is read for each scanning line, the regions 301 having the same position in the vertical direction are simultaneously read in units of scanning lines. For example, seven regions 301 whose vertical position is Y1 and whose horizontal position is any one of X1 to X7 are simultaneously read in units of scanning lines.

  In addition, the size of the region 301 depends on the magnitude of motion assumed as a detection target. When a large movement is assumed, the size of the area 301 is set large, and when a small movement is assumed, the size of the area 301 is set small.

  Further, the number Nv of the regions 301 in the screen 300 is a value obtained by dividing 1/4 of the total number of scanning lines in the screen 300 by the number of pixels Av in the vertical direction in the region 301. Thereby, the readout time per frame of the detection signal is reduced to 1/4 times the readout time per frame of the imaging signal.

  Next, with reference to FIG. 12, the read timing of the detection signal when the third method is used will be described. In FIG. 12, the horizontal axis represents time, and the vertical axis represents the vertical position of the scanning line in the screen.

  Here, the detection signal readout timing at time 321 in FIG. 12 will be described, but the same applies to other times.

  As shown in FIG. 12, at time 321, the vertical position Y1 of the detection frame DF23, the position Y2 of the detection frame DF22, the position Y3 of the detection frame DF21, and the imaging of the imaging frame PF21 within the imaging time of the imaging frame PF22 It is necessary to read the detection signal of the region 301 at the position Y4 of the detection frame DF24 within the time.

  Therefore, at time 321, reading is performed in a time-sharing manner for each of the seven regions 301 having the same vertical position of each frame.

  Specifically, at time 321, first, the detection signals of the seven regions 301 at the position Y1 in the vertical direction of the detection frame DF23 within the imaging time of the imaging frame PF22 are read. Then, the detection signals of the seven areas 301 at the position Y2 of the detection frame DF22 are read out. Next, the detection signals of the seven areas 301 at the position Y3 of the detection frame DF21 are read out. Finally, the detection signals of the seven areas 301 at the position Y4 of the detection frame DF24 within the imaging time of the imaging frame PF21 are read out.

  Note that the detection signal of each region 301 may be interleaved and read for each scanning line instead of reading for each of the seven regions 301 having the same vertical position.

  In this case, at time 321, first, the detection signal of the top scanning line of the seven regions 301 at the position Y1 of the detection frame DF23 is read. Then, the detection signal of the top scanning line of the seven areas 301 at the position Y2 of the detection frame DF22 is read. Next, the detection signal of the first scanning line in the seven areas 301 at the position Y3 of the detection frame DF21 is read out. Then, the detection signal of the top scanning line of the seven areas 301 at the position Y4 of the detection frame DF24 is read out.

  Thereafter, the processing returns to the region 301 at the position Y1 of the detection frame DF23, and the detection signal of the second scanning line from the top of the seven regions 301 at the position Y1 is read out. Thereafter, the readout is performed in the same manner, and as a result, the detection signals in the region 301 of the position Y1 of the detection frame DF23, the position Y2 of the detection frame DF22, the position Y3 of the detection frame DF21, and the position Y4 of the detection frame DF24 are read out. .

  In addition, when the corner tracking method is adopted as a method for detecting the motion vector, when the detection signal is read out by the third method of partially reading out the detection signal, a corner is just present in the read detection signal region. Not exclusively. Therefore, the detection signal area 301 to be read is moved in the horizontal direction or moved in the vertical direction.

  Of course, the number of detection frames within the imaging time is not limited to four. Therefore, for example, when the number of detection frames within the imaging time is 8 or 16, the readout time per frame of the detection signal is 1/8 or 1/16 of the readout time per frame of the captured image signal. Reduced.

  Specifically, in the first method, detection signals of 8 or 16 detection frames are read out in a time-sharing manner for each detection frame, and the detection signals of each detection frame are 8 scanning lines or 16 scanning lines. Read every time.

  As in the case described with reference to FIG. 8, the scanning line to be read and the pixels to be read in the scanning line may be thinned out. In this case, depending on the number of detection frames within the imaging time, flexible readout may not be possible without increasing the number of readout lines 31, readout circuits 34, and the like. In addition, the thinning pattern and the position of the thinned pixel may be fixed.

  In the second method, the detection signals of 8 or 16 detection frames are read out in a time division manner for each detection frame, and the detection signal of each detection frame has 8 or 16 pixels in the vertical direction. It is read in units of windows.

  Further, in the third method, the detection signals of 8 or 16 detection frames are read out in time division for each detection frame, and the detection signals of each detection frame are read out only for the set region. The relationship between the number Nv of the region in the screen and the number of vertical pixels Av in the region is expressed as follows when the number of detection frames in the imaging time is Fn and the total number of scanning lines in the screen is Bv. It is represented by (1).

Bv = Av × Nv × Fn
... (1)

  In the above description, the first to third methods are used to reduce the reading time of the detection signal per frame. However, the method is used to reduce the data amount of the detection signal per frame. It may be. For example, when it is not necessary to detect a fine motion in pixel units, the first to third methods are used to reduce the amount of data per frame of the detection signal and reduce the processing by the signal processing unit 12. You may do it.

  Next, processing performed by the frame difference unit 44 will be described with reference to FIG.

  In the image sensor 11, the accumulated charge is held as it is even when the detection signal is read out. Therefore, except for the first detection frame after the accumulated charge is reset by reading the imaging signal, the image sensor 11 is the same as the previous detection frame. It is necessary to obtain the difference as a motion detection image signal of the current detection frame. Accordingly, the frame difference unit 44 obtains a difference between detection signals of two consecutive detection frames supplied from the image sensor 11 as a motion detection image signal.

  For example, as shown in FIG. 13A, when the relationship between the amount of light received by the photodiode 81 (121) and the detection signal is linear, the frame difference unit 44 calculates the following motion detection image signal. In FIG. 13, the horizontal axis represents the amount of light received by the photodiode 81 (121) corresponding to the accumulated charge, and the vertical axis represents the detection signal.

  In FIG. 13A, for example, the detection signal of a certain detection frame is the detection signal F1, and the detection signal of the next detection frame is the detection signal F2. However, since the reading of the detection signal F1 is nondestructive reading, the accumulated charge corresponding to the detection signal F1 is held as it is. Therefore, the received light amount of the detection frame corresponding to the detection signal F2 is a value obtained by subtracting the received light amount P1 corresponding to the charge accumulated up to the detection frame from the received light amount P2 corresponding to the accumulated charge of the detection frame ( P2-P1).

  Therefore, the frame difference unit 44 obtains a difference between the detection signal F2 corresponding to the received light amount P2 and the detection signal F1 corresponding to the received light amount P1 as a motion detection image signal of the detected frame corresponding to the received light amount P2.

  Also, as shown in FIG. 13B, when the relationship between the amount of light received by the photodiode 81 (121) and the detection signal is nonlinear, the frame difference unit 44 moves in the same manner as in the linear case shown in FIG. 13A. A detected image signal is calculated. That is, the difference between the detection signal F3 of a certain detection frame and the detection signal F4 of the next detection frame is obtained as an image signal for motion detection of the detection frame corresponding to the detection signal F4.

  Next, a method for obtaining a motion vector locus will be described with reference to FIGS.

  In the example of FIGS. 14 and 15, the number of detection frames is eight. In FIGS. 14 and 15, the square represents a pixel. This also applies to FIGS. 16 to 18 described later.

  First, as shown in FIG. 14, for the pixel 352 at the position 351 in the final detection frame, the pixel of the same subject as the pixel 352 from the motion vectors (arrows in the figure) in the previous seven consecutive detection frames. The relative positions 361 to 368 in the respective detection frames are recognized. Then, as shown in FIG. 15, a path 371 that passes through the relative positions 361 to 367 in order and finally passes through the position 351 and is approximated by a smooth curve by a spline function or the like is obtained as a motion vector locus.

  Next, a method for obtaining the motion blur PSF will be described with reference to FIGS.

  First, when the path 371 illustrated in FIG. 15 is obtained as a motion vector trajectory, light from the same subject as the pixel 352 is captured at each point of the path 371 within the imaging time based on the motion vector trajectory. It is required how long you have stayed. The length of the stay time is proportional to the degree of influence on the amount of light received by the stay of light corresponding to the pixel 352 at each point.

  That is, the faster the movement, that is, the longer the distance from the relative position of one detection frame to the relative position of the next detection frame, the shorter the stay time of light corresponding to the pixel 352, and the influence of the stay of light is Few. The slower the movement, that is, the shorter the distance from the relative position of one detection frame to the relative position of the next detection frame, the longer the stay time of light corresponding to the pixel 352, and the influence of the stay of light is large.

  Therefore, when the stay time at each point on the path 371 is obtained, as shown in FIG. 16, the weight of each pixel in the screen is determined according to the stay time, and the function indicating this weight is the motion blur PSF. Desired. In FIG. 16, the weight value of each pixel when the sum of the weights for the pixel 352 is normalized to 100 is shown in a square representing each pixel. However, the value of the weight corresponding to the blank pixel in the screen is 0.

  In FIG. 16, it is assumed that there is no optical blur, and the PSF of the optical system is assumed to be an ideal point. However, optical blur is actually present, and the PSF itself of the optical system has a wide spread. The size of the spread may be the same level as the pixel or may be greater than the pixel size.

  FIG. 17 is a conceptual diagram of how to obtain the motion blur PSF when considering the spread of the PSF of the optical system.

  In FIG. 17, a circle 391 centering on each point constituting the path 371 indicates the spread of the PSF of the optical system. In the PSF of the optical system, generally, the value is large at the center of the circle 391 and decreases as it goes to the periphery. The weight of each pixel is determined in consideration of the spread of the PSF of the optical system, and the motion blur PSF is obtained. In this way, by considering the spread of the PSF of the optical system, it is possible to obtain a motion blur PSF that more accurately reduces motion blur.

  Further, the spread and shape of the PSF of the actual optical system are different for each color. Usually, the optical system is designed so that the PSF of the G component optical system becomes small in order to maximize the resolution at the time of photographing. Therefore, the PSF of the R component or B component optical system is wider than the G component. .

  FIG. 18 is a conceptual diagram of how to obtain a motion blur PSF when considering the spread of the PSF of the optical system for each color.

  In the example of FIG. 18, the PSF spread 401 of the G component optical system is the smallest, and the PSF spread 402 of the B component optical system is slightly larger than the spread 401. Further, the PSF spread 403 of the R component optical system is the largest. Considering the difference in the spread of the PSF of the optical system for each color, a motion blurred PSF is obtained for each color. In this way, by obtaining the motion blur PSF for each color and using a different motion blur PSF for the motion blur reduction processing of each color component, it is possible to improve the motion blur reduction accuracy.

  Since the optical system is generally an axis object, the PSF of the optical system is generally a point object (axis object), but the PSF of the optical system may not be a point object around the imaging surface. In such a case, if there is a rotation component in the motion of the subject or the imaging device 10, it is necessary to obtain the motion blur PSF in consideration of the influence of the rotation on the PSF of the optical system.

  Next, image processing by the imaging apparatus 10 in FIG. 1 will be described with reference to the flowchart in FIG. This image processing is started, for example, when imaging is instructed by the user.

  In step S <b> 11, the imaging unit 21 photoelectrically converts light from the outside in units of pixels and starts accumulating charges. In step S <b> 12, the imaging unit 21 reads non-destructively, as a detection signal, the charge accumulated in units of pixels corresponding to the control line 30 selected by the detection vertical scanning circuit 29 within a time shorter than the imaging time. This detection signal is converted into a digital signal by the detection column parallel ADC 32 and output to the signal processing unit 12 via the readout circuit 34 and the signal line 35.

  In step S <b> 13, the frame difference unit 44 of the signal processing unit 12 uses the difference for each pixel of the detection signals of two consecutive detection frames supplied from the readout circuit 32 via the signal line 35 as a motion detection image signal. Ask. Then, the frame difference unit 44 supplies the motion detection image signal to the noise reduction unit 46 via the signal line 45.

  Note that when only the detection signal of the first detection frame is supplied from the readout circuit 32, the processing of steps S13 to S16 is skipped.

  In step S <b> 14, the noise reduction unit 46 reduces the noise level of the motion detection image signal supplied from the frame difference unit 44, and transmits the resulting motion detection image signal to the motion detection unit 48 via the signal line 47. Supply.

  In step S <b> 15, based on the motion detection image signal supplied from the noise reduction unit 46, the motion detection unit 48 detects a motion vector at each sampling point in the motion detection image corresponding to the motion detection image signal. Then, the motion detector 48 supplies the detected motion vector of each sampling point to the motion vector field generator 50 via the signal line 49.

  In step S <b> 16, the motion vector field generation unit 50 deduces and interpolates the motion vectors of all points on the screen based on the motion vectors of the respective sampling points supplied from the motion detection unit 48. Then, the motion vector field generation unit 50 supplies the motion vectors of all points in the screen obtained as a result of the interpolation to the trajectory interpolation unit 52 via the signal line 51.

  In step S <b> 17, the detection vertical scanning circuit 29 determines whether the detection signals of a predetermined number (in the example of FIG. 6) of detection frames have been read. If it is determined in step S17 that the detection signals of the predetermined number of detection frames have not yet been read, the process returns to step S12 and the above-described process is repeated.

  On the other hand, when it is determined in step S17 that the detection signals of the predetermined number of detection frames have been read, the process proceeds to step S18, and the trajectory interpolation unit 52 is supplied from the motion vector field generation unit 50 and is continuously predetermined. Based on the motion vectors of all the points in each of the motion detection images, the motion trajectory of the subject in the captured image corresponding to the motion detection image is obtained by a spline function or the like. Then, the trajectory interpolation unit 52 supplies the obtained motion trajectory to the PSF generation unit 54 via the signal line 53.

  In step S <b> 19, the PSF generation unit 54 obtains a motion blur PSF based on the motion vector trajectory supplied from the trajectory interpolation unit 52 and the optical PSF data supplied from the optical PSF data memory 55. As a result, a PSF representing the motion blur and the blur situation caused by the optical system is obtained as the motion blur PSF. Then, the PSF generation unit 54 supplies motion blur PSF data representing the obtained motion blur PSF to the motion blur reduction unit 59 via the signal line 57.

  In step S20, the imaging unit 21 reads a signal corresponding to the charge accumulated in units of pixels within the imaging time corresponding to the control line 23 selected by the image vertical scanning circuit 22 as a captured image signal. This captured image signal is supplied to and held in the buffer memory 43. In step S21, the imaging unit 21 resets the accumulated charge.

  In step S <b> 22, the motion blur reduction unit 59 uses the motion blur PSF data supplied from the PSF generation unit 54 to reduce motion blur of the captured image signal stored in the buffer memory 43.

  Specifically, if the inverse problem of the blur problem corresponding to motion blur PSF is solved as it is for the captured image signal, the screen of the captured image signal is greatly disturbed by the influence of noise or strong around the edge of the screen Artifacts may occur. For this reason, in order to achieve a good reduction in motion blur while reducing this problem, the motion blur reduction unit 59 reduces motion blur using a motion blur PSF by a Wiener filter, a Richardson-Lucy method, or the like. For example, the motion blur reduction unit 59 creates a Wiener filter using the motion blur PSF as a window, and reduces motion blur of the captured image signal.

  In step S23, the imaging unit 21 determines whether or not to end imaging, for example, whether or not the user has instructed to end imaging. If it is determined in step S23 that the imaging is not finished, the process returns to step S11, and the above-described process is repeated. On the other hand, if it is determined in step S23 that the imaging is to be ended, the process ends.

  1, the detection vertical scanning circuit 29 and the image vertical scanning circuit 22 are provided separately, but the detection vertical scanning circuit 29 and the image vertical scanning circuit 22 may be shared. Good.

  FIG. 20 shows a detailed configuration example of the image sensor 11 in such a case.

  20 includes an image column parallel ADC 25, readout circuits 27 and 34, a detection column parallel ADC 32, an imaging unit 451, a vertical scanning circuit 452, a control line 453, a readout line 454, and the like.

  In FIG. 20, for convenience of explanation, illustration is omitted, but the image sensor 11 is also provided with a color filter 21 </ b> A. In FIG. 20, for convenience of explanation, the configuration of one pixel in the imaging unit 451, that is, the configuration of a pixel unit that images one pixel is illustrated. In FIG. 20, the same components as those in FIG.

  As shown in FIG. 20, a pixel unit 460 that images one pixel in the imaging unit 451 is obtained by sharing the captured image reading unit 122 and the detection image reading unit 123 of FIG. 3. Specifically, the pixel portion 460 is provided with a photodiode 461 that performs photoelectric conversion, and the back gate 463 of the MOS transistor 462 is connected to the photodiode 461. The source of the MOS transistor 462 is connected to the power supply 464, the gate is connected to the control line 453, and the drain is connected to the readout line 454.

  In the pixel portion 460 configured as described above, charges generated in the photodiode 461 are accumulated in the back gate 463. When the vertical scanning circuit 452 reads the detection signal or the captured image signal of the pixel portion 460, the vertical scanning circuit 452 selects the control line 453 and applies the detection readout signal or the imaging readout signal to the gate of the MOS transistor 462 via the control line 453. To do. As a result, the MOS transistor 462 is turned on, and as a result, a signal depending on the amount of charge accumulated in the back gate 463 is output via the readout line 454 as a detection signal or a captured image signal.

  That is, the control line 453 is obtained by sharing the control lines 30 and 23 shown in FIG. 3, and the read line 454 is obtained by sharing the read lines 31 and 24 shown in FIG.

  As described above, in the image sensor 11 of FIG. 20, since the detection vertical scanning circuit 29 and the image vertical scanning circuit 22 of FIG. 1 are shared, the captured image reading unit 122, the detection image reading unit 123, and the control The lines 30 and 23 and the readout lines 31 and 24 are also shared, so that the circuit scale can be reduced.

  However, as shown in FIG. 20, when the detection vertical scanning circuit 29 and the image vertical scanning circuit 22 in FIG. 1 are shared, the detection signal readout timing is the readout timing of the captured image signal. It is necessary not to overlap. On the other hand, as shown in FIG. 1, when the detection vertical scanning circuit 29 and the image vertical scanning circuit 22 are provided separately, reading of the detection signal and reading of the captured image signal can be performed independently. This is possible and the flexibility of the read operation can be maintained.

  As described above, the image sensor 11 reads the detection signal a plurality of times in a non-destructive manner within the imaging time of one frame, and reads out the captured image signal every imaging time of one frame. Therefore, the image sensor 11 acquires the captured image signal. Can be used to accurately detect the movement of the subject in the captured image.

  In contrast, Non-Patent Document 1 described above describes an example in which one image sensor is used and peripheral pixels are used for motion detection. However, in this case, the motion detection area is limited. Therefore, it is difficult to accurately detect the movement.

  Moreover, since the imaging device 10 reads the detection signal nondestructively, the captured image signal itself is not deteriorated. Furthermore, since the imaging device 10 reads the detection signal nondestructively, it can cope with various read operations (patterns). Therefore, it is possible to adaptively change the detection signal reading method and the motion detection algorithm according to the estimated imaging location and the temporal change of the captured image, and to realize highly accurate motion blur reduction.

  Further, in the image sensor 11, hardware added for reading the detection signal is a detection image reading unit 83 (123), a detection vertical scanning circuit 29, a detection column parallel ADC 32, a reading circuit 34, and the like. There is little increase in hardware.

  The image sensor 11 may be a CCD image sensor instead of a CMOS image sensor. In this case, the charges accumulated in the photodiodes of the image sensor 11 are simultaneously moved to the vertical transfer CCD over the entire area of the screen by the readout gate, so that the charge accumulation period is the same in each frame. This is called simultaneity of charge accumulation time during imaging.

  Further, when a CCD image sensor is used as the image sensor 11, there is a simultaneous charge accumulation time. Therefore, as in the case where a CMOS image sensor is used, the problem of image distortion (details will be described later), FIG. The problem in the case where the exposure time is relatively short as described in (4), the problem in which a plurality of detection frames described in FIG.

  The image distortion problem is a problem that occurs when the image sensor 11 is a CMOS image sensor. Specifically, since the CMOS image sensor sequentially reads out each scanning line from the scanning line on the screen, the readout timing differs for each scanning line. Therefore, the charge accumulation timing is also different for each scanning line. Therefore, when the image sensor 11 is a CMOS image sensor, when an image of a fast-moving object is imaged by the image sensor 11, the imaging timing is different between the upper part and the lower part of the screen, and image distortion may occur.

  In the above description, the read rate of the captured image signal and the detection signal is the same, but may be different. However, when the readout rate of the captured image signal and the detection signal is not the same, after the readout of the captured image signal, that is, after the accumulated charge of the pixel unit 80 (120) is reset, the detection signal is read out. Time varies depending on the position of the screen. Therefore, this point needs to be considered in the processing by the signal processing unit 12. It should be noted that the problem due to this point is reduced during long exposure.

  The present invention can be applied to, for example, a digital still camera and a video camera.

  Note that in this specification, the image processing of FIG. 19 is not limited to processing performed in time series in the order described, but also processing executed in parallel or individually even if not necessarily time-series processing. Is also included.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a block diagram which shows the structural example of one Embodiment of the imaging device to which this invention is applied. It is a figure which shows the detailed structural example of an imaging part. It is a figure which shows the other detailed structural example of the imaging part. It is a figure shown about the read-out timing in the case of performing long time exposure. It is a figure shown about the read-out timing in the case of performing exposure for a comparatively short time. It is a figure explaining the read-out timing required when the frame rate of a detection signal is high. It is a figure explaining a 1st method. It is a figure explaining other methods of the 1st method. It is a figure explaining the 2nd method. It is a figure which shows the detailed structural example of the reading circuit in the case of performing reading using a 2nd method. It is a figure explaining the 3rd method. It is a figure explaining the read-out timing of the detection signal at the time of using a 3rd method. It is a figure explaining the process by a frame difference part. It is a figure which shows the motion vector of all the detection frames. It is a figure which shows the locus | trajectory of a motion vector. It is a figure explaining the method of calculating | requiring motion blur PSF. It is a conceptual diagram of how to obtain a motion blur PSF when considering the spread of the PSF of the optical system. It is a conceptual diagram of how to obtain a motion blurred PSF when considering the spread of the PSF of the optical system for each color. It is a flowchart explaining an image process. It is a figure which shows the other detailed structural example of an image sensor.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 21 Image pick-up part, 22 Image vertical scanning circuit, 29 Detection vertical scanning circuit, 52 Trajectory interpolation part, 59 Motion blur reduction part, 81 Photodiode, 82 Image pick-up image reading part, 83 Image reading part for detection, 103 Back gate, 121 Photodiode, 122 Captured Image Reading Unit, 123 Detection Image Reading Unit, 133,143 Backgate, 451 Imaging Unit, 452 Vertical Scan Circuit, 461 Photodiode, 463 Backgate

Claims (9)

Photoelectric conversion means for photoelectrically converting light from the outside in units of pixels;
Accumulation means for accumulating charges obtained by the photoelectric conversion means in units of pixels;
Detection in which a signal corresponding to the electric charge accumulated by the accumulating means is read out in a non-destructive manner a plurality of times as a detection signal for detecting a motion of a subject in a picked-up image picked up immediately after one frame of image pickup time Signal reading means;
An imaging apparatus comprising: a captured image signal reading unit that reads out a signal corresponding to the charge accumulated by the accumulation unit as a captured image signal every imaging time of one frame. Detection signal reading control means for controlling reading of the detection signal reading means;
And a captured image signal readout control means for controlling readout of the captured image signal readout means,
Each of the detection signal reading means and the captured image signal reading means has at least one transistor,
The storage means is a back gate of the transistor of the detection signal reading means or the captured image signal reading means,
The detection signal readout means accumulates the electric charge in the back gate of the transistor of the detection signal readout means by the potential of the captured image signal readout control means applying a potential to the gate of the transistor of the captured image signal readout means. And when the detection signal reading control means applies a potential to the gate of the transistor of the detection signal reading means, a signal corresponding to the charge accumulated in the back gate of the transistor of the detection signal reading means is provided. , Nondestructive readout of the detection signal by outputting as the detection signal,
The picked-up image signal reading means stores the charge on the back gate of the transistor of the picked-up image signal reading means by the detection signal reading control means applying a potential to the gate of the transistor of the detection signal reading means. In addition, when the captured image signal readout control unit applies a potential to the gate of the transistor of the captured image signal readout unit, it corresponds to the charge accumulated in the back gate of the transistor of the captured image signal readout unit. The imaging apparatus according to claim 1, wherein non-destructive readout of the captured image signal is performed by outputting a signal to be performed as the captured image signal. Read control means for controlling reading of the detection signal read means and the captured image signal read means,
The detection signal reading means and the captured image signal reading means have one common transistor,
The storage means is a back gate of the transistor;
When the readout control unit applies a potential to the gate of the transistor, the detection signal readout unit outputs a signal corresponding to the charge accumulated in the back gate as the detection signal, so that the detection signal Non-destructive readout of the captured image signal is performed by performing non-destructive readout or by outputting a signal corresponding to the charge accumulated in the back gate as the captured image signal. The imaging device according to claim 1. Detection signal reading control means for controlling reading of the detection signal reading means;
And a captured image signal readout control means for controlling readout of the captured image signal readout means,
Each of the detection signal reading means and the captured image signal reading means has at least one transistor,
The storage means is a back gate of the transistor of the detection signal reading means;
When the detection signal reading control unit applies a potential to the gate of the transistor of the detection signal reading unit, the detection signal reading unit outputs a signal corresponding to the charge accumulated in the back gate of the transistor. By outputting as a detection signal, non-destructive reading of the detection signal is performed,
The captured image signal reading means destroys the charge accumulated in the back gate and responds to the charge when the captured image signal read control means applies a potential to the gate of the transistor of the captured image signal read means. The imaging apparatus according to claim 1, wherein the captured image signal is read out by outputting a signal to be performed as the captured image signal. The imaging apparatus according to claim 1, wherein the detection signal reading unit reads, as the detection signal, a signal corresponding to a charge of a part of pixels among the charges in pixel units accumulated by the accumulation unit. Detecting means for detecting movement of the subject in a captured image captured immediately after the detection signal using the detection signal;
The imaging apparatus according to claim 1, further comprising: a reduction unit that reduces motion blur from the captured image signal based on the movement of the subject detected by the detection unit and the captured image signal. The imaging apparatus according to claim 6, wherein the detection unit divides the screen of the captured image into a plurality of areas in units of scanning lines, and detects the movement of the subject in the captured image within the imaging time of each area. 7. The motion blur is reduced from the captured image signal based on the movement of the subject, the captured image signal, and an SPF of an optical system that collects light from the outside. Imaging device. The imaging device
A photoelectric conversion step for photoelectrically converting light from the outside in units of pixels;
An accumulation step for accumulating charges obtained by the photoelectric conversion step in units of pixels;
A signal corresponding to the charge accumulated by the processing of the accumulation step is detected non-destructively a plurality of times as a detection signal for detecting the movement of the subject in the captured image captured immediately after the image capturing time of one frame. A detection signal reading step for reading;
An imaging method including a captured image signal reading step of reading out a signal corresponding to the charge accumulated by the processing of the accumulation step as a captured image signal every imaging time of one frame.

Images