JP2014013981A - Imaging method and imaging device - Google Patents

Abstract

An image pickup method and an image pickup apparatus that reduce shot noise and have a good S / N ratio of light projection reflection information are provided.
The difference in image between the time of light projection and the time of no light projection is taken to eliminate the influence of the external light itself, and shot noise of the external light is reduced by motion compensation and averaging in the time axis direction. At this time, the motion compensation motion vector is detected using the video before the difference is taken, and the motion vector detection accuracy is improved.
[Selection] Figure 1

Description

The present invention relates to an imaging method and an imaging apparatus, and more particularly to an imaging method and an imaging apparatus suitable for irradiating emitted light onto a subject and imaging the subject from the reflected light.

In recent years, a device that emits a planar pattern to irradiate a subject, captures the reflected light, and measures the depth of the subject has been commercialized and used in home game machines and the like. Such an apparatus is described in, for example, Japanese Patent Application Laid-Open No. 2009-531655. It is also possible to simply project infrared light and obtain video information unique to the object from the reflection information without depending on the illumination status. In order to improve the S / N ratio of such an imaging device, Japanese Patent Application Nos. 2011-111669 and 2011-10
As described in No. 1764, a proposal has been made to remove the influence of external light by making the projection light into a pulse shape and subtracting the non-projection video from the projection video.

However, when used outdoors, when the outside light is significantly stronger than the intensity of the projection, such as under sunlight, the shot noise generated by the outside light becomes relatively large, and the necessary information (projection reflection information) ) Becomes worse.
Shot noise is a fundamental random noise generated during photoelectric conversion. When M signal electrons are obtained, random noise of √M is generated with an rms (root mean square) value. For example, 100 random noises with an rms value are generated in a bright part with 10,000 signals. This is a large noise as compared with the case where other noise such as so-called “kTC noise” has become several electrons or less.

Special table 2009-531655 gazette

An object of the present invention is to provide an imaging method and an imaging apparatus that reduce shot noise and have a good S / N ratio of light projection reflection information.

Since shot noise is random noise, it can be reduced by averaging. In the present invention, the SNR is improved by averaging the light projection reflection information in the time axis direction. When averaged in the time axis direction, the projected reflection information becomes blurred when the subject moves or the image pickup device shakes. However, blurring can be obtained by calculating the motion vector (MV) for each pixel and performing motion compensation. Prevent it from occurring. If the projection reflection information is used for MV detection at this time, the MV detection accuracy is deteriorated when the SN ratio is poor, the motion compensation cannot be performed correctly, and the SN ratio of the projection reflection information cannot be improved. May be blurred. Therefore, in the present invention, S is more than the light reflection information.
MV is detected using external light reflection information with a good N ratio or (external light + light projection) reflection information.

According to the present invention, it is possible to improve the noise, which is about 100 in terms of input electrons due to conventional shot noise, to 1/10 or less, for example, when averaging 128 times. As a result, it is possible to detect the projection reflection information with much higher accuracy than in the past, and to improve the recognition accuracy when used for image recognition.

The circuit block diagram which shows one Example of an imaging device. 6 is a timing chart showing an example of the operation of the imaging apparatus. An example of a light projection reflection information extraction circuit. An example of an MV generation circuit. The processing flowchart which shows one Example of the imaging method.

In the present invention, the SNR is improved by averaging the light projection reflection information in the time axis direction. In order to increase the efficiency of averaging and reduce adverse effects, motion compensation is applied to the light reflection information calculated by calculating MV for each pixel, and the same portions of the subject are averaged. The MV is detected by using S rather than the projection reflection information.
External light reflection information with a good N ratio or (external light + light projection) reflection information is used.

FIG. 1 is a circuit configuration diagram showing an embodiment of an imaging apparatus of the present invention. The image sensor 301 is a general-purpose
A global shutter type with a CMOS image sensor or a CCD image sensor and having the same exposure period for all pixels is desirable. The image sensor output S1 is input to the projection reflection information extraction circuit 302, and image information S2 based only on the projected light is generated based on the difference between frames. The video information S2 improves the SN ratio by the averaging process in the time axis direction, but in the example of FIG. 1, a cyclic filter 320 is used for the averaging process. The video information S3 whose SN ratio is improved by the recursive filter 320 is output from the output terminal 309.

In the MV generation circuit 310, the MV is obtained for each pixel from the image sensor output S1, and a cyclic filter is obtained.
Communicate to 320. Since the image sensor output S1 has a larger signal amount and a better SN ratio than the video information S2 obtained by taking the difference between the frames, it is possible to obtain a more accurate MV by obtaining the MV using the image sensor output S1. it can. Therefore, the accuracy of motion compensation performed by the recursive filter 320 is increased, and the SN ratio improvement effect of the video information S3 is also increased.

Next, an operation example of the recursive filter 320 will be described.
One piece of video information is stored in the frame memory 303, and the previous output video information S4 is output. The motion compensation circuit 311 performs motion compensation for each pixel in S4, and obtains video information S8 that is aligned with newly input video information S2. The difference generation circuit 304 takes the difference between this S8 and the input video information S2,

S5 = S2-S8 (Formula 1)

Is output. This S5 is reduced to 1 / n by the divider circuit 305, and then added to S8 by the adder circuit 306.
Video information S6 with improved ratio is generated. n is an integer, but 2, 4, 8, 16, 32, 64,
It is preferable to set the division circuit 305 to a simple circuit by setting the values of 128.

The video information S6 is as follows and the new input video information S2 is gradually influenced to form a recursive filter:

S6 = S8 + S5 / n = (1-1 / n) S8 + (1 / n) S2 (Formula 2)

The larger the numerical value n, the greater the effect of improving the S / N ratio. At the same time, the tailing of the video information increases. Even when n = 64, the SN ratio can be improved about 10 times. The problem of tailing can be countered by accurately performing motion compensation.
From (Equation 2), the feedback rate of the recursive filter, that is, the influence of the previous output on the current output is (1-1 / n). To simplify the divider circuit 305, n is an integer of 2. To the power and the return rate
Choose from 1/2, 3/4, 7/8, 15/16, 31/32, 63/64, 127/128.

In the switch 307, the video information S6 with improved SN ratio is usually selected and becomes the output video information S3. In some cases, a subject moving at such a high speed that the motion compensation cannot catch up may be imaged. In such a case, the input video information S2 is selected instead of the video information S6 with improved SN ratio, and the output video information S3 is selected. And This switch control is performed by the output S7 of the absolute value generation circuit 308 in the example of FIG. That is, when the absolute value S7 of the difference S5 is small, the video information S6 with improved SN ratio is output, and when the absolute value S7 is large, the input video information S2 is output.

Next, specific examples of the projection / reflection information extraction circuit 302 and the MV generation circuit 310 will be described. FIG. 2 is a timing chart showing an example of the operation of the imaging apparatus, and shows a light projection period, an exposure period, and an output period in order from the top. In the example of FIG. 2, light is projected once every two exposure periods, and the exposure period 103,
The image information exposed at 100, 101, 102 is imaged in the output period 123, 120, 121, 122 immediately after
It is output from 301. In the exposure period 101, in addition to the video information by the external light during that period, the video information by the light projected in the light projection period 111 is also accumulated and output in the output period 121. In order to extract only the video information from the light projected from among them, the video information stored in the previous exposure period 100 and output in the output period 120 may be subtracted. Alternatively, the video information accumulated in the exposure period 102 immediately after and output in the output period 122 may be subtracted.

In this way, the projection / reflection information S2 is generated by subtracting the video information of the output period 120 or 122 from the video information of the output period 121. The projection reflection information S2 improves the SN ratio by averaging processing in the time axis direction, but the MV used for the averaging processing is detected by comparing the video information of the output period 121 and the video information of the output period 123. These pieces of video information include not only external light but also video information based on the projected light. Since the amount of information is large, MV can be detected with high accuracy. Output period
It is possible to detect the image information of 122 and the image information of the output period 120 by comparing them, but when the amount of external light is small, the amount of information decreases and the MV detection accuracy decreases.

3A and 3B show an example of the projection reflection information extraction circuit 302. FIG. (a) is an example in which the projection reflection information is simply generated from the difference of two frames. 321 is a frame memory, 322 is a difference generation circuit, and a frame in which video information by (external light + light projection) is output, taking a difference from video information of the previous frame, that is, video information by only external light, The image information (projection reflection information) S2 only by the projection is generated. (b) is an example which produces | generates light projection reflection information by the calculation of 3 frames. Reference numeral 324 denotes an averaging circuit, which is a frame in which video information based only on external light is output.
Take the average of video information from the previous external light only. The video information of one frame before is based on (external light + light projection), and the light projection reflection information S2 is generated from the difference between this and the output of the averaging circuit 324.
Compared to the example of (a), there is a drawback that it is delayed by one frame, but the average time of two inputs of the difference generation circuit 322 can be adjusted.

4A and 4B show an example of the MV generation circuit 310. FIG. (a) is an example of generating an MV in real time, and the frame memory 331 operates only a frame for detecting the MV. As shown in FIG. 2, when MV is detected in a frame in which light is projected in alternating frames and projection reflection information is output, the output of the frame memory 331 is video information of two frames before.
The MV detection circuit 332 compares the video information of the previous two frames with the new video information,
It is detected how much the video information has moved in which direction (that is, MV). This detection is performed for each pixel, for example, and MV is detected using information around the pixel.
A difficult point in MV detection is that it may not be possible to detect a part with poor change. Among the MVs, MVs due to device vibrations such as camera shakes and vehicle shakes have a uniform value across the entire image,
Of the subjects, the inside of the rigid body often shows a uniform value. Therefore, when MV cannot be detected, it is possible to interpolate spatially from neighboring MVs.

Moreover, even if it can be detected, the accuracy and reliability may be poor. Even in such a case, the accuracy and reliability can be further improved by making corrections by referring to the surrounding MVs spatially and temporally. FIG. 4B shows an example of an MV generation circuit to which such an MV correction circuit 333 is added. MV
The memory 334 stores one frame of MV, and corrects from the surrounding MV for each pixel. Although a delay time occurs depending on the time until the peripheral MV information is gathered and the calculation time, FIG.
As described above, in the case of delaying the generation of the projection reflection information S2 by one frame, the delay of the MV correction is not a problem.

FIG. 5 is a process flowchart showing an embodiment of the imaging method of the present invention. In general, if the difference is taken in video information, the signal amount is reduced and the SN ratio is deteriorated. When it is desired to improve the SN ratio of the difference information, the SN ratio can be improved by averaging while compensating for the motion. However, if the accuracy of the MV used for motion compensation is poor, the intended SN ratio cannot be improved. Therefore, the MV is generated with a large signal amount before taking the difference.
Reference numeral 501 denotes input video information data, which is output from, for example, an image sensor. Difference processing 503 is performed on this to obtain desired information. This is subjected to motion compensation processing 504 and averaging processing 505 to improve the SN ratio, and output information data 506 is output. The MV used for the motion compensation process 504 is generated by the MV generation process 502. By using the data before taking the difference, an MV with higher accuracy than the case of using the difference data is generated.

100 ... exposure period,
101 ... exposure period,
102 ... exposure period,
103 ... exposure period,
111 ... lighting period,
113 ... Lighting period,
120 ... Output period,
121 ... Output period,
122 ... output period,
123 ... Output period,
301 ... Image sensor,
302 ... Projection reflection information extraction circuit,
303: Frame memory,
304 ... difference generation circuit,
305: Dividing circuit,
306... Addition circuit,
307 ... switch,
308... Absolute value generation circuit,
309: Output terminal,
310 ... MV generation circuit,
311: Motion compensation circuit,
320 ... cyclic filter,
321 ... Frame memory,
322... Difference generation circuit,
323 ... Frame memory,
324 ... Averaging circuit,
331: Frame memory,
332 ... MV detection circuit,
333 ... MV correction circuit,
334 ... MV memory,
501 ... Input video information data,
502 ... MV generation processing,
503 ... Difference processing,
504 ... motion compensation processing,
505 ... Averaging process,
506: Output information data.

Claims (8)

In an imaging method for obtaining desired information by a difference in video information,
Means for motion compensation of the difference;
While improving the SN ratio of the difference by means for averaging the motion compensated difference in the time axis direction,
An imaging method, wherein motion compensation motion vectors are detected using video information before taking a difference. In the imaging method according to item 1,
An imaging method, wherein the difference in video information is an inter-frame difference. In the imaging method according to item 1 or 2,
An imaging method, wherein the averaging means is a recursive filter. In the imaging method according to any one of Items 1 to 3,
An imaging method characterized in that video information is based on external light and is based on external light and light projection. In an imaging device that obtains desired information based on the difference between imaging element outputs,
Means for motion compensation of the difference;
While improving the SN ratio of the difference by means for averaging the motion compensated difference in the time axis direction,
An image pickup apparatus that detects a motion compensation motion vector using an image pickup element output before taking a difference. In the imaging device according to item 5,
An imaging apparatus characterized in that the difference in output between imaging elements is an inter-frame difference. In the imaging device according to Item 5 or 6,
An imaging apparatus characterized in that the means for averaging is a recursive filter. In the imaging device according to any one of Items 5 to 7,
An image pickup apparatus provided with means for intermittently projecting light.

Images