JP2011114486A - Imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform stable FP (focal plane) distortion correction even when a panning operation, or the like is performed. <P>SOLUTION: The imaging device includes: an image motion-detecting part 52 for generating image motion information by performing optical flow detection of an input image from image data of the input image; a sensor part 53 for using an angular velocity sensor to generate sensor motion information; a pan and tilt-determining part 54 for using those information to perform pan and tilt determination; an FP correction amount-calculating part 55 for calculating an FP correction amount from the image motion information when panning or tilting is performed, and on the other hand, calculating the FP correction amount from the sensor motion information when they are not performed; and a correction processing part 56 for executing FP distortion correction processing of removing the FP distortion of the input image on the basis of the FP correction amount and electronic camera-shake correction processing on the basis of the image motion information or the sensor motion information. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital video camera or a digital still camera, and more particularly to a technique for removing focal plane distortion.

  Electronic shutters used in solid-state imaging devices are roughly classified into a global shutter in which all light receiving pixels on the imaging surface are exposed at a common timing, and a rolling shutter in which exposure timing is different between different horizontal lines. The rolling shutter is used in an XY address type solid-state imaging device, for example, a CMOS image sensor, which can read out an output signal of a light receiving pixel in a pixel unit.

  In an imaging apparatus that employs a rolling shutter, a focal plane distortion (hereinafter referred to as FP distortion) occurs in a captured image in principle when a positional change occurs in a solid-state imaging device during shooting due to the influence of a camera shake of a photographer. It has been known. However, FP distortion can be reduced by performing appropriate image processing on the captured image.

  For example, a method is known in which the movement of the imaging apparatus due to camera shake or the like is detected from the output of a sensor unit provided in the imaging apparatus, and the FP distortion is reduced by correcting the captured image based on the detected movement. FIG. 23 shows an internal configuration diagram of such a sensor unit. The sensor unit is provided with an angular velocity sensor 901 that outputs a detection signal representing the angular velocity of the imaging apparatus, and an HPF (High Pass Filter) 902 that reduces a direct current component and a low frequency component included in the detection signal. Although not shown in FIG. 23, the sensor unit is provided with a plurality of blocks each including an angular velocity sensor 901 and an HPF 902, and the plurality of angular velocity sensors 901 provided in the plurality of blocks have different angular velocities in different directions. To detect.

  Since the output signal of the angular velocity sensor 901 inevitably contains a DC offset component due to the nature of the angular velocity sensor 901, an HPF 902 is provided to remove this component. The HPF 902 can remove unnecessary signal components included in the output signal of the angular velocity sensor 901 and extract only effective signal components. However, when panning or tilting is performed on the imaging apparatus, the HPF 902 The presence of this adversely affects the output signal of the sensor unit.

  The adverse effect will be described with reference to FIG. For example, when a panning operation is performed in which the imaging apparatus is shaken at a constant speed in a constant direction, an angular velocity signal having a large signal value is output from the angular velocity sensor 901 continuously during the panning period. The signal obtained by removing the DC offset component from the output signal of the angular velocity sensor 901 at this time is an ideal signal 921 that should be output from the sensor unit. However, during the panning period, the output signal of the sensor unit attenuates from the original signal due to the influence of the HPF 902, and at the end of panning, the output signal of the sensor part should excessively decrease due to the influence of the HPF 902 and should be positive in nature. The output signal of the sensor unit becomes negative. If the FP distortion correction is performed based on the signal attenuated from the original signal, the correction is insufficient, and if the FP distortion correction is performed based on the signal having the opposite sign, a corrupted image or an overcorrected image is obtained. As described above, there is a problem in that it is difficult to obtain an accurate signal from the sensor unit during the panning operation, and as a result, it is difficult to accurately correct the FP distortion.

  By the way, various conventional methods using a sensor unit that detects the movement of the imaging apparatus have been proposed. For example, Patent Documents 1 and 2 below propose a method of performing optical camera shake correction using both angular velocity sensor information and motion vector information based on image data in consideration of panning or tilting operations. However, this method is a method related to optical image stabilization, and does not provide any solution to the above-described problem with respect to FP distortion. Further, Patent Document 3 below discloses a method of changing the output signal of the vibration detecting means when panning or tilting is performed in the method of correcting camera shake from the detection result of the vibration detecting means. However, this method is a method for reducing a so-called swing-back phenomenon that may occur during a panning or tilting operation, and does not provide any solution to the above problem with respect to FP distortion.

JP 11-187308 A JP 05-14801 A JP 2007-324929 A

  Therefore, an object of the present invention is to provide an imaging apparatus capable of performing stable image correction even when panning or the like is performed.

  An image pickup apparatus according to the present invention includes an image pickup device that generates an input image by shooting while varying exposure timing between different horizontal lines, and includes a plurality of input images based on the output of the image pickup device. A motion detection unit that detects the optical flow of the imaging device based on the output of the imaging device, a sensor unit that detects the motion of the imaging device, and / or a detection result of the motion detection unit and a detection result of the sensor unit. A correction unit that selects either a detection result of the motion detection unit or a detection result of the sensor unit based on the correction value, and generates an output image by correcting the input image based on the selected detection result. It is characterized by that.

  Thereby, when the accuracy of the detection result of the sensor unit is suspected, such as when panning or the like is performed, the input image can be corrected using the detection result of the motion detection unit. As a result, erroneous correction of the input image by using the detection result of the sensor unit suspected of being accurate is avoided.

  Specifically, for example, the correction unit corrects distortion generated in the input image due to different exposure timings between different horizontal lines.

  Further, for example, the correction unit further corrects a shake on the input moving image generated by a movement of the imaging apparatus during a shooting period of the input moving image including a plurality of input images. This is performed based on the detection result of the motion detection unit or the detection result of the sensor unit.

  More specifically, for example, the imaging apparatus further includes a determination unit that determines the presence or absence of panning or tilting of the imaging apparatus based on both or one of the detection result of the motion detection unit and the detection result of the sensor unit. The correction unit selects the detection result of the motion detection unit when it is determined that the panning or tilting is present, and selects the detection result of the sensor unit otherwise.

  In addition, for example, the correction unit corrects distortion generated in the input image due to different exposure timings between different horizontal lines, and the correction unit during a shooting period of an input moving image including a plurality of input images A blur correction unit that corrects a blur on the input moving image generated by a motion of the imaging apparatus, and the distortion correction is performed based on the selected detection result, while the blur correction is the motion detection; Is performed based on the detection result of the part or the detection result of the sensor part, and the correction part uses the distortion correction part and the shake correction part when the panning or tilting is not performed on the imaging device. When the output image is generated from the input image and panning or tilting is performed on the imaging device, the blur correction unit should not be used. Generating the output image from the input image using the distortion correction unit.

  Since panning or tilting is based on the user's intention, when panning or tilting is performed, it is natural to generate an output image without using the shake correction unit. Therefore, an imaging device may be formed as described above.

  Further, for example, the correction unit is formed so as to be able to change the intensity of correction of the distortion.

  Thereby, optimization of the distortion correction intensity according to the situation is achieved.

  According to the present invention, it is possible to provide an imaging apparatus capable of performing stable image correction even when panning or the like is performed.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is an overall block diagram of an imaging apparatus according to a first embodiment of the present invention. It is an internal block diagram of the imaging part shown by FIG. It is a figure which shows the horizontal line arrangement | sequence in the effective pixel area | region of the image pick-up element of FIG. It is a figure showing the exposure timing and signal read-out timing in the image sensor of FIG. It is an internal block diagram of the image correction function part which concerns on 1st Embodiment of this invention. It is a figure which shows the relationship between an effective pixel area | region and the memory area for input images. It is a figure showing the several input image arranged on a time series, and the exposure timing and signal read-out timing of each input image. It is a figure which shows a mode that nine motion detection blocks are set in the whole image area | region of an input image. It is a figure which shows the optical flow between two input images. It is a figure showing the internal structural example of the sensor part of FIG. It is a figure which shows the input image (a) in which FP distortion does not exist, the input image (b) in which horizontal FP distortion has arisen, and the input image (c) in which vertical FP distortion has arisen. It is a figure for demonstrating the method to produce | generate the intra-frame motion vector of an input image from the inter-frame motion vector of an input image (overall motion vector between adjacent input images). FIG. 4 is a diagram illustrating an input image including horizontal FP distortion and a distortion correction image obtained by performing FP distortion correction processing on the input image according to the first embodiment of the present invention. FIG. 4 is a diagram illustrating an input image including vertical FP distortion and a distortion correction image obtained by performing FP distortion correction processing on the input image according to the first embodiment of the present invention. It is a figure for demonstrating the camera-shake correction process which concerns on 1st Embodiment of this invention. 3 is an operation flowchart of the imaging apparatus according to the first embodiment of the present invention. It is an internal block diagram of the image correction function part which concerns on 2nd Embodiment of this invention. It is a figure which concerns on 2nd Embodiment of this invention and shows the input image containing a horizontal FP distortion, and the distortion correction image obtained by performing an FP distortion correction process to this input image. It is a figure which concerns on 2nd Embodiment of this invention and shows the input image containing a vertical FP distortion, and the distortion correction image obtained by performing an FP distortion correction process to this input image. It is a figure which concerns on 2nd Embodiment of this invention and shows the relationship between both in the case of setting correction | amendment intensity | strength coefficients (kh, kv) from zoom magnification (ZF). It is a figure which concerns on 2nd Embodiment of this invention and shows the relationship between both in the case of setting correction | amendment intensity | strength coefficients (kh, kv) from the reliability (RL) of motion detection. It is an operation | movement flowchart of the imaging device which concerns on 2nd Embodiment of this invention. It is an internal block diagram of the sensor part provided in the conventional imaging device. It is a figure which showed the output signal of the sensor part during a panning period with an ideal signal according to a prior art.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

<< First Embodiment >>
A first embodiment of the present invention will be described. FIG. 1 is an overall block diagram of an imaging apparatus 1 according to the first embodiment of the present invention. The imaging device 1 has each part referred by the codes | symbols 11-28. The imaging device 1 is a digital video camera, and can capture a moving image and a still image, and also can simultaneously capture a still image during moving image capturing. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25. The display unit 27 and / or the speaker 28 may be interpreted as being provided in an external device (not shown) of the imaging device 1.

  The imaging unit 11 captures a subject using an imaging element. FIG. 2 is an internal configuration diagram of the imaging unit 11. The imaging unit 11 includes an optical system 35, a diaphragm 32, an imaging element 33 that is a solid-state imaging element, and a driver 34 that drives and controls the optical system 35 and the diaphragm 32. The optical system 35 is formed of a plurality of lenses including a zoom lens 30 for adjusting the angle of view of the imaging unit 11 and a focus lens 31 for focusing. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. Based on the control signal from the CPU 23, the position of the zoom lens 30 and the focus lens 31 in the optical system 35 and the opening of the diaphragm 32 are controlled, so that the focal length (field angle) and the focal position of the imaging unit 11 and the imaging are controlled. The amount of light incident on the element 33 is controlled.

  The image sensor 33 is formed by arranging a plurality of light receiving pixels in the horizontal and vertical directions. Each light receiving pixel of the image sensor 33 photoelectrically converts an optical image of a subject incident through the optical system 35 and the diaphragm 32, and outputs an electric signal obtained by the photoelectric conversion to an AFE 12 (Analog Front End).

  The AFE 12 amplifies the analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The amplification degree of signal amplification in the AFE 12 is controlled by a CPU (Central Processing Unit) 23. The video signal processing unit 13 performs necessary image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The microphone 14 converts the ambient sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 using a predetermined compression method. The internal memory 17 is composed of a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores various data. The external memory 18 as a recording medium is a non-volatile memory such as a semiconductor memory or a magnetic disk, and records a video signal and an audio signal compressed by the compression processing unit 16.

  The decompression processing unit 19 decompresses the compressed video signal and audio signal read from the external memory 18. The video signal expanded by the expansion processing unit 19 or the video signal from the video signal processing unit 13 is sent to the display unit 27 such as a liquid crystal display via the display processing unit 20 and displayed as an image. Further, the audio signal that has been expanded by the expansion processing unit 19 is sent to the speaker 28 via the audio output circuit 21 and output as sound.

  The TG (timing generator) 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and gives the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The TG 22 further generates a drive pulse for the image sensor 33 under the control of the CPU 23 and supplies the drive pulse to the image sensor 33. The CPU 23 comprehensively controls the operation of each part in the imaging apparatus 1. The operation unit 26 includes a recording button 26a for instructing start / end of moving image shooting and recording, a shutter button 26b for instructing shooting and recording of a still image, and a zoom button for designating a zoom magnification. 26c and the like, and accepts various operations by the user. The operation content for the operation unit 26 is transmitted to the CPU 23.

  The image sensor 33 has an electronic shutter function, and performs exposure of each light receiving pixel by a so-called rolling shutter. The imaging element 33 is provided with an imaging surface formed by a plurality of light receiving pixels, and the light receiving pixels are two-dimensionally arrayed along a plurality of horizontal lines and a plurality of vertical lines defined on the imaging surface. . In the rolling shutter, the timing at which each light receiving pixel on the imaging surface is exposed differs for each horizontal line. That is, the exposure timing of the light receiving pixels is different between different horizontal lines on the imaging surface.

A supplementary description of the rolling shutter of the image sensor 33 will be given with reference to FIGS. 3 and 4. As shown in FIG. 3, on the imaging surface, the direction along the vertical line (that is, the vertical direction of the imaging surface) is the vertical direction, and the horizontal line arranged at the uppermost end of the effective pixel area 33 _{A on} the imaging surface is the first. It is assumed that the horizontal line and the horizontal line arranged at the lowermost end of the effective pixel area 33 _A is the Mth horizontal line (M is an integer equal to or larger than 2, generally in the range of several hundreds to several thousand. is there). Then, as shown in FIG. 4, for a certain target frame, the exposure start time at the light receiving pixels on the (i + 1) th horizontal line is Δt seconds from the exposure start time at the light receiving pixels on the i th horizontal line. Slow (Δt> 0 and i is an integer). In one frame, the exposure time of the light receiving pixels on the i-th horizontal line is the same as that of the light receiving pixels on the (i + 1) -th horizontal line, and Δt is constant regardless of the value of i. It is. After the exposure of each light receiving pixel is completed, the output signal of each light receiving pixel can be read, and the read output signal is sent to the AFE 12.

  An integer multiple of Δt and Δt is called an exposure timing difference between different horizontal lines. Δt is an exposure timing difference between adjacent horizontal lines.

  Of the first to Mth horizontal lines, there may be a plurality of horizontal lines having the same exposure timing. For example, the exposure timing may be different for every two horizontal lines. In this case, although the exposure timing of the second horizontal line and the exposure timing of the third horizontal line are different, the exposure timing of the first and second horizontal lines is the same and the exposure timing of the third and fourth horizontal lines is the same. It becomes. Further, although the exposure timing of each light receiving pixel is sequentially shifted in one horizontal line, the difference in exposure timing between different light receiving pixels on one horizontal line is sufficiently smaller than that between different horizontal lines, so The exposure timing of each light receiving pixel on the line can be regarded as the same.

  The image sensor 33 is, for example, an XY address type CMOS (Complementary Metal Oxide Semiconductor) image sensor. A CMOS image sensor is formed by forming an imaging surface composed of a plurality of light receiving pixels, a vertical scanning circuit, a horizontal scanning circuit, a pixel signal output circuit, and the like on a semiconductor substrate capable of forming a CMOS. However, if the image pickup device 33 performs exposure of each light receiving pixel by a rolling shutter, the image pickup device 33 may not be an XY address scanning type CMOS image sensor.

  As described in the description of the background art, FP distortion (focal plane distortion) occurs in an image taken with a rolling shutter. The FP distortion occurs due to Δt being larger than 0, and the FP distortion tends to increase as Δt increases. The imaging device 1 has a function of appropriately removing this FP distortion.

  FIG. 5 shows a block diagram of the image correction function unit 50 having the FP distortion removal function according to the first embodiment. The image correction function unit 50 is provided in the imaging apparatus 1, and the image correction function unit 50 includes each part referred to by reference numerals 51 to 56. The frame memory 51 is realized by the internal memory 17 of FIG. The image motion detection unit 52 and the correction processing unit 56 are provided, for example, in the video signal processing unit 13 of FIG. The pan / tilt determination unit 54 and the FP correction amount calculation unit 55 are realized by the video signal processing unit 13, the CPU 23, or the video signal processing unit 13 and the CPU 23.

  The output signal of each light receiving pixel of the image sensor 33 is read out at a predetermined frame rate (for example, 30 frames / second), and the output signal of each light receiving pixel of the image sensor 33 is used as an image signal of the input image. Saved in. However, it is assumed that a signal obtained by performing necessary signal processing on the output signal of each light receiving pixel is an image signal of the input image. The signal processing here includes signal amplification and signal digitization in the AFE 12, and further includes demosaicing processing for converting the signal format of the image signal from the RAW data format to the YUV format. The YUV image signal is formed from a luminance signal and a color difference signal.

  Since the contents of the signal processing are not particularly related to the characteristic functions of the present invention, the presence of the signal processing is ignored in the following description for the sake of simplicity. In this specification, an image signal and a video signal are synonymous. Further, an image signal for a certain light receiving pixel or pixel is also referred to as a pixel signal. In the following description, the image signal and the pixel signal are also referred to as image data. Moreover, in this specification, the name corresponding to a code | symbol or a symbol may be abbreviated or abbreviate | omitted by attaching a code | symbol or a symbol. For example, the motion detection block corresponding to the symbol BL [1] referred to in FIG. 8 described later may be referred to as a block BL [1] or simply BL [1].

In the frame memory 51, a two-dimensional memory area 51 _A as a memory space is allocated for one input image. As shown in FIG. 6, the memory area 51 _A has the same structure as the effective pixel area 33 _A of the image sensor 33, and the output signal of the light receiving pixel at the position (x, y) on the effective pixel area 33 _A is It is stored at an address (x, y) on the memory area 51 _A (x and y are integers). In the image sensor 33, the light receiving pixel at the position (x, y) is a light receiving pixel located on the xth vertical line and on the yth horizontal line. The vertical line arranged at the left end of the effective pixel area 33 _A is a first vertical line, and in the effective pixel area 33 _A , the (i + 1) th vertical line is a vertical line adjacent to the right side of the i-th vertical line ( i is an integer). In the same manner as the horizontal and vertical lines are arranged in the effective pixel region on 33 _A, the horizontal and vertical lines are also defined in the memory area 51 _A. Two-dimensional image represented in the memory region 51 _A is the same as the image to be imaged on the effective pixel region on 33 _A. Incidentally, any of the two-dimensional image represented by the input image, the image sensor 33, in the effective pixel region 33 _A and the memory area 51 _A, vertical horizontal direction be parallel to the horizontal direction and the horizontal lines in the vertical direction and the vertical Parallel to the line.

By providing a plurality of memory areas 51 _A in the frame memory 51, image data of a plurality of input images can be stored in the frame memory 51. The image data of each input image stored in the frame memory 51 is sent to the image motion detection unit 52 and the correction processing unit 56.

Now, as shown in FIG. 7, it is assumed that the current time becomes times t _n−1 , t _n , t _{n + 1} ,... Each time the unit time T _A elapses (n is an integer). Therefore, the time when the unit time T _A has elapsed from the time t _n-1 is the time t _n . The unit time T _A is a frame period that is the reciprocal of the frame rate. However, the unit time T _A may be an integral multiple of the frame period. The _nth input image is shot starting from time tn. The n-th input image is denoted by reference numeral I _n. Times t _n−1 and t _n−1 ′ are the start time and end time of the exposure period for acquiring the input image I _n−1 , respectively, and the times t _n and t _n ′ are respectively the input image I _n. Are the start time and end time of the exposure period for obtaining the same (the same applies to the times t _{n + 1} and t _{n + 1} ′). The time length of the exposure period of each input image, which is equal to the time length between times t _n-1 and t _n-1 ′ and the time length between times t _n and t _n ′, is denoted by T _B. In the present embodiment, T _B (for example, about 10 msec) is shorter than T _A (for example, about 33 msec). However, it may be a T _B = T _A.

A collection of a plurality of images arranged in time series is called an image sequence. Therefore, for example, an image sequence composed of the input images I _n−1 , I _n and I _{n + 1} is referred to as an input image sequence. It can be said that the image sequence is a moving image.

  The image motion detection unit 52 in FIG. 5 calculates an optical flow between the two input images based on the image data of the two input images that are temporally adjacent to each other, and the two input images are calculated from the optical flow. Generate and output image motion information for. In calculating the optical flow, the image motion detection unit 52 divides the entire image area of each input image into a plurality of areas, and sets a motion detection block in each divided area.

  A reference numeral 210 in FIG. 8 represents a certain input image. As an example, nine motion detection blocks BL [1] to BL [9] are set by defining nine divided regions by dividing the entire image region of the input image 210 into three equal parts in the horizontal and vertical directions. Assume that. When i is 0, 1 or 2, BL [i + 1] to BL [i + 3] are motion detection blocks arranged in the horizontal direction, and BL [i + 1], BL [i + 4] and BL [i + 7] are vertical. It is a motion detection block arranged in the direction. Therefore, when i is 0, 1 or 2, the horizontal line included in the motion detection block is the same between the blocks BL [i + 1] to BL [i + 3], and the vertical line included in the motion detection block is the block. It is the same among BL [i + 1], BL [i + 4] and BL [i + 7]. Further, when i is 0, 1 or 2, BL [i + 1] is arranged on the left end side of the input image 210, BL [i + 3] is arranged on the right end side of the input image 210, and BL [i + 2] is BL [i + 2]. i + 1] and BL [i + 3], BL [i + 1] is arranged on the upper end side of the input image 210, BL [i + 7] is arranged on the lower end side of the input image 210, and BL [i + 4] is BL [i + 1]. ] And BL [i + 7].

With reference to FIG. 9, a method of generating image motion information will be described after two temporally adjacent input images are represented by reference numerals 220 and 230. The input image 230 is an input image generated next to the input image 220. For example, if the input image 220 is I _n−1 , the input image 230 is I _n , and if the input image 220 is I _n , the input image 230 is input. Image 230 is In _{+ 1} .

  The optical flow between the input images 220 and 230 obtained by the image motion detection unit 52 is composed of a plurality of motion vectors. The image motion detection unit 52 calculates a motion vector for each motion detection block using a block matching method, a representative point matching method, a gradient method, or the like. As shown in FIG. 9, the motion vectors obtained for BL [1] to BL [9] are represented by symbols MV [1] to MV [9], respectively (in FIG. 9, the complication prevention shown in the figure is prevented). Therefore, only a part of the symbols MV [1] to MV [9] is shown). The motion vector is obtained with reference to the input image 220. Therefore, when the position of the feature point in the block BL [i] of the input image 220 is the first position and the position corresponding to the feature point on the input image 230 is the second position, The motion vector MV [i] is a vector having a first position as a start point and a second position as an end point.

  The image motion detection unit 52 obtains an average vector of MV [1] to MV [9] as the entire motion vector MV. The overall motion vector MV for the input images 220 and 230 is the direction and magnitude of the movement of the imaging apparatus 1 during the shooting period of the input images 220 and 230 (more specifically, for example, the intermediate point in the exposure period of the input image 220 and the input It is obtained as an inverse vector of a vector representing the direction and magnitude of the movement of the imaging apparatus 1 between the exposure point of the image 230 and the intermediate point in time. The inverse vector of the entire motion vector MV is also a vector representing the direction and size of hand movements described later. The image motion information for the input images 220 and 230 includes vectors MV [1] to MV [9] and MV between the input images 220 and 230.

  Unlike the image motion detection unit 52, the sensor unit 53 in FIG. 5 detects the motion of the imaging device 1 (in other words, the motion of the image sensor 33) using a sensor. The movement of the imaging device 1 is caused by a so-called camera shake or the like. Therefore, the movement of the imaging device 1 is also referred to as camera shake. The sensor unit 53 detects the direction and size of camera shake in an arbitrary period by detecting the direction and size of camera shake at a sampling interval (for example, 1 msec interval) that is sufficiently shorter than a frame period (for example, about 33 msec). To do. A vector that is detected by the sensor unit 53 and represents the direction and size of camera shake is referred to as a sensor camera shake vector. The sensor unit 53 can generate a sensor camera shake vector at one sampling interval, or can generate a sensor camera shake vector during a period that is an integral multiple of the sampling interval. The sensor unit 53 outputs information specifying a sensor shake vector for an arbitrary period as sensor motion information.

  FIG. 10 shows an angular velocity sensor 71 and a high-pass filter 72 provided inside the sensor unit 53. The angular velocity sensor 71 outputs a detection signal indicating the angular velocity of the imaging device 1. The high-pass filter 72 reduces the direct current component and the low frequency component included in the detection signal in order to remove the direct current offset component in the detection signal. The sensor unit 53 includes a plurality of blocks each including an angular velocity sensor 71 and a high-pass filter 72, and the plurality of angular velocity sensors 71 provided in the plurality of blocks detect angular velocities in different directions. A sensor camera shake vector is generated from the output signal of the high-pass filter 72 of each block (that is, the detection signal in which the DC component and the low-frequency component are reduced). The sensor provided in the sensor unit 53 may be other than an angular velocity sensor (for example, an acceleration sensor).

  Whether the pan / tilt determination unit 54 (hereinafter may be abbreviated as the determination unit 54) of FIG. 5 is panned or tilted in the imaging device 1 based on the image motion information and / or sensor motion information. The information including the determination result is output as pan / tilt determination information. This determination is hereinafter referred to as pan / tilt determination. As generally understood, panning refers to an operation in which the user intentionally shakes the imaging device 1 in the horizontal direction, and tilting refers to an operation in which the user intentionally shakes the imaging device 1 in the vertical direction. Point to. Determining that panning or tilting has been performed on the imaging apparatus 1 is expressed as “with pan / tilt”, and determining that panning or tilting has not been performed on the imaging apparatus 1 is “panning”. • Expressed as “no tilt”.

  Any known pan / tilt determination performed based on the image motion information and / or sensor motion information can be performed by the determination unit 54. For example, the methods described in Japanese Patent Application Laid-Open Nos. 2008-129554 and 11-187308 can be used for pan / tilt determination.

  Specifically, for example, the determination unit 54 determines that panning or tilting has been performed on the imaging device 1 when any of the following first to fifth determination conditions is satisfied, When information indicating “tilt” is included in the pan / tilt determination information and none of the first to fifth determination conditions is satisfied, it is determined that panning or tilting is not performed on the imaging apparatus 1 and “panning” is performed. Information including “no tilt” is included in the pan / tilt determination information.

  The first determination condition is a condition that “the entire motion vector MV facing in the same direction is continuously detected during a predetermined determination period”. The determination period is a period having a certain time length, and the determination period can be an integral multiple of the frame period. When the determination period is 10 times the frame period, a total of 10 total motion vectors MV obtained for 11 temporally continuous input images are referred to. When the directions of the ten vectors MV are the same, the first determination condition is satisfied, and when not, the first determination condition is not satisfied. Here, the same vector direction is a concept having a width, and even if the direction of the compared vector is different by a predetermined minute angle, the direction of the compared vector is regarded as the same (No. The same applies to 2 judgment conditions).

  The second determination condition is a condition that “a sensor shake vector pointing in the same direction is continuously detected during a predetermined determination period”. The second determination condition is satisfied when the directions of the sensor shake vectors during the determination period are the same, and otherwise the second determination condition is not satisfied.

  The third determination condition is a condition that “the size of the combined vector of all the entire motion vectors MV belonging to the predetermined determination period is equal to or larger than a predetermined reference size”.

  The fourth determination condition is a condition that “the size of the combined vector of all sensor shake vectors belonging to the predetermined determination period is equal to or larger than a predetermined reference size”.

  The fifth determination condition is a condition that “the magnitude of one sensor camera shake vector indicating the direction and magnitude of camera shake during a predetermined period is greater than or equal to a predetermined magnitude”.

  It is also possible to perform pan / tilt determination using both image motion information and sensor motion information. For example, panning or tilting is performed on the imaging apparatus 1 only when both the first and second determination conditions are satisfied, or only when both the third and fourth determination conditions are satisfied. You may make it judge.

  Here, how FP distortion appears will be described with reference to FIGS. Consider that an input image is acquired by photographing a rod-like subject extending in the vertical direction in real space with the imaging device 1. It is assumed that the rod-shaped subject is stationary in real space.

  An image 310 in FIG. 11A is an input image obtained when the imaging apparatus 1 is stationary during the exposure period of the input image, and an image 320 in FIG. 11B is during the exposure period of the input image. 11 is an input image obtained when the imaging apparatus 1 moves in the right direction, and an image 330 in FIG. 11C is an input obtained when the imaging apparatus 1 moves downward during the exposure period of the input image. It is an image. Attention is now paid to a specific portion on the bar-shaped subject that appears at the lowermost end of the input image 310. Positions 311, 321, and 331 in FIGS. 11A to 11C represent the positions of the specific portions on the input images 310, 320, and 330, respectively. Due to the rightward movement of the imaging apparatus 1 during the exposure period of the input image, the rod-shaped subject is tilted on the input image 320, and as a result, the position 321 is separated from the position 311 to the left by the distance Dh. Due to the downward movement of the imaging apparatus 1 during the exposure period of the input image, the rod-like subject contracts in the vertical direction on the input image 330, and as a result, the position 331 is separated from the position 311 by the distance Dv. If the imaging apparatus 1 moves upward during the exposure period of the input image as in the surroundings, the bar-shaped subject appears to extend from the original appearance on the obtained input image. The FP distortion that horizontally distorts the position of the specific part from the position 311 to the position 321 is called horizontal FP distortion, and the FP distortion that distorts the position of the specific part from the position 311 to the position 331 in the expansion / contraction direction is vertical FP distortion ( (Or stretching FP distortion).

  The FP correction amount calculation unit 55 (hereinafter may be abbreviated as the calculation unit 55) in FIG. 5 selectively uses the image motion information and the sensor motion information based on the pan / tilt determination information, thereby obtaining the FP correction amount. Is calculated. The FP correction amount is a correction amount used when the correction processing unit 56 reduces the FP distortion included in the input image, and is a two-dimensional amount having a direction and a size. In a period in which it is determined that there is no pan / tilt, sensor motion information is selected, and an FP correction amount is calculated based on the sensor motion information. On the other hand, during a period in which it is determined that there is pan / tilt, image motion information is selected, and an FP correction amount is calculated based on the image motion information. As described in the background art, when panning or tilting is performed, the accuracy of the sensor motion information is suspected.

  The correction processing unit 56 corrects the FP distortion of the input image based on the FP correction amount generated by the calculation unit 55 and the input image sequence (in other words, the input moving image) generated by the camera shake. And a camera shake correction process for correcting the image blur. However, camera shake correction processing may not be executed (details will be described later). An image obtained by performing FP distortion correction processing on an input image or an image obtained by performing FP distortion correction processing and camera shake correction processing on an input image is referred to as an output image. A plurality of sequentially obtained output images can be recorded as output moving images in the external memory 18 and displayed on the display unit 27. It can also be considered that the correction processing unit 56 includes a distortion correction unit that performs FP distortion correction processing and a shake correction unit that performs camera shake correction processing.

[FP distortion correction processing]
During the period in which it is determined that there is pan / tilt, the FP correction amount is calculated from the image motion information. For this reason, FP distortion correction processing based on image motion information is performed during a period in which it is determined that there is pan / tilt. As the FP distortion correction processing based on the image motion information (in other words, processing for reducing the FP distortion of each input image based on the optical flow of the input image sequence), any known processing can be used.

  A method for calculating an FP correction amount based on image motion information and an FP distortion correction process employed in this embodiment will be described.

The entire motion vector MV between the input images I _n-1 and I _n is particularly represented by MV [n-1, n]. The same applies to the entire motion vector MV for the other two input images. The overall motion vector MV [n−1, n] between the input images I _n−1 and I _n represents the average motion of the imaging device 1 between times t _n−1 and t _n ′ (FIG. 7). reference). On the other hand, FP distortion of the input image I _n is caused by movement of the imaging apparatus 1 during the exposure period of the input image I _n. Therefore, in order to accurately remove the FP distortion of the input image I _n is estimated FP correction amount corresponding to the motion of the imaging apparatus 1 during the exposure period of the input image I _n, the FP correction amount for the input image I _n There is a need to.

In order to satisfy this need, the calculation unit 55 calculates an intra-frame motion vector representing the motion of the imaging apparatus 1 during the exposure period of each input image from a plurality of overall motion vectors MV using interpolation. The FP correction amount is derived from the motion vector. Frame motion vector for the input image I _n represents the motion of the imaging apparatus 1 during the exposure period of the input image I _n.

Specifically, for example, as shown in FIG. 12 (a) and (b), the overall motion between the input image I _n-1 and I _n vectors MV [n-1, n] and the input image I _n and I _{n +} the total motion vector MV [n, n + 1] mean vector 250 between ₁ can be obtained as frame motion vector of the input image I _n. The graph of FIG. 12B is a graph in which the direction and size of each vector are imaged (the same applies to FIG. 12D described later). 12A and 12B, the time position corresponding to each vector is represented by a white circle (the same applies to FIGS. 12C and 12D described later).

Or, for example, as shown in FIGS. 12C and 12D, the entire motion vector MV [n−2, n−1] between the input images I _n−2 and I _n−1 can be represented as times t _n−2 and t 'together handled as a vector of intermediate time between input images I _n-1 and the overall motion vector MV [n-1, n] the time t _n-1 and t _n between I _{n' n-1} intermediate time between vector at after having handled as a vector, at an intermediate time of the exposure period of the vector MV [n-2, n- 1] and MV [n-1, n] by linear interpolation in the time direction, the input image I _n 260 estimates the may be obtained the estimated vector 260 as frame motion vector of the input image I _n.

  In the above example, one intra-frame motion vector is obtained based on two overall motion vectors, but one intra-frame motion vector may be obtained based on three or more overall motion vectors.

FP correction amount for the input image I _n is a frame in the motion vector itself of the input image I _n, or is an inverse vector of it. From the size of the FP correction amount of the input image I _n, Sadamari the distance the imaging apparatus 1 has moved during the exposure period of the input image I _n, the orientation of the FP correction amount of the input image I _n, the exposure of the input image I _n The direction in which the imaging device 1 has moved during the period is determined. Distance and orientation determined here is the distance and orientation on the input image I _n. In the example described with reference to FIGS. 11 (a) ~ (c) , the FP correction amount of the input image I _n, determined orientation and distance Dh toward the position 321 from the position 311, or position from the position 311 331 The direction and distance Dv to go to are determined.

The input image after the FP distortion correction process is called a distortion corrected image. In FP distortion correction processing, based on the FP correction amount of the input image I _n, by performing a geometric transformation to remove FP distortion of the input image I _n in the input image I _n, the distortion of the input image I _n A corrected image can be generated. Geometric transformation includes translation and resampling.

For example, if the input image 320 is an input image I _n, and a horizontal FP distortion to distort and to it it from the input image 320 of the rodlike subject input image 310 an image of generated, the horizontal FP distortion input image because what is contained in the I _n are defined in the FP correction amount, based on the FP correction amount, the inverse transformation of the geometric transformation that transforms into an input image 320 input image 310, as an input image 320 applied to the input image I _n of. FIG. 13 shows a distortion corrected image 320a obtained as a result.

If the the horizontal lines of the input image 320 is formed from the first to M horizontal lines, in the right direction on the input image a i-th horizontal line of the input image I _n as the input image 320 (i-1) × By translating by (Dh / (M−1)), the i-th horizontal line of the distortion-corrected image 320a is obtained. In the case of the FP distortion horizontal FP distortion of the input image I _n is as well known, even by controlling the readout start position at the time of reading the image data from the memory area 51 _A can be removed FP distortion.

Also, for example, if the input image 330 is an input image I _n, and the vertical FP distortion to distort and to it it from the input image 330 of the rodlike subject input image 310 an image of occurs, its vertical FP distortion input because what is contained in the image I _n are defined in the FP correction amount, based on the FP correction amount, the inverse transformation of the geometric transformation that transforms into an input image 330 input image 310, an input image 330 applied to the input image I _n as. FIG. 14 shows a distortion corrected image 330a obtained as a result.

If the horizontal line of the input image 330 is formed from the first to Mth horizontal lines and the position 331 belongs to the M _O horizontal line (1 <M _O <M), the input image I as the input image 330 is input. The i-th horizontal line of the distortion-corrected image 330a is obtained by translating the _n- th horizontal line in the downward direction on the input image by (i−1) × (Dv / (M _O −1)).

Although the detection and correction method of the horizontal FP distortion and vertical FP distortion explained separately, when the horizontal FP distortion and perpendicular FP distortion in the input image I _n is occurring, both corrected horizontal FP distortion and vertical FP distortion As a result, a distortion-corrected image is generated. At this time, the horizontal FP distortion and the vertical FP distortion may be corrected simultaneously, or the horizontal FP distortion and the vertical FP distortion may be corrected in order.

During the period when it is determined that there is no pan / tilt, the FP correction amount is calculated from the sensor motion information. For this reason, FP distortion correction processing based on sensor motion information is performed during a period in which it is determined that there is no pan / tilt. Any known process can be used as the FP distortion correction process based on the sensor motion information (in other words, the process of reducing the FP distortion of each input image based on the motion detection result of the imaging device 1 by the sensor). is there. Included in the sensor motion information, a composite vector of all sensors hand movement vector obtained during the exposure period of the input image I _n, or, inverse vector of the resultant vector is the FP correction amount of the input image I _n. The content of the FP distortion correction process based on the FP correction amount may be the same between the case where the FP correction amount is obtained based on the image motion information and the case where the FP correction amount is obtained based on the sensor motion information.

[Image stabilization processing]
The correction processing unit 56 in FIG. 5 performs camera shake correction processing based on image motion information or sensor motion information. The camera shake correction process executed by the correction processing unit 56 is an electronic camera shake correction process for correcting image blur on an input image sequence (in other words, an input moving image) caused by camera shake. A known electronic camera shake correction process can be used. The camera shake correction process may be performed after the FP distortion correction process, or the FP distortion correction process may be performed after the camera shake correction process. The camera shake correction process and the FP distortion correction process may be executed simultaneously.

  In the present embodiment, the camera shake correction process is performed after the FP distortion correction process. In this case, the distortion correction image sequence obtained by performing the FP distortion correction processing on each input image forming the input image sequence is the target of camera shake correction processing. In the camera shake correction process, a cutout frame is set for each distortion correction image, and an image in the cutout frame is cut out as an output image from each distortion correction image. The image in the cutout frame is a part of the distortion corrected image. At this time, the position of the clipping position frame is adjusted based on the image motion information or the sensor motion information so that the image blur based on the camera shake does not occur on the output image sequence.

For example, as shown in FIG. 15, a stationary point as a subject appearing at a position (x, y) on the distortion correction image 350 based on the input image I _n-1 is on the distortion correction image 360 based on the input image I _n. Let us consider a case where it appears at a position (x + Δx, y) (assuming that the stationary point is stationary in real space). In this case, the entire motion vector between the input images I _n−1 and I _n included in the image motion information is a vector representing a shift between the positions (x, y) and (x + Δx, y) (however, a detection error) Is ignored). Further, from the sensor motion information, a vector representing a deviation between the positions (x, y) and (x + Δx, y) is obtained (however, the detection error is ignored). Therefore, the correction processing unit 56 uses the center position of the cutout frame 361 set for the distortion correction image 360 based on the image motion information or the sensor motion information in the camera shake correction processing as the center position of the cutout frame 351 of the distortion correction image 350. The stationary point is made still on the output image sequence based on the input images I _n-1 and I _n by shifting by Δx in the horizontal direction.

[Operation flow]
Next, an operation procedure of the imaging apparatus 1 will be described with reference to FIG. FIG. 16 is a flowchart illustrating an operation procedure of the imaging apparatus 1.

  First, in step S11, an input image is generated using the imaging unit 11, and in step S12, image motion information for the latest input image and sensor motion information for the latest input image are generated. In subsequent step S13, pan / tilt determination is made based on the image motion information and / or sensor motion information generated in step S12.

  If it is determined in step S13 that there is no pan / tilt, the process proceeds from step S13 to step S14, and the calculation of the FP correction amount based on the sensor motion information and the FP distortion correction process are performed on the latest input image. An output image is generated by performing a camera shake correction process in step S15 on the distortion corrected image obtained by the FP distortion correction process. The output image generated in step S15 is recorded in the external memory 18 and displayed on the display unit 27. The camera shake correction process in step S15 may be performed based on the sensor motion information. In this case, the position of the clipping frame for camera shake correction processing is determined from the sensor motion information for one frame from the sensor unit 53. However, as described above, the camera shake correction processing in step S15 may be performed based on the image motion information.

  If it is determined in step S13 that there is pan / tilt, the process proceeds from step S13 to step S16. In step S16, the calculation of the FP correction amount based on the image motion information and the FP distortion correction process are performed on the latest input image, but the camera shake correction process is not performed. That is, without changing the position of the cutout frame for the distortion correction image obtained by the FP distortion correction processing (in other words, without changing the position of the cutout frame between the previous and current distortion correction images), An output image is generated by cutting out the image within the cutout frame of the distortion corrected image. The output image generated in step S16 is recorded in the external memory 18 and displayed on the display unit 27. Since the movement by panning or tilting is a movement by the intention of the user, it is natural to reflect the movement on the recorded image. Therefore, the camera shake correction process is not executed in step S16.

  After the process of step S15 or S16, the process returns to step S11, and the processes of steps after step S11 are repeatedly executed. During the period when it is determined that there is no pan / tilt, the processes of S11 to S15 are repeatedly executed, and during the period when it is determined that there is pan / tilt, the processes of S11 to S13 and S16 are repeatedly executed. .

  According to this embodiment, when the accuracy of the detection result of the sensor unit 53 is suspected, such as when panning is performed, FP distortion correction is performed based on image motion information based on image data. As a result, erroneous correction of the input image by using the detection result of the sensor unit 53 suspected of accuracy is avoided.

<< Second Embodiment >>
A second embodiment of the present invention will be described. Since the entire block diagram of the imaging apparatus according to the second embodiment is the same as that of FIG. 1, the imaging apparatus according to the second embodiment is also referred to by reference numeral 1. FIG. 17 is a block diagram of the image correction function unit 50a responsible for the FP distortion removal function according to the second embodiment. The image correction function unit 50 a is provided in the imaging apparatus 1, and the image correction function unit 50 a includes each part referred to by reference numerals 51 to 57. By adding the correction intensity adjustment unit 57 to the image correction function unit 50 of FIG. 5 according to the first embodiment, the image correction function unit 50a of FIG. 17 is obtained. Except for this addition, the imaging device 1 according to the second embodiment and the imaging device 1 according to the first embodiment are the same, and differences from the first embodiment will be described below.

  In the second embodiment, the FP correction amount calculated by the FP correction amount calculation unit 55 is once sent to the correction strength adjustment unit 57 before being sent to the correction processing unit 56. The correction intensity adjustment unit 57 adjusts the FP correction amount based on the adjustment information, and sends the adjusted FP correction amount to the correction processing unit 56. Accordingly, the correction processing unit 56 in the second embodiment performs FP distortion correction processing based on the adjusted FP correction amount.

In the present embodiment, for the sake of concrete explanation, the horizontal FP distortion that distorts the input image 310 in FIG. 11A into the input image 320 in FIG. 11B or the input image 310 in FIG. vertical FP distortion distorting to the input image 330 shown in FIG. 11 (c) is assumed to be caused in the input image I _n.

Therefore, when the horizontal FP distortion has occurred in the input image I _n, only the left side of the distance Dh in the input image I _n position 321 belonging to the horizontal line of the lowermost is the horizontal FP distortion of the input image I _n original It is assumed that the FP correction amount indicating that the position is shifted from the position 311 is obtained from the image motion information or the sensor motion information. Similarly, if the vertical FP distortion occurs in the input image I _n, the position 311 belonging to the horizontal line of the vertical FP undistorted input image 310 lowermost end is in the input image I _n by a vertical FP distortion It is assumed that the FP correction amount indicating that the position is shifted to the position 331 by being shifted by the distance Dv is obtained from the image motion information or the sensor motion information.

The correction intensity adjustment unit 57 sets correction intensity coefficients kh and kv based on the adjustment information. Then, the magnitude Dh of the FP correction amount for the horizontal FP distortion is changed to the magnitude Dh ′ using the correction intensity coefficient kh, and the magnitude Dv of the FP correction amount for the vertical FP distortion is changed to the correction intensity coefficient kv. To change to the size Dv ′. In particular,
Dh ′ = kh × Dh,
Dv ′ = kh × Dv,
Accordingly, Dh ′ and Dv ′ are obtained. In the correction intensity adjustment unit 57, no change is made to the direction of the FP correction amount.

  The correction strength coefficients kh and kv are set so as to satisfy 0 ≦ kh ≦ 1 and 0 ≦ kv ≦ 1. Dh ′ is the magnitude of the FP correction amount after adjustment for horizontal FP distortion, and Dv ′ is the magnitude of the FP correction amount after adjustment for vertical FP distortion. As for the horizontal FP distortion, the closer the coefficient kh is to zero, the smaller the FP correction amount Dh ′ after adjustment is, so the correction strength of the FP distortion correction process is smaller, and the closer the coefficient kh is to 1, the FP correction after adjustment. Since the magnitude Dh ′ of the quantity increases, the correction strength of the FP distortion correction process increases. The same applies to the vertical FP distortion.

If the above horizontal FP distortion in the input image I _n as the input image 320 is generated, the size Dh of FP correction amount is changed to the size Dh '. In this case, the correction processing unit 56 on the basis of the FP correction amount having a size of Dh ', the geometrical transformation to approximate the input image 310 input image 320, the input image I _n as an input image 320 Apply. FIG. 18 shows a distortion corrected image 320b obtained as a result. The distortion correction image 320b is a distortion correction image obtained when 0 <kh <1.

If the the horizontal lines of the input image 320 is formed from the first to M horizontal lines, in the right direction on the input image a i-th horizontal line of the input image I _n as the input image 320 (i-1) × The i-th horizontal line of the distortion-corrected image 320b is obtained by translating only (Dh ′ / (M−1)). Incidentally, by controlling the read start position at the time of reading the image data from the memory area 51 _A, may be generated distortion correction image 320b.

If the above vertical FP distortion in the input image I _n as the input image 320 is generated, the size Dv of the FP correction amount is changed to the size Dv '. In this case, the correction processing unit 56 on the basis of the FP correction amount having a size of Dv ', the geometrical transformation to approximate the input image 310 input image 330, the input image I _n as an input image 330 Apply. FIG. 19 shows a distortion correction image 330b obtained as a result. The distortion correction image 330b is a distortion correction image obtained when 0 <kv <1.

If the horizontal line of the input image 330 is formed from the first to Mth horizontal lines and the position 331 belongs to the M _O horizontal line (1 <M _O <M), the input image I as the input image 330 is input. The i-th horizontal line of the distortion-corrected image 330b is obtained by translating the _n- th horizontal line in the downward direction on the input image by (i−1) × (Dv ′ / (M _O −1)).

It has been separately described a method of correcting the horizontal FP distortion and vertical FP distortion, horizontal FP distortion and vertical FP distortion in in which case, the horizontal FP distortion and vertical FP distortion are both corrected in the input image I _n Thus, a distortion corrected image is generated. At this time, the horizontal FP distortion and the vertical FP distortion may be corrected simultaneously, or the horizontal FP distortion and the vertical FP distortion may be corrected in order.

  A plurality of types of information can be used as the adjustment information used for setting the correction strength coefficients kh and kv. Hereinafter, first to third setting methods will be described as examples of methods for setting the correction strength coefficients kh and kv based on the adjustment information.

[First setting method]
The first setting method will be described. As is apparent from the description of the first embodiment, the camera shake correction process by the correction processing unit 56 is a camera shake correction process for moving images. The user can freely specify to the imaging apparatus 1 using the operation unit 26 or the like whether to enable or disable the camera shake correction process. When it is specified that the camera shake correction process is to be enabled, the camera shake correction process is performed on the input image according to the method described in the first embodiment. When it is specified that the camera shake correction process is to be disabled, panning is performed. Regardless of the result of the tilt determination, the camera shake correction process is not always performed on the input image (the camera shake correction process is not performed also in step S15 in FIG. 16 described above and FIG. 22 described later).

  The adjustment information in the first setting method is information indicating whether the camera shake correction process is valid / invalid. Then, the correction strength coefficient is set so that kh and kv in the case where invalidation of the camera shake correction process is specified are smaller than kh and kv in the case where it is specified to enable the camera shake correction process. Set kh and kv. Specifically, when it is specified that the camera shake correction process is enabled, both kh and kv are set to 1, and when it is specified that the camera shake correction process is disabled, kh and kv. Can be set to any value within the range of 0 or more and 0.5 or less (in particular, it is also beneficial to set kh = kv = 0, and when kh = kv = 0, correction of FP distortion) The intensity is zero).

  When the camera shake correction process is disabled, the positional deviation between frames on the output moving image becomes larger than when the camera shake correction process is enabled. When the positional deviation between the frames is small, the FP distortion is visually noticeable on the output moving image, but when it is large, the FP distortion is not very noticeable. Considering this, in the first setting method, as described above, when the camera shake correction process is invalidated, the correction strength of the FP distortion is weakened or made zero.

[Second setting method]
The second setting method will be described. The adjustment information in the second setting method is information indicating the zoom magnification. When shooting at the wide end of the zoom magnification (that is, when shooting at a low zoom magnification), when shooting at the tele end of the zoom magnification (ie, shooting at a high zoom magnification) Are less affected by camera shake, movement of the subject in real space, and FP distortion. Accordingly, when shooting at the wide end of the zoom magnification, the FP distortion is small and the FP distortion is not conspicuous, so the correction strength of the FP distortion may be weak or zero.

For example, the correction strength coefficients kh and kv may be set from the zoom magnification ZF according to a broken line 410 as shown in the graph of FIG. The graph of FIG. 20 is a graph showing the relationship between ZF and kh and kv. That is, when the inequality “1 ≦ ZF <TH _A1 ” is satisfied, “kh = kv = 0” is set, and when the inequality “TH _A1 ≦ ZF <TH _A2 ” is satisfied, “kh = kv = (ZF −TH _A1 ) / (TH _A2 −TH _A1 ) ”, and when the inequality“ TH _A2 ≦ ZF ”is satisfied,“ kh = kv = 1 ”. Here, TH _A1 and TH _A2 take a predetermined value satisfying “TH _A1 <TH _A2 ”. The magnification TH _A1 is greater than or equal to the lower limit magnification (ie, 1 ×) of the variable range of the zoom magnification ZF, and the magnification TH _A2 is less than or equal to the upper limit magnification of the variable range of the zoom magnification ZF.

The zoom magnification ZF is the optical zoom magnification ZF _OPT when the optical zoom is used, the electronic zoom magnification ZF _EL when the electronic zoom is used, and the optical zoom magnification when both the optical zoom and the electronic zoom are used. It is the product of ZF _OPT and electronic zoom magnification ZF _EL . The user can instruct to change the zoom magnification ZF through the operation of the zoom button 26c in FIG.

The change of the optical zoom magnification ZF _OPT is realized by moving the zoom lens 30 in FIG. Based on the state optical zoom magnification ZF _OPT is some scale factor ZF _REF, if the optical zoom magnification ZF _OPT is the k _Z times the reference magnification ZF _REF, in comparison with the reference, the angle of view of the input image horizontal And (1 / k _Z ) times in each of the vertical directions.

The electronic zoom is realized by trimming an image according to the electronic zoom magnification ZF _EL at the stage of generating an input image from the image sensor 33 or the stage of generating an output image from the input image. For example, when the electronic zoom is performed at the stage of generating the output image from the distortion corrected image, the size of the clipping frame set in the distortion corrected image may be adjusted according to the electronic zoom magnification ZF _EL . In this case, based on the state electronic zoom magnification ZF _EL is some scale factor ZF _REF, if the electronic zoom magnification ZF _EL is the k _Z times the reference magnification ZF _REF, in comparison with the reference, the distortion correction image The horizontal and vertical sizes of the cutout frame to be set are each (1 / k _Z ) times.

[Third setting method]
A third setting method will be described. The adjustment information in the third setting method is the reliability of motion detection by the image motion detection unit 52. When the reliability is low, the correction strength of the FP distortion correction can be reduced or zero. In the image motion detection unit 52, motion vector detection between temporally adjacent input images is performed as motion detection, but the reliability of the motion detection changes depending on the condition of the subject. For example, when shooting a subject with relatively high contrast, a motion vector with high reliability can be obtained, while shooting a subject with relatively low contrast (eg, sky, sea, wall, etc.). In such a case, the reliability of the obtained motion vector is low.

  When detecting the motion vector, the image motion detection unit 52 also calculates the reliability RL of the detected motion vector, and includes the reliability RL in the image motion information. As a method for calculating the reliability RL of the motion vector detected based on the image data, any method including a known method (for example, a method described in Japanese Patent Laid-Open No. 2008-60927) can be used. Yes, the reliability RL can be calculated based on the contrast of the image in the motion detection block BL [i], the variation of the motion vectors MV [1] to MV [9] obtained for each motion detection block, and the like. (See FIG. 9).

For example, based on the motion vectors MV [1] to MV [9] obtained for the input images 220 and 230 in FIG.
σ = Σ {|| MV [i] −MV _AVE || ² }
Accordingly, the reliability RL can be calculated from the variation σ so that the reliability RL decreases as the variation σ increases (for example, RL = 1 / σ). Here, MV _AVE is an average vector of vectors MV [1] to MV [9], and || MV [i] −MV _AVE || is a norm of the vector (MV [i] −MV _AVE ). , Σ are || MV [1] -MV _AVE || ² , || MV [2] -MV _AVE || ² ,..., || MV [8] -MV _AVE || ² and || MV [9] is the sum of -MV _AVE || ^2.

  On the principle of calculating a motion vector based on image data, σ decreases if all subjects within the shooting range are stationary, but increases when a subject occupying a part of the shooting range moves. The FP distortion is a distortion caused by the movement of the imaging apparatus 1, and the moving component of the subject is not involved in the FP distortion. Accordingly, when the overall motion vector and the FP correction amount are derived by averaging the motion vectors MV [1] to MV [9], if the variation σ is large and the reliability RL is low, a relatively large error occurs. The possibility of being included in the FP correction amount is high.

In consideration of this, the correction strength coefficients kh and kv may be set from the reliability RL according to the broken line 420 as shown in the graph of FIG. The graph of FIG. 21 is a graph showing the relationship between RL and kh and kv. That is, when the inequality “1 ≦ RL <TH _B1 ” is satisfied, “kh = kv = 0” is set, and when the inequality “TH _B1 ≦ RL <TH _B2 ” is satisfied, “kh = kv = (RL −TH _B1 ) / (TH _B2 −TH _B1 ) ”, and when the inequality“ TH _B2 ≦ RL ”is satisfied,“ kh = kv = 1 ”. Here, TH _B1 and TH _B2 take a predetermined value satisfying “TH _B1 <TH _B2 ”. The magnification TH _B1 is not less than the lower limit value of the variable range of the reliability RL, and the magnification TH _B2 is not more than the upper limit value of the variable range of the reliability RL.

  Note that the kh and kv setting method based on the reliability RL is effective only when the FP distortion correction process is performed based on the image motion information, and when the FP distortion correction process is performed based on the sensor motion information. Kh and kv are fixed at 1.

[Operation flow]
FIG. 22 is a flowchart illustrating an operation procedure of the imaging apparatus 1 according to the second embodiment. As can be seen from the comparison between FIG. 16 and FIG. 22, in the second embodiment, steps S14 and S16 in the first embodiment are replaced with steps S14a and S16a, respectively. The operation procedure is the same between the first and second embodiments.

  If it is determined in step S13 that there is no pan / tilt, the process proceeds from step S13 to step S14a. In step S14a, after the FP correction amount based on the image motion information is calculated for the latest input image, the FP correction amount is adjusted based on the adjustment information, and based on the adjusted FP correction amount. FP distortion correction processing is performed. An output image is generated by performing camera shake correction processing in step S15 on the distortion corrected image obtained by the FP distortion correction processing.

  If it is determined in step S13 that there is pan / tilt, the process proceeds from step S13 to step S16a. In step S16a, after the FP correction amount based on the image motion information is calculated for the latest input image, the FP correction amount is adjusted based on the adjustment information, and based on the adjusted FP correction amount. FP distortion correction processing is performed. If it is determined in step S13 that there is pan / tilt, camera shake correction processing is not performed as in the first embodiment. That is, without changing the position of the cutout frame for the distortion correction image obtained by the FP distortion correction processing (in other words, without changing the position of the cutout frame between the previous and current distortion correction images), An output image is generated by cutting out the image within the cutout frame of the distortion corrected image.

  According to this embodiment, the same effect as that of the first embodiment can be obtained. Furthermore, when it is determined that the FP distortion is not conspicuous, or when it is determined that the error of the FP distortion correction is likely to be large, the FP distortion correction strength is reduced to reduce the FP distortion correction according to the situation. Can be achieved. In particular, if the FP distortion correction intensity is set to zero and the FP distortion correction process itself is not performed according to the adjustment information, the calculation load can be reduced and the power consumption can be reduced accordingly.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 4 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the above example, nine motion detection blocks are set in the input image by dividing the entire image area of the input image into nine. However, the number of motion detection blocks to be set in the input image is other than nine. There may be.

[Note 2]
In the example described above with reference to FIG. 6, the output signal of the light receiving pixel at the position (x, y) on the effective pixel area 33 _A is stored at the address (x, y) on the memory area 51 _A for the input image. Is done. That is, in the above example, it is assumed that the input image is generated by reading the output signal of the light receiving pixels on the effective pixel region 33 _A All individual. However, it may be generated an input image to be read or the addition reading decimated output signals of the light receiving pixels in the effective pixel region 33 _A.

The skip readout, only the output signal of the part of the light receiving pixels in the effective pixel region 33 _A is read out as image data of the input image. In addition reading, the reading of the output signal of the light receiving pixels on which was considered as the output signal of one light receiving pixel addition signal of the output signals of a plurality of light receiving pixels in the effective pixel region 33 _A is made. When generating an input image by using a thinning-out reading or addition reading, the number of pixels of the input image is smaller than the number of light receiving pixels in the effective pixel region 33 _A.

[Note 3]
In each of the above-described embodiments, when performing the FP distortion correction process based on the image motion information, an overall motion vector is calculated by averaging the motion vectors calculated for each motion detection block, and the FP based on the overall motion vector is obtained. FP distortion correction processing is performed with the correction amount. However, the method for performing the FP distortion correction process based on the image motion information is not limited to this. In other words, when performing FP distortion correction processing based on image motion information, for example, a method described in Japanese Patent Application Laid-Open No. 2006-186482, for each motion detection block based on a motion vector calculated for each motion detection block. FP distortion correction processing may be performed.

[Note 4]
The imaging apparatus 1 in FIG. 1 can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device (for example, a computer).

DESCRIPTION OF SYMBOLS 1 Image pick-up device 11 Image pick-up part 33 Image pick-up element 33 _A effective pixel area 50, 50a Image correction function part 51 Frame memory 52 Image motion detection part 53 Sensor part 54 Pan / tilt determination part 55 FP correction amount calculation part 56 Correction processing part 57 Correction Strength adjustment section

Claims (6)

In an imaging apparatus including an imaging element that performs imaging while generating different exposure timings between different horizontal lines,
A motion detector for detecting an optical flow between a plurality of input images based on the output of the image sensor based on the output of the image sensor;
A sensor unit for detecting movement of the imaging device;
Select either the detection result of the motion detection unit or the detection result of the sensor unit based on both or one of the detection result of the motion detection unit and the detection result of the sensor unit, and based on the selected detection result An image pickup apparatus comprising: a correction unit that generates an output image by correcting an input image. The imaging apparatus according to claim 1, wherein the correction unit corrects distortion generated in the input image due to different exposure timings between different horizontal lines. The correction unit further corrects a shake on the input moving image generated by a movement of the imaging apparatus during a shooting period of the input moving image including a plurality of input images,
The imaging apparatus according to claim 2, wherein the blur correction is performed based on a detection result of the motion detection unit or a detection result of the sensor unit. A determination unit for determining the presence or absence of panning or tilting of the imaging device based on both or one of the detection result of the motion detection unit and the detection result of the sensor unit;
The correction unit selects the detection result of the motion detection unit when it is determined that there is panning or tilting, and selects the detection result of the sensor unit otherwise. The imaging device according to any one of claims 1 to 3. The correction unit includes a distortion correction unit that corrects distortion generated in the input image due to different exposure timings between different horizontal lines, and the imaging apparatus during the shooting period of the input moving image including a plurality of input images. A shake correction unit that corrects a shake on the input moving image generated by the movement;
The distortion correction is performed based on the selected detection result, while the blur correction is performed based on the detection result of the motion detection unit or the detection result of the sensor unit,
The correction unit is
When panning or tilting is not performed on the imaging device, the output image is generated from the input image using the distortion correction unit and the shake correction unit,
2. The output image is generated from the input image using the distortion correction unit without using the blur correction unit when panning or tilting is performed on the imaging apparatus. Imaging device. The imaging apparatus according to claim 2, wherein the correction unit is formed to be capable of changing a strength of correcting the distortion.

Images