JP2019083407A - Image blur correction device and control method therefor, and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for performing good image shake correction regardless of an estimation state of three-dimensional coordinates of a feature point of a captured image, a control method thereof, and an imaging device. An image pickup apparatus detects runout information and detects a motion vector from a captured image signal. The feature point tracking unit 210 calculates the coordinate values of the same feature points as tracking information in the captured image based on the motion vector. The tracking period calculation unit 209 calculates the tracking period of the feature points based on the parallax amount between the captured images and the feature points. The integrated processing unit 205 estimates the position / orientation information of the image pickup device based on the runout information and the tracking information of the feature points, and estimates the three-dimensional coordinates of the feature points showing the positional relationship including the depth between the subject and the image pickup device. The target position calculation unit 212 synthesizes the shake information and the position / orientation information of the image pickup device according to the contribution of the three-dimensional coordinate estimation based on the tracking period, and calculates the target position of the drive control of the image shake correction lens 103. As the tracking period becomes longer, the contribution of the 3D coordinate estimation is controlled to be higher. [Selection diagram] Fig. 2

Description

  The present invention relates to a technique for correcting image blurring in an imaging device such as a video camera and a digital still camera.

  At the time of imaging by a digital camera or the like, an image blur of a subject may occur due to a camera shake of a user who holds the camera body. Image blur correction performed by the imaging apparatus includes optical image blur correction and electronic image blur correction. For example, in the optical image shake correction, control is performed to detect a vibration applied to the camera body with an angular velocity sensor or the like and move a correction lens provided in the imaging optical system according to the detection result. Image blurring is corrected by changing the optical axis direction of the imaging optical system to move the image formed on the light receiving surface of the imaging device. In electronic image shake correction, image shake correction processing is performed in a pseudo manner by image processing on a captured image.

  In order to correct an image blur generated due to camera shake or the like, it is necessary to detect a change in position and orientation of the imaging device. As a self-position estimation method for detecting an attitude and a position of an imaging device, there is a method to which a SFM (Structure from Motion) and a position and orientation estimation (Visual and Inertial Sensor Fusion) technology using an inertial sensor are applied. There is known a method of estimating the three-dimensional coordinates of an object existing in a real space (real space) and the position and orientation of an imaging device. Patent Document 1 discloses a technique of tracking the movement of feature points and correcting an image blur using the tracking result.

JP, 2016-119572, A

In the prior art, when the amount of parallax between captured images is small, there is a possibility that three-dimensional coordinate estimation of an object by SFM may fail. In Patent Document 1, reduction in the amount of parallax is avoided by selecting an image to be tracked such that the amount of parallax between images is large. However, this technique can not take into consideration the comparison result between images with small parallax amounts. Therefore, a period in which the process is not performed may occur, and a period in which the image blur correction is not controlled may occur. It is difficult to correct the image blur particularly when the image blur occurs over a long period of time due to fine hand movement.
An object of the present invention is to perform good image blur correction regardless of the estimation state of three-dimensional coordinates in image blur correction using three-dimensional coordinates of feature points of a captured image.

  An apparatus according to an embodiment of the present invention is an image shake correction apparatus that corrects an image shake generated by a shake applied to an imaging apparatus by a shake correction unit, and captures an image using motion vector information obtained from an image signal of a captured image. Tracking means for extracting and tracking feature points in an image and calculating coordinates of the feature points as tracking information, shake information related to shake applied to the imaging device, and position and orientation information of the imaging device using the tracking information A second estimation means for estimating the three-dimensional coordinates of the feature point indicating the positional relationship including the depth between the subject and the imaging device using the position and orientation information; The first calculation means for calculating the tracking period of the feature point from the amount of parallax between the captured images of the first and second feature points, and the tracking period obtained from the first calculation means is long when the tracking period is long. A second calculation of calculating a target position of the shake correction unit by combining the shake information and the position and attitude information by setting the ratio of combining the position and attitude information to the shake information higher than when the period is short And means for controlling the shake correcting means in accordance with the target position.

  According to the present invention, in image blur correction using three-dimensional coordinates of feature points of a captured image, it is possible to perform good image blur correction regardless of the estimated state of three-dimensional coordinates.

It is a block diagram showing an example of composition of an imaging device of a 1st embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of an image shake correction apparatus. 5 is a flowchart illustrating processing of the image shake correction apparatus. It is a figure which shows the structural example of a feature point tracking part. It is a flowchart explaining a feature point tracking process. It is explanatory drawing of feature point extraction. It is explanatory drawing of template matching. It is explanatory drawing of a correlation value map. It is a figure explaining the expression method of a correlation value map. It is a figure explaining the correlation value index showing the tracking reliability of a feature point. It is a figure explaining the calculation method of tracking reliability of a feature point. It is explanatory drawing of a perspective projection model. It is a figure explaining the estimation method of a three-dimensional coordinate. It is a flowchart of a correction process of a position and orientation estimated value. It is a figure which shows the structural example of a target position calculation part. It is a flowchart of target position calculation processing. It is explanatory drawing of the gain amount with respect to a tracking period. It is a figure which shows the structural example of the image blurring correction | amendment apparatus of 2nd Embodiment. It is a flowchart explaining the process of the image blurring correction apparatus of 2nd Embodiment. It is a figure which shows the structural example of the motion vector calculating part of 2nd Embodiment. It is a flowchart of the process which the motion vector calculating part of 2nd Embodiment performs. It is a figure which shows the structural example of the target position calculation part of 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. The present invention is applicable to an imaging apparatus such as a video camera, a digital camera, and a silver halide still camera, and an electronic apparatus having an imaging unit. Hereinafter, an operation of correcting an image shake using a shake detection signal of an imaging device is referred to as an “image shake correction operation”.

First Embodiment
FIG. 1 is a view showing an example of the arrangement of an image pickup apparatus according to this embodiment. The imaging device shown in FIG. 1 is, for example, a digital still camera. Note that the imaging device may have a moving image shooting function. The imaging device includes a zoom unit 101. The zoom unit 101 is a part of a variable magnification imaging lens that constitutes an imaging optical system (imaging optical system), and includes a zoom lens that changes the imaging magnification. The zoom drive unit 102 drives the zoom unit 101 in accordance with the control signal from the control unit 117.

  An image shake correction lens (hereinafter, also simply referred to as a correction lens) 103 is a correction member (shake correction means) used for correcting an image shake caused by a shake applied to an imaging device. The correction lens 103 is movable in a direction orthogonal to the optical axis direction of the photographing lens. The image shake correction lens drive unit 104 drives the correction lens 103 in accordance with the control signal from the control unit 117. The aperture / shutter unit 105 includes a mechanical shutter having an aperture function. The diaphragm / shutter driving unit 106 drives the diaphragm / shutter unit 105 in accordance with the control signal from the control unit 117. The focus lens 107 used for focusing is a part of the photographing lens, and can change its position along the optical axis of the photographing lens. The focus drive unit 108 drives the focus lens 107 in accordance with the control signal from the control unit 117.

  The imaging unit 109 includes an imaging element such as a CCD image sensor or a CMOS image sensor. CCD is an abbreviation of "Charge Coupled Device", and CMOS is an abbreviation of "Complementary Metal-Oxide Semiconductor". The imaging element converts an optical image formed by the imaging optical system into an electrical signal in pixel units.

  The imaging signal processing unit 110 performs A (Analog) / D (Digital) conversion, correlated double sampling, gamma correction, white balance correction, color interpolation processing, and the like on the electrical signal output from the imaging unit 109, and Convert to a signal. The video signal processing unit 111 processes the video signal output from the imaging signal processing unit 110 according to the application. Specifically, the video signal processing unit 111 generates video data for display, and performs encoding processing and data filing for recording. The display unit 112 performs image display as needed based on the display video signal output from the video signal processing unit 111.

  The power supply unit 113 supplies power to the entire imaging apparatus according to the application. The external input / output terminal unit 114 is used to input / output communication signals and video signals to / from an external device. The operation unit 115 includes a button, a switch, and the like for the user to give an instruction to the imaging apparatus. The storage unit 116 stores various data including video information and the like.

  The control unit 117 includes, for example, a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM), and controls the entire imaging apparatus. The control program stored in the ROM is expanded on the RAM and executed by the CPU to control each part of the imaging apparatus, and various operations described below are realized. The shake detection unit 118 includes a sensor that detects a shake applied to the imaging device, and outputs a shake detection signal to the control unit 117.

  The operation unit 115 has a release switch configured such that the first switch (denoted as SW1) and the second switch (denoted as SW2) are sequentially turned on in accordance with the pressing amount of the release button. When the release button is pressed halfway, SW1 turns on, and when the release button is fully pressed, SW2 turns on. When the switch SW1 is turned on, the control unit 117 calculates an AF (Auto Focus) evaluation value based on the display video signal output from the video signal processing unit 111 to the display unit 112. Then, the control unit 117 performs automatic focus detection and focus adjustment control by controlling the focus drive unit 108 based on the AF evaluation value. Further, the control unit 117 performs AE (automatic exposure) processing for determining an aperture value and a shutter speed for obtaining an appropriate exposure amount based on luminance information of the video signal and a predetermined program chart. When the switch SW2 is turned on, the control unit 117 performs imaging with the determined aperture value and shutter speed, and controls each processing unit to store the image data obtained by the imaging unit 109 in the storage unit 116.

  The operation unit 115 also has an operation switch used to select an image blur correction (vibration reduction) mode. When the image blur correction mode is selected by the operation of the operation switch, the control unit 117 acquires a shake detection signal from the shake detection unit 118 and instructs the image blur correction lens driving unit 104 to perform an image blur correction operation. The image shake correction lens drive unit 104 that has received the control command from the control unit 117 performs an image shake correction operation until an instruction to turn off the image shake correction is issued.

  The operation unit 115 has a shooting mode selection switch capable of selecting a still image shooting mode or a moving image shooting mode. A shooting mode selection process is performed by a user operation of the shooting mode selection switch, and the control unit 117 changes the operating condition of the image shake correction lens driving unit 104. The operation unit 115 also has a playback mode selection switch for selecting a playback mode. When the reproduction mode is selected by the user operation of the reproduction mode selection switch, the control unit 117 performs control to stop the image blur correction operation. In addition, the operation unit 115 includes a magnification change switch that issues a zoom magnification change instruction. When an instruction to change the zoom magnification is given by the user operation of the magnification change switch, the zoom drive unit 102 that has received the instruction via the control unit 117 drives the zoom unit 101, and the zoom lens is moved to the instructed position. Move it.

  FIG. 2 is a view showing an example of the arrangement of the image shake correction apparatus of this embodiment. The image shake correction apparatus detects shake applied to the imaging device and drives the image shake correction lens 103 to perform image shake correction. The first vibration sensor 201 is, for example, an angular velocity sensor, and the vertical direction (pitch direction), horizontal direction (yaw direction), light of the imaging apparatus in the normal posture (the posture in which the longitudinal direction of the captured image substantially matches the horizontal direction) Vibrations in the direction of rotation around the axis (rolling direction) are detected. The second vibration sensor 203 is, for example, an acceleration sensor, and detects the movement amount in the vertical direction, the movement amount in the horizontal direction, and the movement amount in the optical axis direction in the normal posture.

  The A / D converter 202 converts an analog value detected by the first vibration sensor 201 into a digital value, and outputs the digital value to a target position calculation unit 212 and a subtractor 215 described later. The A / D converter 204 converts an analog value detected by the second vibration sensor 203 into a digital value, and outputs the digital value to the integration processing unit 205 and the tracking period calculation unit 209 described later. Further, the A / D converter 217 converts an analog value indicating the position of the correction lens detected by the position detection sensor 211 into a digital value, and outputs the digital value to a subtractor 213 and a differentiator 216 described later.

  The output signal of the imaging unit 109 is processed by the imaging signal processing unit 110, and the feature point tracking unit 210 further processes it. The feature point tracking unit 210 extracts a feature point from the video signal input from the imaging signal processing unit 110 based on the amount of parallax between the plurality of captured images estimated by the tracking period calculation unit 209, and performs tracking processing I do.

  The main subject separation unit 208 separates the feature points belonging to the main subject region and the feature points belonging to the background region other than the main subject from the feature points acquired from the feature point tracking unit 210. Data of the separated feature points is output to the tracking period calculation unit 209 and the integration processing unit 205. The tracking period calculation unit 209 obtains each output of the main subject separation unit 208 and the A / D converter 204, and calculates the parallax amount and the tracking period between the plurality of captured images. The tracking period calculation unit 209 calculates the parallax amount using the tracking information of the feature points belonging to the background area separated by the main subject separation unit 208, and calculates the tracking period of the feature points based on the parallax amount. The tracking information is, for example, information indicating coordinates of feature points extracted in a captured image and to be tracked.

  The integration processing unit 205 includes a three-dimensional coordinate estimation unit 206 and a position and orientation estimation unit 207. The integration processing unit 205 integrates the detection information by the first vibration sensor 201 and the second vibration sensor 203 and the image information by the main subject separation unit 208, and estimates the position and orientation of the imaging apparatus and the three-dimensional coordinates of the feature points. . The position and orientation estimation unit 207 receives a signal obtained by subtracting the output of the differentiator 216 from the output of the A / D converter 202 by the subtractor 215 and an output signal of the A / D converter 204. Details of the three-dimensional coordinate estimation and the position and orientation estimation will be described later.

  The target position calculation unit 212 acquires a shake detection signal (angular velocity signal) representing a shake in the pitch direction and the yaw direction of the imaging device obtained from the first vibration sensor 201 and position and orientation information output from the integration processing unit 205. The target position calculation unit 212 acquires information on the tracking period calculated by the tracking period calculation unit 209 via the integration processing unit 205. The target position calculation unit 212 generates a correction position control signal for driving the image shake correction lens 103 in the pitch direction or the yaw direction based on the acquired information. The correction position control signal is supplied to the control filter 214 through the subtracter 213. The subtractor 213 subtracts the position detection signal in the pitch direction and the yaw direction of the correction lens, which is the output of the A / D converter 217, from the acquired correction position control signal, and outputs the subtracted signal to the control filter 214.

  The control filter 214 performs feedback control relating to driving of the image blur correction lens 103 in accordance with the output of the subtractor 213. As a result, control of the actuator in the image shake correction lens drive unit 104 is performed such that the position detection signal value of the correction lens converges to the correction position control signal value by the target position calculation unit 212.

  The image blur correction process of the present embodiment will be described with reference to the flowchart of FIG. 3. The processes of S301 to S304 and the processes of S305 to S309 are executed as parallel processes. In step S301, the imaging unit 109 photoelectrically converts an optical signal of a subject image into an electrical signal, and acquires an image signal of the captured image. In S302, the imaging signal processing unit 110 converts an image signal from an analog signal to a digital signal, and performs predetermined image processing. The feature point tracking unit 210 tracks feature points in a captured image and calculates the reliability of feature point tracking (hereinafter, referred to as tracking reliability). Details of the internal configuration and processing of the feature point tracking unit 210 will be described later.

  In step S303, the main subject separation unit 208 separates feature points belonging to the main subject area and feature points belonging to background areas other than the main subject, with respect to the feature points tracked in step S302. For separation of feature points, for example, clustering by a known K-means method or an EM algorithm is used.

  In step S304, the tracking period calculation unit 209 estimates the amount of parallax between the plurality of captured images using the feature coordinates of the background area separated in step S302. In the present embodiment, the amount of movement on the image plane due to the translational movement of the imaging device is calculated as the amount of parallax. Specifically, an inter-frame displacement amount of feature coordinates belonging to the background area is calculated as a parallax amount. Although the process of calculating the amount of parallax from the image signal will be described, the amount of translational movement of the imaging device may be calculated from the shake acceleration by the second vibration sensor 203 calculated in S308 described later and used as the amount of parallax. Further, the accuracy of estimation of the amount of parallax can be further enhanced by considering the depth direction as to the amount of movement on the image plane from the three-dimensional position coordinates estimated in S310 described later. The tracking period of the feature point is calculated based on the calculated amount of parallax. Specifically, the tracking period is calculated to be shorter as the amount of parallax between images is smaller.

  The processing of S305 to S309 is processing in which the position and orientation estimation unit 207 estimates the position and orientation of the imaging device. In S305, the first vibration sensor 201 detects a shake angular velocity. The arrow from S305 to S312 indicates that the process of outputting the detected shake angular velocity to the target position calculation unit 212 is performed. In S306, the differentiator 216 calculates the moving speed of the image shake correction lens 103 by calculating the difference between the shooting frames related to the detection position of the image shake correction lens 103.

  In step S307, the subtractor 215 subtracts the moving speed of the image shake correction lens 103 from the angular velocity information of the first vibration sensor 201. Thus, information corresponding to the shake correction remaining angular velocity of the imaging device is calculated and output to the position and orientation estimation unit 207. In S308, the second vibration sensor 203 detects a shake acceleration. In step S309, the position and orientation estimation unit 207 estimates the position and orientation of the imaging device in the real space based on the information on the shake correction remaining angular velocity calculated in step S307 and the information on the shake acceleration detected in step S308. Since this position and orientation are estimated using the shake correction remaining angular velocity, it corresponds to the correction remaining position and posture after image shake correction.

  After the processing of S304 and S309, the process proceeds to S310. The three-dimensional coordinate estimation unit 206 determines the depth of the feature point in the real space based on the position and orientation information of the imaging device estimated by the position and orientation estimation unit 207 in S309 and the feature point information of the background area obtained in S302. Estimate three-dimensional coordinates including. That is, processing for estimating and calculating three-dimensional coordinates of feature points indicating a positional relationship including the depth of the subject and the imaging device is executed. The position and orientation estimation unit 207 determines the position of the shake correction remaining in step S309 based on the three-dimensional coordinates of the feature point estimated in step S310 and the two-dimensional feature point coordinates on the captured image of the background area obtained in step S303. Correct the pose estimate. Details of the method of estimating the three-dimensional coordinates and the method of correcting the estimated value will be described later.

  The target position calculation unit 212 in S312 uses the tracking period acquired in S304, the shake angular velocity signal acquired in S305, and the position / posture estimation value of the shake correction remaining in S311 to obtain the image blur correction lens 103. A correction position control signal related to driving is generated. The details will be described later. In step S313, the image shake correction lens driving unit 104 drives the image shake correction lens 103 based on the target position indicated by the correction position control signal generated in step S312. The processing of each step of S302, S310, S311, and S312 will be described in detail below.

  Details of feature point tracking in S302 of FIG. 3 will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram illustrating an internal configuration of the feature point tracking unit 210. The feature point extraction unit 401 acquires an image signal from the imaging signal processing unit 110 and extracts feature points from the image. The feature point setting unit 402 sets feature points to be tracked based on the outputs of the feature point extraction unit 401, the motion vector detection unit 404, and the tracking reliability calculation unit 405.

  The image memory 403 temporarily stores the image signal input from the imaging signal processing unit 110, and accumulates the image signals of a plurality of images having different imaging times. The motion vector detection unit 404 detects a motion vector for the image signal input from the imaging signal processing unit 110 and the image memory 403 based on the feature points set by the feature point setting unit 402. The tracking reliability calculation unit 405 acquires the motion vector detected by the motion vector detection unit 404, and calculates the reliability of feature point tracking, that is, the tracking reliability.

FIG. 5 is a flowchart illustrating processing of the feature point tracking unit 210.
In S501, the feature point extraction unit 401 extracts feature points from the input image from the imaging signal processing unit 110. FIG. 6 is a diagram for explaining an example of feature point extraction. The feature point extraction unit 401 extracts, for example, a predetermined number of feature points for each of a plurality of divided image areas. In FIG. 6A, a white rectangular area is a feature extraction area for performing feature point extraction. A peripheral region indicated by hatching is provided around the feature point extraction region. Depending on the position of the feature point to be extracted, a template region and a search region used for motion vector detection to be described later extend beyond the feature extraction region. Therefore, a peripheral area indicated by oblique lines is provided as an extra image area by an amount corresponding to the protrusion. FIG. 6B shows an example of processing for extracting one feature point 601 for each of the image areas divided in a lattice.

As a method of extracting feature points, there is a commonly known method such as Harris corner detector or Shi and Tomasi. The luminance value at the pixel position (x, y) of the image is expressed as I (x, y). Then, an autocorrelation matrix H represented by equation (1) is created from the results Ix and Iy of applying the horizontal and vertical first derivative filters to the image.

In Formula (1), G represents the function of smoothing by the Gaussian distribution shown in Formula (2).

式（３）において、“ｄｅｔ”は行列式を表し、“ｔｒ”は対角成分の和を表す。また、αは定数であり、実験的に０．０４〜０．１５の値が良いとされる。 The Harris corner detector extracts, as a feature point, a pixel whose feature amount becomes large according to the feature evaluation equation shown in Equation (3).

In equation (3), "det" represents the determinant and "tr" represents the sum of diagonal components. Also, α is a constant, and a value of 0.04 to 0.15 is experimentally good.

式（４）は、式（１）の自己相関行列Ｈの固有値λ１，λ２のうち、小さい方の固有値を特徴量とすることを表す。Ｓｈｉ ａｎｄ Ｔｏｍａｓｉを用いる場合でも、特徴量が大きくなる画素を特徴点として抽出する処理が行われる。式（３）または式（４）により画素の特徴量が算出され、特徴量が高い方から所定数の画素が特徴点として抽出される。 On the other hand, in the method of Shi and Tomasi, the feature evaluation formula shown in Formula (4) is used.

Expression (4) represents that the smaller one of the eigenvalues λ1 and λ2 of the autocorrelation matrix H of Expression (1) is used as the feature value. Even in the case of using Shi and Tomasi, a process of extracting a pixel whose feature amount becomes large as a feature point is performed. The feature amount of the pixel is calculated by the equation (3) or the equation (4), and a predetermined number of pixels are extracted as feature points from the one having the higher feature amount.

  In S502 of FIG. 5, the feature point setting unit 402 sets feature points to be tracked. With regard to the initial frame, the feature points newly extracted in S501 may be set as tracking targets as they are. The setting method for the second and subsequent frames will be described later.

  In S503, the motion vector detection unit 404 detects a motion vector using the feature points set as tracking targets in S502. As a method of detecting a motion vector, there are known a correlation method and a block matching method. Any known method can be applied as a method of calculating a motion vector. An application example of the block matching method will be described below.

  FIG. 7 is a schematic view showing an outline of the block matching method. FIG. 7A shows a reference image of two vector detection images, and FIG. 7B shows a reference image. In this example, by using a frame image held in the image memory 403 as a reference image and image data directly input from the imaging signal processing unit 110 as a reference image, movement from a past frame image to a current frame image A vector is calculated. The reference image and the reference image may be interchanged and applied. To replace and apply the reference image and the reference image means to calculate a motion vector from the current frame image to the past frame image.

  The motion vector detection unit 404 arranges the template area 701 in the reference image, arranges the search area 702 in the reference image, and calculates a correlation value between the template area 701 and the search area 702. The template area 701 is disposed centering on the feature points set in S502 of FIG. 5, and the search area 702 is disposed with a predetermined size so that the template area is uniformly included in the upper, lower, left, and right directions. Good.

In this embodiment, a sum of absolute differences (hereinafter abbreviated as SAD) is used as a method of calculating the correlation value. The calculation formula (5) of SAD is shown.

  In Formula (5), f (i, j) shows the luminance value in the coordinate (i, j) in the template area | region 701. As shown in FIG. Further, g (i, j) represents a luminance value at each coordinate in an area (hereinafter referred to as a correlation value calculation area) 703 for which a correlation value is to be calculated in the search area 702. In SAD, the absolute value of the difference between the brightness values f (i, j) and g (i, j) in both regions 702 and 703 is calculated, and the sum is obtained to obtain the correlation value S_SAD. The smaller the value of the correlation value S_SAD, the higher the similarity of the texture between the template area 701 and the correlation value calculation area 703. As a method other than SAD, for example, a method using differential sum of squares (SSD) or normalized cross correlation (NCC) may be used to calculate the correlation value.

  The motion vector detection unit 404 moves the correlation value calculation area 703 all over the search area 702 to calculate a correlation value. Thereby, the correlation value map shown in FIG. 8 is created for the search area 702.

  FIG. 8 is a diagram showing an example of the correlation value map. FIG. 8A shows a correlation value map calculated in the coordinate system of the search area 702. FIG. The X axis and the Y axis represent correlation value map coordinates, and the Z axis represents the magnitude of the correlation value at each coordinate. Moreover, FIG. 8 (B) is a figure which shows the contour line of FIG. 8 (A).

  In FIGS. 8A and 8B, the smallest correlation value is the local minimum value 801, and in the area where the local minimum value 801 is calculated in the search area 702, a texture very similar to the template area 701 is obtained. It can be determined that it exists. The minimum value 802 is a second minimum value, and the minimum value 803 is a third minimum value. In the area in which the local minimum values 802 and 803 are calculated, a texture similar to the area in which the local minimum value 801 is calculated exists next. As described above, the motion vector detection unit 404 calculates the correlation value between the template area 701 and the search area 702, and determines the position of the correlation value calculation area 703 where the correlation value is the smallest. Thereby, the motion vector detection unit 404 specifies the movement destination of the template area 701 on the reference image on the reference image. Then, the motion vector detection unit 404 detects a motion vector whose direction and amount of movement to the movement destination on the reference image are based on the position of the template region on the reference image.

  In S504 of FIG. 5, the tracking reliability calculation unit 405 calculates the tracking reliability using at least one of the feature point information obtained in S501 and the correlation value information obtained in S503. The correlation value information is the calculation result of the correlation value performed when calculating motion vector information. First, an example of calculating tracking reliability using correlation value information will be described. FIG. 9 is a diagram in which correlation values are arranged in raster order and represented in one dimension as indicated by an arrow 804 in the two-dimensional correlation value map of FIG. 8 (B). The vertical axis in FIG. 9 represents a correlation value, and the horizontal axis represents a pixel address uniquely determined by the X coordinate and Y coordinate of the correlation value map. Hereinafter, in order to calculate the tracking reliability, the expression of FIG. 9 is used. The position 901 is a position corresponding to the minimum value 801 shown in FIG.

  FIG. 10 shows an example of a measure of correlation value indicating tracking reliability. The horizontal axis of FIG. 10 represents the address of the pixel, and the vertical axis represents the correlation value. In the example shown in FIG. 10A, the difference Da between the minimum value and the maximum value of the correlation value is used as the index. The difference Da represents the range of the correlation value map, and indicates that the tracking reliability is low when the difference Da is small, for example, when the texture contrast is low. In the example shown in FIG. 10B, a ratio Db (= B / A) of the difference A between the minimum value and the maximum value of the correlation value and the difference B between the minimum value and the average value is used as an index. The ratio Db represents the steepness of the correlation value peak, and indicates that the tracking reliability is low when the ratio Db is small, for example, the similarity between the template region and the search region is low.

  In the example shown in FIG. 10C, the difference Dc between the minimum value (minimum value) of the correlation value and the second minimum value is used as the index. The correlation values 1001, 1002 and 1003 correspond to the local minimum values 801, 802 and 803 in FIG. 8, respectively. Therefore, in FIG. 10 (C), it can be confirmed whether or not there is a minimum value similar to the minimum value of the correlation value in the contour line of FIG. 8 (B). The difference Dc represents the periodicity of the correlation value map, and indicates that the tracking reliability is low when the difference Dc is small, for example, when the texture is a repetitive pattern or an edge. Although the minimum value and the second minimum value are selected in this example, other minimum values may be selected as long as the periodicity of the correlation value map can be determined.

  In the example shown in FIG. 10D, the minimum value Dd of the correlation value is used as an index. When the minimum value Dd is large, for example, the similarity between the template region and the search region is low, it indicates that the tracking reliability is low. Since the minimum value Dd and the tracking reliability are in inverse proportion to each other, the reciprocal of Dd (1 / Dd) is used as an index.

  The index of the correlation value can be used as the reliability as it is, but there is a method of calculating the reliability from the index of the correlation value. For example, in FIG. 11A, the correlation value index D is associated with the tracking reliability. The correlation value index D shown on the horizontal axis of FIG. 11A represents any of Da, Db, Dc, and 1 / Dd, and the vertical axis represents the tracking reliability R1. In this example, two threshold values T1 and T2 (T1 <T2) are provided. When the D value is T1 or less, the value of R1 is zero, and when it is T2 or more, the value of R1 is one. The threshold values T1 and T2 may be changed for each correlation value index. Further, in a section where the D value is between the threshold values T1 and T2, D and R1 are in a linear relationship, and the value of R1 increases as the D value increases. Alternatively, the correlation value index D and the tracking reliability R1 may be associated with a non-linear relationship. In the following description, the degrees of reliability obtained from the correlation value indicators Da, Db, Dc, and 1 / Dd are denoted as Ra, Rb, Rc, and Rd, respectively. Here, using an arbitrary function (denoted as f), the relationship of Ra = f (Da), Rb = f (Db), Rc = f (Dc), Rd = f (Dd) is established.

例えば、重みをＷａ＝０．４，Ｗｂ＝０．３，Ｗｃ＝０．２，Ｗｄ＝０．１とする。全ての信頼度が十分に高く、Ｒａ＝Ｒｂ＝Ｒｃ＝Ｒｄ＝１の場合には、式（６）よりＲ１＝１．０となる。またＲａ＝０．６，Ｒｂ＝０．５，Ｒｃ＝０．７，Ｒｄ＝０．７の場合には、式（６）よりＲ１＝０．６となる。 The final tracking reliability R1 is calculated by combining Ra, Rb, Rc, and Rd. The weighted addition method and the logical operation method will be described. In the weighted addition method, the weights of Ra, Rb, Rc, and Rd are denoted as Wa, Wb, Wc, and Wd, respectively. The tracking confidence R1 is calculated from equation (6).

For example, the weights are set to Wa = 0.4, Wb = 0.3, Wc = 0.2, and Wd = 0.1. If all the reliabilities are sufficiently high and Ra = Rb = Rc = Rd = 1, then R1 = 1.0 according to equation (6). Further, in the case of Ra = 0.6, Rb = 0.5, Rc = 0.7, and Rd = 0.7, R1 = 0.6 according to equation (6).

式（７）中の、

は論理積演算を表す記号である。Ｒａ≧Ｔａ、Ｒｂ≧Ｔｂ，Ｒｃ≧Ｔｃ，Ｒｄ≧Ｔｄが全て成立する場合にＲ１＝１（高信頼度）であり、それ以外の場合にＲ１＝０（低信頼度）となる。 In the method by logical operation, the threshold values for Ra, Rb, Rc, and Rd are denoted as Ta, Tb, Tc, and Td, respectively. The tracking reliability R1 is calculated from equation (7) using, for example, an AND operation.

In equation (7),

Is a symbol representing an AND operation. When Ra ≧ Ta, Rb ≧ Tb, Rc ≧ Tc, and Rd ≧ Td are all satisfied, R1 = 1 (high reliability), and otherwise R1 = 0 (low reliability).

↓は否定論理和演算を表す記号である。Ｒａ＜Ｔａ、Ｒｂ＜Ｔｂ，Ｒｃ＜Ｔｃ，Ｒｄ＜Ｔｄの全てが成立しない場合にＲ１＝１（高信頼度）であり、それ以外の場合にＲ１＝０（低信頼度）となる。 Also, the tracking reliability R1 may be calculated from the equation (8) using an OR operation.

↓ is a symbol representing a negative disjunction operation. When all of Ra <Ta, Rb <Tb, Rc <Tc, and Rd <Td are not satisfied, R1 = 1 (high reliability), and otherwise R1 = 0 (low reliability).

  Next, an example of calculation of the tracking reliability using the feature amount of the feature point will be described. If the same feature point can be correctly tracked between images at different imaging times, the change in feature value of the feature point becomes small before and after tracking. Therefore, the tracking reliability calculation unit 405 calculates the tracking reliability according to the amount of change in the feature amount before and after the tracking. The amount of change of the feature amount can be obtained by calculating the feature amount using Equation (3) or Equation (4) before and after tracking, and calculating the difference between the two. A specific example will be described with reference to FIG.

  FIG. 11B shows an example of the correspondence between the variation of the feature amount and the reliability. The horizontal axis of FIG. 11B represents the variation of the feature amount, and the vertical axis represents the tracking reliability R2. In this example, two threshold values T1 and T2 (T1 <T2) are provided. When the amount of change of the feature amount is equal to or less than the threshold value T1, the same feature point can be correctly tracked, so the tracking reliability R2 is 1. If the amount of change in the feature amount is equal to or greater than the threshold T2, the possibility of mistracking of different feature points is high, and thus the tracking reliability R2 is zero. In a section where the feature amount variation is between the threshold values T1 and T2, the feature amount variation and R2 have a linear relationship, and the value of R2 decreases as the feature amount variation increases. Alternatively, the change amount of the feature amount and the reliability may be associated with a non-linear relationship.

  From the above description, the tracking reliabilities R1 and R2 can be calculated from each of the correlation value index information and the feature point information. As the final tracking reliability R, either one may be used, or both may be used in combination. With regard to the combination, as described in the equations (6) to (8), weighted addition or logical operation may be used.

  In S505 of FIG. 5, the feature point tracking unit 210 determines whether the process is completed up to the final frame, and ends the series of processes described above when the process is completed. On the other hand, if the process is not completed up to the final frame, the process returns to S501 to continue the process. In step S502, the feature point setting unit 402 controls the feature points to be tracked based on the tracking period calculated in step S304 in FIG. During the tracking period, the end point coordinates of the motion vector detected in S503 are set as feature points to be tracked in the next frame. As a result, since the end point coordinates of the motion vector represent the movement destination of the feature point, it becomes possible to track the feature point across a plurality of frames. When the tracking period ends, in S501, processing is performed to replace the newly extracted feature points.

  Next, three-dimensional coordinate estimation of the feature point in S310 of FIG. 3 will be described in detail with reference to FIGS. FIG. 12 is a schematic diagram showing a perspective projection model, in which three-dimensional coordinates P = (X, Y, Z) of feature points and two-dimensional coordinates p = (u, v) obtained by projecting them on a captured image Represents a relationship. Here, it is assumed that the virtual imaging plane is set in front of the lens at a position separated by the focal length f. OW represents the origin of the world coordinate system, and OC represents the camera lens center which is the origin of the camera coordinate system. The Z axis represents the camera optical axis, the point P is represented in the world coordinate system, and the point p is represented in the camera coordinate system.

In perspective projection conversion, the relationship between a point P in a three-dimensional coordinate system and a point p in a two-dimensional coordinate system is expressed by equation (9).

In the equation (9), K is called an internal parameter matrix and is a camera-specific parameter. K consists of a focal length f in pixel units and a principal point c = (cx, cy). Here, the principal point is set as the image center (0, 0), and it is treated as cx = cy = 0. Also, R and t are called external parameter matrices and represent the position and orientation of the camera with respect to the world coordinate system. R is a rotation matrix, and t is a translation matrix. The elements of R are denoted as R _{11 to} R _33, and the elements of t are denoted as t _x to t _z .

FIG. 13 is a schematic view showing how three-dimensional coordinates P are observed from two different camera positions and orientations. In FIG. 13, the three-dimensional coordinates to be estimated are P = (X, Y, Z), the position of the camera at frame 1 is O (origin), and the attitude is I (unit matrix). The coordinates are p0 = (u ₀ , v ₀ ). Further, it is assumed that the position of the camera at frame 2 is T, the posture is R, and the feature coordinates on the captured image at that time are p1 = (u ₁ , v ₁ ).

From equation (9), the relationship between the three-dimensional coordinates P and the two-dimensional coordinates p0 and p1 on the captured image is expressed as equations (10) and (11), respectively.

  In the equations (10) and (11), f is the focal length of the imaging optical system. The position and orientation estimation values obtained in S309 of FIG. 3 and the coordinate values of p0 and p1 of R and T may be known using the feature point information of the background area obtained in S304 of FIG. Therefore, by solving the simultaneous equations of Equations (10) and (11), the unknowns X, Y, Z can be determined, and three-dimensional coordinates can be obtained. Note that while the number of unknowns is three, that is, X, Y, Z, the number of simultaneous equations is 2n, where n is the number of frames for tracking feature points. Therefore, when the three-dimensional coordinates P are observed from only one frame, P can not be determined uniquely because the number of unknowns is larger than the number of equations. On the other hand, since two or more frames result in simultaneous equations under excess conditions, they can be solved using, for example, a known least squares method.

と表記し、図３のＳ３０３で得られた実際に観測された２次元座標を、

と表記する。 The estimation and correction of the shake correction remaining position and posture in S311 of FIG. 3 will be described with reference to FIG. FIG. 14 is a flowchart showing the processing of the position and orientation estimation unit 207.
In S1401, a process of converting the three-dimensional coordinates estimated in S310 into two-dimensional coordinates on a captured image is performed using Expression (9). In S1402, calculation processing of reprojection error is performed. The two-dimensional coordinates obtained in S1401 are

And the actually observed two-dimensional coordinates obtained in S303 of FIG.

It is written as

The reprojection error is expressed as E, which is expressed by equation (12).

In the equation (12), i represents a frame number, and j represents a feature point number. Assuming that the number of frames is n and the number of feature points is m, it changes in the range of i = 0 to n−1 and j = 0 to m−1. W _i is a matrix representing feature point weights in the i-th frame. The element at the i-th row and the j-th column of the matrix W _i is the weight w _ij of the j-th feature point in the i-th frame, which is obtained in S304 of FIG. The feature point having a larger weight (weighting coefficient) has a greater influence on the reprojection error, and the degree of contribution to the correction of the position and orientation estimation value in S1403 becomes larger.

  In S1403, a camera position and orientation estimation correction process is performed. The position and orientation estimation unit 207 corrects the position and orientation estimated value after the shake correction so that the reprojection error obtained from the equation (12) in S1402 is minimized. The re-projection error is an error between two-dimensional coordinates on an actually observed captured image and two-dimensional coordinates obtained by projecting three-dimensional coordinates on the captured image. Therefore, when the three-dimensional coordinates and the tracked two-dimensional feature coordinates are correct, the coordinate error is caused by an error in position and orientation estimation when the three-dimensional coordinates are projected onto the captured image. As the reprojection error becomes smaller, the position and orientation estimation value approaches the true value. As described above, since the correction of the position and orientation estimated value is a minimization problem of the reprojection error using the position and orientation estimated value as a variable, it can be performed by, for example, a known weighted least squares method.

  The internal configuration of the target position calculation unit 212 will be described with reference to FIG. FIG. 15 is a block diagram showing a detailed internal configuration of the target position calculation unit 212. As shown in FIG. The high pass filter 1501 obtains the output signal of the A / D converter 202 and removes the DC offset component included in the output of the first vibration sensor 201. The low pass filter 1502 obtains the output of the high pass filter 1501 and converts the angular velocity signal into a signal equivalent to an angle. The integral gain unit 1503 multiplies the output of the low pass filter 1502 by the integral gain. The gain multiplier 1504 multiplies the output of the integration processing unit 205 by a gain. The adder 1505 adds the output signals of the integral gain unit 1503 and the gain multiplier 1504, and outputs a signal of the result of the addition operation to the subtractor 213. That is, a process of adding the feedback amount from the gain multiplier 1504 to the output of the integral gain unit 1503 is performed.

  The target position calculation in S312 of FIG. 3 will be described in detail with reference to FIG. FIG. 16 is a flowchart of processing performed by the target position calculation unit 212. In S1601, the high pass filter 1501 removes the DC offset component from the shake angular velocity signal of the imaging device detected by the first vibration sensor 201 in S305 of FIG. The low pass filter 1502 performs low pass filter processing on the shake angular velocity signal from which the DC offset component has been removed in step S1601 in step S1602. In step S1603, the integral gain unit 1503 performs gain multiplication. Thus, the shake angular velocity signal is converted to a shake angle signal.

  In step S1604, the gain multiplier 1504 determines a gain to be multiplied by the shake correction remaining angle signal output from the integration processing unit 205. The method of determining the gain will be described with reference to FIG. The horizontal axis in FIG. 17 represents the tracking period, and the vertical axis represents the amount of gain. In this example, a threshold time T3 is provided, and the amount of gain is zero when the length of the tracking period is zero. Starting from this origin, in a section where the length of the tracking period is less than the threshold time T3, a linear relationship in which the amount of gain increases as the tracking period increases is shown. When the length of the tracking period is equal to or longer than the threshold time T3, the amount of gain is one. Therefore, when the tracking period is long, the ratio of synthesis of the shake correction remaining angle signal (position and posture information) to the shake detection signal is higher than when the tracking period is short.

  In S1605, the adder 1505 adds the shake angle signal calculated in S1603 and the shake correction remaining angle signal after gain multiplication obtained in S1604 to calculate the target position of the shake correction lens 103.

  In this embodiment, the shake correction remaining angle signal generated using the three-dimensional coordinates of the feature point is controlled according to the amount of parallax with respect to the shake angle signal of the imaging device. Based on the tracking information, the contribution of the estimation of the three-dimensional coordinates based on the tracking period calculated from the parallax amount between images is calculated, and the processing is performed to set the contribution higher as the tracking period becomes longer. Therefore, shake information and position and orientation information are synthesized. Image blur correction can be performed without interrupting processing even in a situation where the accuracy of three-dimensional coordinate estimation is insufficient. Therefore, a good captured image can be obtained regardless of the estimated state of three-dimensional coordinate estimation.

Second Embodiment
Next, in the second embodiment of the present invention, differences from the first embodiment will be mainly described.
FIG. 18 is a view showing an example of the arrangement of an image shake correction apparatus according to this embodiment. In the figure, the parts common to the components shown in the first embodiment are denoted by the reference numerals already used, and the detailed description thereof will be omitted. The image shake correction apparatus of the present embodiment includes a motion vector calculation unit 1801. The motion vector calculation unit 1801 obtains the output of the imaging signal processing unit 110 to calculate the motion vector of the image, and outputs the calculation result to the main subject separation unit 208 and the target position calculation unit 1802.

  The processing in the present embodiment will be described in detail with reference to the flowchart shown in FIG. The difference from FIG. 3 is the process shown in steps S1901 and S1902. After S302 (feature point tracking) performed by the motion vector calculation unit 1801, the process proceeds to S1901.

  In S1901, the motion vector computing unit 1801 outputs the tracking result of the feature point obtained in S302 and two results of updating the feature point for each frame and estimating the global vector between the frames. The global vector is a vector that represents uniform motion over the entire shooting screen. The arrows from S1901 to S1902 indicate that the process of outputting the calculated global vector to the target position calculation unit 1802 is performed.

  FIG. 20 is a block diagram showing an internal configuration of the motion vector computing unit 1801. Each part from the feature point extraction part 401 to the tracking reliability calculation part 405 is as having demonstrated in FIG. 4 in 1st Embodiment. In the present embodiment, the motion vector detection unit 404 is a first motion vector detection unit, and the added motion vector detection unit 2001 is a second motion vector detection unit. The second motion vector detection unit 2001 obtains signals from the imaging signal processing unit 110, the feature point extraction unit 401, and the image memory 403 to calculate a motion vector. That is, based on the feature points extracted by the feature point extracting unit 401, the second motion vector detecting unit 2001 performs motion vector detection on the image signals input from the imaging signal processing unit 110 and the image memory 403. Do. A global vector estimation unit 2002 estimates a global vector from the motion vector input from the second motion vector detection unit 2001.

  The process performed by the motion vector calculator 1801 will be described with reference to the flowchart in FIG. The processes of added S2101 and S2102 which are different from the processes of S501 to S505 shown in FIG. 5 will be described. The processes of S2101 and S2102 are executed as parallel processes with respect to the processes shown in steps S502 to S504.

  In S2101, the second motion vector detection unit 2001 detects a second motion vector using the feature points extracted in S501. The method of detecting the motion vector is the same as that of step S503, and therefore the description is omitted. In S2102, the global vector estimation unit 2002 estimates the global vector using the motion vector estimated in S2101. As a method of estimating a global vector, there is a method of performing histogram processing from a detected motion vector and estimating a mode value as a global vector. There is also a method of estimating the average value of detected motion vectors as a global vector. There is also a method of excluding outliers specified by the histogram processing and estimating the average value of the remaining motion vectors as a global vector. After the process of S2102, the process proceeds to the process of S505.

  In S1902 of FIG. 19, the target position calculation unit 1802 calculates the target position of the correction lens. Correction position control for driving the image shake correction lens 103 in the pitch direction or the yaw direction using the tracking period calculated in S304, the position / posture estimation value after shake correction in S311, and the global vector acquired from S1901 A signal is generated. After S1902, the process proceeds to S313.

  FIG. 22 is a block diagram showing an internal configuration of the target position calculation unit 1802. A difference from FIG. 15 is a gain multiplier 2201. The gain multiplier 2201 multiplies each of the output of the integration processing unit 205 and the output of the global vector estimation unit 2002 by a gain.

  The target position calculation process of the present embodiment will be described using the flowchart of FIG. The difference from the processing in the first embodiment is the processing content of S1604. In this embodiment, the gain multiplier 2201 determines a gain by which the shake correction remaining angle signal output from the integration processing unit 205 and the global vector output from the global vector estimation unit 2002 are multiplied. The method of determining the gain will be described with reference to FIG. The horizontal axis represents the tracking period based on the tracking information, and the vertical axis represents the amount of gain of the global vector. The threshold time is T4, and the amount of gain of the global vector when the length of the tracking period is zero is zero. In this example, the gain amount of the global vector is proportional to the length of the tracking period in an interval in which the length of the tracking period is from zero to less than T4. When the length of the tracking period is equal to or longer than the threshold time T4, the amount of gain of the global vector is one. The gain multiplication for the shake correction remaining angle signal has been described in S1604 of the first embodiment, and thus the detailed description thereof will be omitted. Gain multiplier 2201 outputs each output after gain multiplication to adder 1505.

  In this embodiment, feature points updated for each frame are used, and a global vector estimated for each frame is applied to the shake correction remaining angle signal. By correcting and reducing an error accumulated due to long-term tracking of feature points, it is possible to perform image shake correction better than in the first embodiment.

103 image blur correction lens 104 image blur correction lens drive unit 117 control unit 201, 203 vibration sensor 205 integrated processing unit 209 tracking period calculation unit 210 feature point tracking unit 212 target position calculation unit

Claims (11)

An image shake correction apparatus that corrects an image shake generated by a shake applied to an imaging device by a shake correction unit.
Tracking means for extracting and tracking feature points in a captured image using motion vector information obtained from an image signal of the captured image, and calculating coordinates of the feature points as tracking information;
First estimation means for estimating position and orientation information of the image pickup apparatus using shake information relating to a shake applied to the image pickup apparatus and the tracking information;
A second estimation unit configured to estimate three-dimensional coordinates of the feature point indicating a positional relationship including a depth of an object and the imaging device using the position and orientation information;
First calculating means for calculating a tracking period of the feature point from a parallax amount between a plurality of captured images and the feature point;
When the tracking period acquired from the first calculation means is long, the ratio of the position and orientation information to the vibration information is made higher than in the case where the tracking period is short, and the vibration information and the position and posture information are Second calculating means for calculating a target position of the shake correcting means by combining
An image blur correction apparatus, comprising: control means for controlling the shake correction means in accordance with the target position. The second calculation means calculates a contribution of the estimation of the three-dimensional coordinates based on the tracking period, and controls the contribution to be higher as the tracking period becomes longer to calculate the shake information and the position and orientation information. The image blur correction apparatus according to claim 1, wherein the image blur correction is performed. The second calculation means is
Multiplying means for multiplying the position and orientation information by a gain corresponding to the tracking period;
And adding means for adding the shake information to the position and orientation information multiplied by the gain,
The image blur correction device according to claim 1, wherein the value of the gain is larger when the tracking period is long than when the tracking period is short. The image blur correction device according to any one of claims 1 to 3, wherein the first calculation unit calculates the tracking period to be shorter as the amount of parallax is smaller. The tracking means comprises extraction means for extracting feature points for each frame of the captured image, storage means for storing a plurality of image signals at different imaging times, and detection means for detecting a motion vector, and the tracking information The image blur correction apparatus according to claim 1 or 2, wherein processing for extracting and tracking the same feature point in a plurality of the captured images is performed. The detection means
First detection means for detecting a first motion vector based on the feature points tracked by the tracking means;
The image shake correction apparatus according to claim 5, further comprising: second detection means for detecting a second motion vector from the plurality of image signals. The apparatus further comprises third estimation means for estimating a global vector from the second motion vector,
The second calculating means multiplies the global vector by a gain corresponding to the tracking period;
And adding means for adding the shake information to the global vector multiplied by the gain,
The image blur correction apparatus according to claim 6, wherein the value of the gain is larger when the tracking period is longer than when the tracking period is short. Reliability calculation means for calculating tracking reliability which is reliability of the tracking information;
The image blur correction apparatus according to any one of claims 1 to 7, further comprising: feature point setting means for setting a feature point to be tracked using the tracking reliability. The reliability calculation means calculates the tracking reliability using correlation values calculated from a plurality of image signals used when the motion vector is calculated, or the amount of change of the feature of the feature point before and after tracking. The image blur correction apparatus according to claim 8, wherein the image blur correction apparatus calculates the image blur.   An image pickup apparatus comprising the image blur correction device according to any one of claims 1 to 9. A control method executed by an image shake correction apparatus that corrects an image shake generated by a shake applied to an imaging device by a shake correction unit.
Extracting and tracking a feature point in the captured image using motion vector information obtained from an image signal of the captured image, and calculating coordinates of the feature point as tracking information;
Estimating position and orientation information of the image pickup apparatus using shake information related to shake applied to the image pickup apparatus and the tracking information;
Estimating three-dimensional coordinates of the feature point indicating a positional relationship including a depth of the subject and the imaging device using the position and orientation information;
Calculating a tracking period of the feature point from the amount of parallax between the plurality of captured images and the feature point;
When the tracking period is long, the shake correction means combines the shake information and the position / posture information by combining the ratio of the position / posture information to the shake information as compared to the case where the tracking period is short. Calculating the target position of
And controlling the shake correction means in accordance with the target position.

Images