JPH08262517A - Camera shake prediction device - Google Patents

Abstract

PURPOSE: To provide a camera shake prediction device provided with a high accuracy and a function making prediction feasible time as long as possible and constituted so that camera shake is prevented from occurring by predicting the irregular action of the camera shake in the midst of an exposure action by adding the characteristic movement of a photographing device and the characteristic movement specific to a photographer which are changed. CONSTITUTION: This camera shake prediction device is provided with a signal storage part 102 in which a camera shake signal obtained based on the irregular camera shake specific to the photographer is previously stored as history data and a prediction arithmetic part 103 predicting the camera shake after prescribed time by obtaining the local linear mapping of the time series of a fluctuation signal viewed to be irregular on the basis of the recorded history data by using a method applying an algorithm such as a Jacobi method or an orthonormal base method and constituted so that an image pickup system is controlled to be corrected by an actuator 105 based on the predicted result.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a device for preventing deterioration of image quality of an image pickup device due to camera shake, and more particularly to a camera shake prediction device for correcting movement of the image pickup device due to camera shake.

[0002]

2. Description of the Related Art Conventionally, there has been proposed a device such as a still camera or a video camera having a function of preventing deterioration of an image due to camera shake and improving the image. For example, depending on the signal detected using a mechanical vibration detector,
It is possible to drive the optical system of the photographic device so that it does not shake,
In an electronic image pickup device such as a CD, measures such as correcting an address for reading have been taken. Also, if you want to improve the shooting speed of the image, or if there is a large blur,
The method of prohibiting the photography was adopted. In addition, there is also a method of reproducing an image captured in a camera shake state into a fine image by a predetermined image processing.

[0003]

In the former method for correcting a camera shake signal in real time, a predetermined correction process is performed on an optical system and an electronic image pickup device in response to camera shake vibration. , A camera shake detection element,
Since the reaction time (response) of the mechanical drive device for the optical system correction and the operation time of the device that guides the optical path to the image sensor are required, the image capture device starts the shooting operation from the actual time of camera shake detection. It is necessary to predict the camera shake vibration and then correct the optical system. As a prediction method, there are a method of approximating a camera shake signal as a periodic fluctuation for correction, and a method of approximating as a characteristic “fluctuation” of a frequency by Fourier transform. However, since the actual camera shake signal is aperiodic fluctuations, the former approximation method cannot be said to be appropriate.

Further, in the method based on the Fourier transform, since the temporal information of "fluctuation" is lost, the prediction accuracy is not desirable although the related calculation is complicated.

In contrast to these correction methods, the change from the time when the camera shake is detected to the predicted time is considered as a linear function, and the coefficient of a linear temporal function is sequentially determined from the waveform of the camera shake vibration in the past to perform prediction. The method is as follows: Japanese Patent Application No. 4-11225, Japanese Patent Application No. 413772, Japanese Patent Application No. 411373, Japanese Patent Application No. 4-1343.
It is proposed in Japanese Patent No. 112780 and Japanese Patent Application No. 3-342441.

[0006] However, in these conventional techniques, the predictable time is short, and a small amplitude fluctuation component such as so-called "white noise" caused by the camera shake signal itself and camera shake vibration sensor sensitivity and measurement system sensitivity. The influence of the measurement error component due to quantization on the predicted value is not considered.

As described above, the conventional camera-shake preventing device still has many problems to be solved in practical use. An object of the present invention is to realize more effective camera shake prevention than conventional ones, and to sufficiently provide a response time of a camera shake vibration detector of a photographing device such as a camera, an operation time of an optical system correction device and an operation time of an imaging system. Is to consider.

An object of the present invention is to provide a camera shake prediction device having an irregular signal prediction function capable of predicting irregular camera shake vibration for a long time.

[0009]

SUMMARY OF THE INVENTION Therefore, the present invention is a camera shake prediction apparatus which is a camera shake prevention apparatus made in view of the above problems, and in order to solve the problems and achieve the above object, the following is provided. Are taking various measures.

The predicting device of the present invention is a camera shake predicting device for a photographing apparatus, wherein the detecting device detects a camera shake vibration and converts it into an electric signal, and a predetermined number of signals from the time series signals from the detecting device. By extracting each of them as the respective components of the coordinates of the multidimensional space to make the time-series signal correspond to a point in the multidimensional space, and in the multidimensional space generated by the trajectory generating means. And a prediction means for predicting a point to be arranged next in time series in the multidimensional space based on a time series of a plurality of points.

[0011]

The camera shake prediction device having the above means has the following functions.

The shake signal peculiar to the photographer is detected by the detecting means and converted into an electric signal. A predetermined number of electric signals extracted in time series are associated with points in the multidimensional space as respective components of coordinates in the multidimensional space by the trajectory generating means. Then, the predicting means predicts a point to be arranged next in time series in the multidimensional space. As a result, by applying the method of local linearization based on the detected camera shake signal, the position of the point after a predetermined time,
That is, the state of blurring is predicted.

[0013]

First, two main methods for predicting an irregular time series applied to the present invention will be described.

(1) "Jacobi method": The principle of local linearization prediction will be explained. That is, the signal detected by the camera shake signal detection unit is 1 in the Cartesian coordinate system x, y, z.
Regarding the variation (displacement) in the direction of one coordinate axis, a vector is formed by the following set of time series from the one-dimensional time series s _i , i = 1,2,3, ... N.

[0015]

[Equation 1]

FIG. 5 (a) shows the displacement of the camera shake signal in the x-axis direction in time series. D in the above equation is called “embedding dimension”, and x shown in FIG.
The trajectory drawn by _i is called the "attractor." These figures show how a three-dimensional attractor is constructed from a one-dimensional time series. The "mapping" from the i-th point on this attractor to the i + m-th point after a certain elapsed time m is formally expressed as x _{i + m} = f _m (X _i ). The prediction problem is to find an approximate solution of this function that represents the "mapping" from the current state to the state after a predetermined time. Here, 1
Next approximation will be described below. From past data recorded by the measurement, when the x _j and x _{j + m} is assumed to be known, state after m steps of x _j + .delta.x point near the x _j by using a series expansion as follows Represented by.

[0017]

[Equation 2]

Here, D _fm (x _j ) is called a “Jacobi matrix” around x _j . For example, if x has three components of i, j, and k in three dimensions, the mapping of each of these is performed. For each component, three functions f ₁ , f ₂ , f _{3 such as}
It is assumed that can be expressed by.

[0019]

(Equation 3)

At this time, the Jacobi matrix is expressed by the following determinant.

[0021]

[Equation 4]

The first-order partial differential matrix O (2) expressed by the above equation (4) represents a second-order or higher-order term with respect to δx. In the first-order approximation, it is assumed that the value of δx is sufficiently small, and the higher-order terms higher than the second-order are considered to be zero. Therefore, Jacobi
If the matrix can be obtained approximately, the equation (2) can be approximately represented.

Referring to FIG. 6, a reference point x _j in a given time series, a state of the point after a predetermined time, and a point near x _j and a state after the time are known. A method for approximating the Jacobi matrix when given as a thing will be described. The point you want to predict now is x _NOW . Next, the closest point x _j from x _NOW on the attractor is searched, and this is set as the reference point of the mapping. Then the radius ε around x _j
Set of points on attractor within _max {x _ki | ‖x _ki −x
Let _j ‖ <ε _max } be x _ki , i = 1,2, ..., N.
Then, the deviation vector of these points with respect to x _{j is} calculated as y _i = x
_ki −x _j . Also, let x _j and x _ki be τ = mΔt (
However, Δt is the time resolution of measurement, and τ = in discrete time series.
m) is the deviation vector after time evolution, z _i = x _{ki + m}
-X _{j + m} . The mapping between them is formally expressed as

[0024]

(Equation 5)

Here, A _j is a matrix expressed as follows.

[0026]

(Equation 6)

However, C and V are called "covariant matrices" and have elements of k rows and 1 columns represented by the following two equations. That is,

(Equation 7)

[0028]

(Equation 8)

Here, y ⁱ _l and z ⁱ _k are _l of y ⁱ and z ⁱ .
Represents the th and kth components.

Next, _assuming that the deviation vector between x _NOW and x _j is γ, the prediction vector after m points of x _NOW can be expressed by the following equation.

[0031]

[Equation 9]

According to Definition 1 above, the first element of x _{NOW + m} is the predicted value of the scalar variable m steps after the present.

As described above, the above-mentioned predicted value can be obtained by applying the so-called "Jacobi method" algorithm for obtaining the Jacobi matrix, and it can also be used for camera shake correction control based on this value.

Next, another prediction method applied to the present invention will be described. (2) “Orthonormal basis method”: The above-mentioned calculation method includes calculation of the inverse matrix V ⁻¹ . This calculation is complex and can often result in errors even if the matrix V is a regular matrix. Therefore, next, with reference to FIGS. 7A to 7D, a description will be given of this method, which simplifies the above-described method and gives a predicted value without including calculation of an inverse matrix.

First, similarly to the above-mentioned formula (1), the attractor is constructed with the embedding dimension d. Here, points approaching x _NOW are x _p1 , x _p2 , x _p3 , ..., x
Let _pk and k = d. (Reference: FIG. 7 (a)) A method for actually calculating the approach point on the attractor at high speed is taught in detail in Japanese Patent Application No. 5-212976, which is a prior art.

Next, the center of gravity R of these points is defined. (See FIG. 7 (b)) the center of gravity and x _pi i = 1,2,3, the deviation vector ·· d ξ _i,
The deviation vector of x _NOW is ξ. (Reference: FIG. 7 (c)) ξ _i i = 1,2,3, ... (Reference: Figure 7 (d))

[Equation 10]

Where ‖‖ is the norm of the vector,
<,> Is the dot product.

Next, the inner product of the orthogonal system and ξ is defined by the following equation.

[0039]

[Equation 11]

According to this definition, ξ is expressed by the following equation using the orthogonal basis V _m .

[0041]

(Equation 12)

Next, the states of the center of gravity and the deviation vector after m steps are defined as follows.

[0043]

(Equation 13)

Here, only x _{NOW + m} is an unknown point,
Obtaining this is a prediction problem. Here, the orthonormal system is obtained again and expressed as the following equation.

[0045]

[Equation 14]

Here, assuming that the elapsed time of step m is short and the inner product of ξ and ξ _pi does not change, the following equation is obtained.

[0047]

(Equation 15)

The vector predicted after m steps is expressed by the following equation.

[0049]

[Equation 16]

As shown in FIG. 8, this method uses a new orthogonal system V _i and a previous coefficient set L _i to represent what the orthonormal basis is by the mapping after m steps. And gives a predicted value for an unknown point.

As described above, the above-mentioned predicted value can be obtained by applying the algorithm based on the "orthogonal basis method", and this value can also be used for camera shake correction control.

Next, as another method, a method of correcting the predicted value related to the above-mentioned mapping by "smoothing" in consideration of the minute "fluctuation" of the measured value will be described.

As a method of canceling a minute fluctuation in the measured value, there are a method of taking a moving average numerically, a method of using a filter for electrically band limiting, and the like. Here, a method that can be added to the above-mentioned "local linearization method" will be described.

As a method of averaging the motion vectors from x _i to x _{i + m} on the attractor, x ′ _{i + m} as given in the following equation is used as a substitute value for x _{i + m} .

[0055]

[Equation 17]

Alternatively, the motion from x _i to x _{i + m} is tracked for each component, and a value after m points on the locus where the fluctuation of the high frequency component is blocked by using the low pass filter, x ′ _A method of using _{i + m} as a value instead of x _{i + m} can be considered. Regarding prediction performed by local linearization by the Jacobi method equipped with such an averaging operation, FIG.
The description will be made with reference to (e).

First, as shown in FIG. 9 (a), a point x _{o in} the vicinity of the point x _i to be predicted is searched for, and x x is calculated from that x _o.
Correction value x'that tracks the trajectory up to _{o + m} and averages
_{Find o + m} .

As shown in FIG. 9B, a point X _o (k) k = 1,2, ..., N near ε is searched with the above x _o as a reference point. Further, as shown in FIG. 9 (c), with respect to points in the vicinity of x _o as well, similar to the above, the trajectory is traced by m steps and an operation for averaging is performed.

Next, as shown in FIG. 9 (d), by using x _o and x _o (k) constructed in the above procedure, the above-mentioned FIG.
The Jacobi matrix A is calculated by the method described in 1. Then, as shown in FIG. 9 (e), the predicted value x _{i + m} can be obtained by operating the Jacobi matrix A on the deviation vector of x _i and x _o .

Embodiments according to the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 is a functional block diagram showing the structure of a camera shake prediction apparatus according to the first embodiment of the present invention. In the image stabilization apparatus of the image capturing apparatus, the apparatus of the present invention as the first embodiment includes an image stabilization signal detection unit 101 including a gyro-type angular velocity sensor, an image stabilization signal recording unit 102 that stores a detected signal, and a recording unit. Prediction calculation unit 103 that predicts the state of the camera shake signal after a specified time from the time series of the camera shake signal, and mechanically corrects the optical axis of the optical system of the image capturing device including a motor or a voice coil based on the state prediction The actuator 105 includes a correction signal calculation unit 104 that generates an operation signal.

It is conceivable to statistically process the "fluctuation component" of the small signal relating to the irregular camera shake signal and the error component relating to the measurement to achieve the smoothing. In the configuration of the first embodiment shown by 1,
Smoothing may be performed from the signal recorded in the signal recording unit 102 by numerical calculation such as moving average.

According to the first embodiment, the signal recording section 10
2 records the state up to the present of the time-series signal of camera shake vibration detected by the camera shake signal detection unit 101. The prediction calculation unit 103 is set in advance with the prediction time, the dimension of the linearization matrix when performing local linearization, and the number of past data samples required for linearization. The prediction calculation unit 103 selects a past history close to the current state from the recorded signal and performs "local linearization". (However, the detailed method of the actual "local linearization" will be described later.) The blurring state after a predetermined time from the present is predicted by the local linearization method. The correction signal calculation unit 104 converts the predicted value into an electric signal and generates an operation signal to the actuator 105 that mechanically corrects the optical axis of the imaging optical system 106 of the image capturing apparatus. With the above operation, this actuator adjusts the optical axis so as to correct the movement due to the camera shake, and performs the camera shake correction control.

The structure shown in FIG. 2 is a modification of the first embodiment described above. The feature of this configuration is that the angular velocity sensor 101 and the signal recording unit 102 are provided in order to smooth the "fluctuation component" of the signal and the error component related to the measurement.
The smoothing processing unit 107 (filter means) for low-pass filtering of the analog signal is disposed between and.

As shown in the figure, the signal detected by the camera shake signal detection unit 101 passes through the smoothing processing unit (for example, high frequency filter) 107 to remove high frequency components, and the signal is recorded in the signal recording unit 102. Sent to.

According to the modification, the error signal processed by the smoothing processing unit 107 is read by the prediction calculation unit 103, and the parameter of the prediction means is changed by a method using a predetermined algorithm described later to minimize the error. "Learn" to become. As a result, the characteristics of the shake signal peculiar to the photographer are learned and temporarily stored in the signal recording unit.

Next, the configuration shown in FIG.
The present invention relates to other modified examples of the embodiment. The feature of this configuration is that the error calculation unit 108 is provided with the above-described angular velocity sensor 101.
And the prediction calculation unit 103. The numerical value predicted by the prediction calculation unit 103 and the camera shake signal actually measured are sent to the error calculation unit 108, and the error between the predicted value and the actual measurement value is calculated. Then, the calculated error is fed back to the prediction calculation unit 103.

According to this modification, since the error between the predicted value and the actual measured value is obtained by calculation, the control and prediction correction processing is performed based on the error value. As a result, accuracy of correction control and the like can be improved.

(Second Embodiment) Next, FIG. 4 is a functional block diagram showing the configuration of the second embodiment of the hand movement prediction apparatus of the present invention.

When the image pickup device is composed of the electronic image pickup device 201 which is not the optical image pickup device as in the first embodiment,
The camera-shake prediction device is characterized by including a pixel correction signal calculation unit 205 that generates a signal for correcting an address of a pixel of the electronic image sensor 201, instead of the actuator 105 that is a component of the first embodiment. In addition, as an element constituting the camera shake signal detection unit, the electronic image pickup device 2
The motion vector detection unit 202 is provided as a sensor that detects a shake based on the signal from 01. In the apparatus having such a configuration, when the prediction calculation unit 204 predicts a blurring state after a predetermined time from the current state, a local “linear image projection function” is generated from the history of past camera shake vibration data. Make that prediction.

Further, in this prediction calculation unit 204,
It is characterized by statistically processing the "fluctuation components" of small signals related to irregular camera shake signals and the error components related to measurement, smoothing them, and generating local linear projection functions for prediction. To do.

According to the second embodiment, the pixel correction signal calculation unit 205 generates a signal for correcting the address of the pixel of the electronic image pickup device 201 that captures an image. The motion vector detection unit 202 also functions as a camera shake detection sensor based on the signal from the electronic image sensor 201. Then, the shake calculation data (that is, history) from the past accumulated in the irregular signal recording unit 203 unique to the photographer by the prediction calculation unit 204.
Based on, the local linear projection function is generated, and the "fluctuation component" and measurement error component of the signal are statistically processed for smoothing and prediction.

(Operation 1 of Embodiment) Next, the operation of the apparatus of this embodiment will be described with reference to FIGS. FIG. 10 is a flow chart showing the processing procedure performed by the prediction calculation units 103 and 204 of the apparatus shown in FIGS. 1 to 4 based on the above-mentioned “Jacobi method”.

First, when predicting a camera shake, a prediction time m is calculated in advance in consideration of the response speed of the camera shake detection section and the operation time of the image pickup system, and this value is set in the prediction calculation section (S1). ).

The dimension for linearization is set (S2).

The number n of past history measurement points required for linearization is set (S3).

Measurement of camera shake with a set number of points is shown in FIGS.
The hand-shake signal detecting unit including the elements 101 and 102 of FIG. The prediction is made on the newest recorded data s (n). A rearrangement vector is constructed for this s (n) by the method of equation (1) (S5).

Next, the closest point of x _{n is obtained in} the reorganization state space, and a specific method for obtaining the close point is disclosed in Japanese Patent Application No. 5-212976 which discloses the prior art. There is a method in which a sort sequence is generated from the scalar sequence s (i), i = 1, 2, ..., N and a proximity point is searched for each element of the vector.

FIG. 11 shows a subroutine showing the detailed procedure of step S6 in the flowchart of FIG.

First, in S61, a sort string is calculated from time series data.

In S62, the sort string is referenced to 1 of x _n .
The m-dimensional norm is calculated by sequentially calculating the difference from the points having similar sort orders from the th element s (n).

In S63, it is judged whether or not the norm is the minimum value, and when it is the minimum value, the process is terminated.

Next, a set of adjacent points of the closest point x _o is obtained (S7). FIG. 12 shows a subroutine showing the detailed procedure of step S7 in the flowchart of FIG.

In S71, ε is set as the reference for the approach point.

In and after S72, the approach point of x _o is calculated with reference to the sort sequence calculated in S61.

At S73, the difference Ds of the first element is calculated. Similar to S62 described above, the search is performed in the order of close sorting order, and when the difference D _{s at} this time reaches the reference ε, the search ends (S74).

In S75 after this procedure, d
Compute the norm of the dimension and if the value does not exceed the reference value ε, add the point to the list of points in the neighborhood of x _o .

In S8 of FIG. 10, the Jacobi matrix is calculated using the neighboring points searched in S7. This calculation is performed according to the above equations (5) to (9).

FIG. 13 shows a subroutine showing the detailed procedure of step S8 in the flowchart of FIG.

First, in S81, the list of adjacent points searched in S7 is referred to.

In S82, the deviation vector y _i = x _ki −x _j from x _o is calculated for the near point.

In S83, the history of the past signals recorded in the signal recording units 102 and 202 is referred to, and the deviation vector z _i = x after time evolution of x _{o + m} and the adjacent point after m steps.
Calculate _{ki + m} −x _{j + m} .

In S84, the covariant matrices C and V are calculated according to the equation (7).

In S85, the Jacobi matrix A = C · V _inv
Is calculated. Here, V _inv means an inverse matrix.

In S9, the Jacobi matrix obtained in S8 is applied to x _n -x _o to obtain the prediction vector x _{n + m} according to the equation (9).

The first element of the vector at this time is the predicted value of the current scalar observation value s _n after m steps.

(Operation 2 of the Embodiment) The operation of the present embodiment will be described with reference to FIGS.

FIG. 14 is a flowchart showing the processing procedure performed by the prediction calculation units 103 and 204 of the apparatus shown in FIGS. 1 to 4 based on the above-mentioned “orthonormal basis method”.

First, when predicting the camera shake, the predicted time m is calculated in advance in consideration of the response speed of the camera shake detection section and the operation time of the image pickup system, and this value is set in the prediction calculation section (S1). .

Further, the dimension for performing linearization is set (S2).

The number of measurement points n of the history from the past, which is necessary for performing the linearization process, is set (S3).

Measurement of camera shake with the set number of measurement points
1 to 4 are detected by the camera shake signal detection units 101 and 201,
The signal is recorded in the signal recording units 102 and 202 in FIG. The prediction is given by the expression (() for the newest recorded data s (n).
A reorganization vector is constructed by the method of 1) (S5).

In addition, the search for the closest contact point of s (n) is performed as shown in FIG.
The procedure is the same as the step S6 of 0.

In S6, the d points closest to x _{n in the} d-dimensional space are searched.

FIG. 15 shows a subroutine showing the detailed procedure of step S6 in the flowchart of FIG.
In this example, in order to speed up the search process, S6 in FIG.
The sorting sequence is used similarly to.

First, in S61, the number q of search point candidates is set. The value of q is selected to be larger than the dimension d of the orthonormal system.

At S62, the sorting sequence is referred to.

Step S63 is a judgment procedure for ending the search for the approach point candidates.

Further, in S64 and S65, the d-dimensional norm is calculated by referring to the sort order of the first element s (n) of x _n and calculating the difference between the elements in the order of decreasing sort order.

In S66, a list is created in which the indices of the candidates of the x _n neighboring points are associated with the norms. And i = 1,2 ,,, d | number of search points is completed a search at in excess of q, carried out the sorting of the norm (S67), this to create a list in the d number of points in the ascending order of the norm x _pl To do.

Next, in step S7 of FIG. 14, the predicted value of m steps after x _n is calculated using the orthonormal system.

FIG. 16 shows a subroutine showing the detailed procedure of step S7 in the flow chart of FIG.

In S71, the barycenter R is calculated by referring to the d neighboring points of x _n created in S6.

In S72, the deviation vector ξ _i between the neighboring points and the center of gravity is calculated.

At S73, a d-dimensional orthonormal system V is constructed from ξ _i based on the equation (10).

In S74, the inner product L ₁ of ξ and V _i is calculated.

Further, in S75, the signal recording units 102, 20
The weight R of the point x _pi after m steps of points in the vicinity of x _n is calculated with reference to the past signal history recorded in 2.

In S76, the deviation vector ξ ^* between the center of gravity R ^* and x ^* pi is calculated.

[0119] In S77, calculates the xi] _i ^* orthonormal V _i ^* of in accordance with equation (14).

In S78, V _{i is} calculated according to the above equation (15).
The state ξ ^* after m steps of ξ is predicted from ^* and L _l .

In S79, x _{n is} calculated according to the above equation (16).
The point x _{n + m} after m steps of is given.

The first element of this state vector is the predicted value of the current scalar observation value s _n after m steps.

(Operation 3 of the Embodiment) The operation of the present embodiment will be described with reference to FIGS.

FIG. 17 shows the procedure of FIG. 10 showing the processing procedure performed by the predictive calculation units 103 and 204 of the apparatus shown in FIGS. 1 to 4, in which a prediction error due to “fluctuation” of a small small signal is reduced. Therefore, an embodiment having a smoothing procedure processing operation of history data by moving average is shown.

However, the procedure of S1 to S4 is as shown in FIG.
This is the same as the description content of 0.

At S5, the measurement data s (i) is smoothed by a moving average to obtain sa (i).

At S6, the latest measurement data s (n) is vectorized according to the equation (1).

At S7, the smoothed data sa (
In the state space reorganized from i), find the closest point of xn.

Note that FIG. 18 shows a subroutine representing the detailed procedure of step S7 in the flowchart of FIG.

At S71, the smoothed time series data sa
Calculate the sorted sequence from (i) and x _n .

In step S72, the sort sequence is referenced and x _n is 1
The m-dimensional norm is calculated by sequentially calculating the difference from the points having similar sort orders from the th element s (n).

In S73, it is determined whether or not the norm is the minimum value, and when it is determined to be the minimum value, the process is terminated.

Next, a set of neighboring points of the point x _o closest to x _n is obtained by the attractor composed of the smoothed data (S
8). FIG. 19 shows this step S in the flowchart of FIG.
The subroutines representing the detailed procedure of 8 are shown.

First, in S81, the approach point reference ε is set.

In S82 and subsequent steps, the proximity point of x _o is calculated with reference to the sort sequence calculated in S71.

In S83, the difference Ds of the first element is calculated.

Further, similar to S72 described above, the search is performed in the order of close sorting order, and the difference Ds at this time is the reference ε.
Is reached, a series of search processing is terminated (S84
).

At S85, the d-dimensional norm is calculated, and if the value does not exceed the reference ε, the point is added to the list of points near x _o .

Next, in S9 of FIG. 17, the Jacobi matrix is calculated using the neighboring points searched in step S8. This calculation is performed according to the equations (5) to (9).

FIG. 20 shows a subroutine showing the detailed procedure of step S9 in the flowchart of FIG.

First, in S91, the list of neighboring points searched in S8 is referred to.

At S92, the deviation vector y ⁱ = x _ki −x _j from x _o is calculated for the near point.

At S93, the smoothed data string sa (i) is referred to.

At S94, the deviation vector z _i = x _{ki + m} generated between x _{o + m} and the neighboring point after the m step time development.
Calculate -x _{j + m} .

Further, in S95, the covariant matrices C and V are calculated according to the defined equation (7).

In S96, the Jacobi matrix A = C · V _inv
Is calculated. However, V _inv represents an inverse matrix.

Next, in S10, the Jacobi matrix obtained in S9 is applied to x _n -x _o to obtain the prediction vector x _{n + m} according to the equation (9).

At this time, the first element of this vector is the predicted value of the current scalar observation value s (n) after m step time.

(Operation 4 of the Embodiment) The operation of the apparatus of this embodiment will be described with reference to FIGS.

In FIG. 21, the prediction error due to the “fluctuation” of a small small signal is reduced to the procedure of FIG. 10 showing the processing procedure performed by the prediction calculation units 103 and 204 of the apparatus shown in FIGS. Therefore, an example is shown in which prediction is performed by local linearization in which “time evolution” is corrected. Note that S1
The procedure from S7 to S7 is the same as each step in FIG.

In S8, the time evolutions of x _o and its neighboring points are calculated by the equation (17) or the Jacobi matrix corrected by the method shown in FIG. 9 according to the equations (5) to (9).

FIG. 22 shows a subroutine showing the detailed procedure of step S8 in the flowchart of FIG.

First, in S81, the list of neighboring points searched in S7 is referred to.

In S82, the deviation vector y _i = x _ki −x _j from x _o is calculated for the neighboring points.

[0155] In S83, the time evolution of the time elapsed m steps of x _o with reference to the history of past signal recorded on the signal recording unit 102, 202 development and x _o and the neighboring point x _o (i) Correction is performed using equation (17), and _x'o ,
Get _x'o (i).

At S84, the deviation vector of the correction point z i =
to calculate the _{x 'o -x' o (i} ).

In S85, the covariant matrices C and V are calculated according to the defined equation (7).

In S86, the Jacobi matrix A = C · V _inv is calculated. However, this V _inv represents an inverse matrix.

In S9, the corrected Jaco obtained in S8 described above is used.
The bi matrix is operated on x _n −x _o to obtain the prediction vector x _{n + m} according to the equation (9). At this time, the first element of this vector is the predicted value of the current scalar observation value s _n after m steps.

(Operation 5 of the Embodiment) FIG. 23 is a flowchart showing the operation of the apparatus as a modification according to the prediction methods according to the first and second embodiments of the present invention.

In S6, the prediction after the elapsed time m steps is obtained.

At S7, the state after the actual elapsed time m steps is read.

At S8, the error between the actually measured value and the predicted value is calculated. Here, the actually measured value of this time is stored, the process returns to S2 in which the linearization dimension d is changed in S10, and the predicted value is calculated again.

In this way, the linearization dimension d is changed and the difference between the successive prediction value and the actual measurement value is calculated. As a result, it is possible to store the dimension of linearization of the characteristics of the user's camera shake.

Further, in S10, the value of the sample number n of the past data used for prediction may be changed instead of changing the dimension d. By doing so, it is possible to obtain the prediction value that minimizes the error while changing n, and it is possible to obtain the optimum number of samples required for the prediction.

(Other Modifications) Data relating to camera shake vibration can be saved by statistically collecting average data of a large number of people and storing it in ROM.
As a result, the probability of statistical improvement is improved, and the predictive accuracy of the attractor is also increased.

As for the embodiment of the device of the present invention, various modifications can be made within the scope not departing from the gist of the present invention, other than the examples listed above.

In each of the embodiments illustrated above,
The following inventions are included.

(1) In the camera-shake predicting device of the photographing apparatus, a detecting means for detecting camera-shake vibration and converting it into an electric signal, and a predetermined number of signals from the time-series signal from the detecting means are extracted, and each of them is extracted. A trajectory generation unit that associates a time-series signal with a point in the multidimensional space by using each coordinate component of the multidimensional space, and a time series of a plurality of points in the multidimensional space generated by the trajectory generation unit. And a prediction means for predicting a point to be arranged next in time series in a multidimensional space based on the above.

Action 1: The detecting means operates as a detecting sensor and converts the detected signal into a signal for detecting a vector of camera shake. The locus generating means time-sequentially spatially coordinates the detected signal to generate a continuous locus, and the predicting means performs local linearization based on the locus to predict a state after a predetermined time from the present.

Effect 1: It is possible to provide a camera shake prediction device capable of predicting camera shake vibration for a long time, and implement a device for substantially preventing camera shake by performing correction control for camera shake vibration of a photographing device based on the prediction information. it can.

(2) It further comprises storage means for storing the signal from the detection means, and a plurality of points in the multidimensional space, and a plurality of points in the multidimensional space, which are before the plurality of points in time series. The point to be arranged next in time series in the multidimensional space is predicted based on the point in the multidimensional space generated from the signal stored in the storage means (1). The described camera shake prediction device.

Action 2: The storage unit records the time-series state of the camera shake vibration detected by the detection unit up to the present time as history data as an irregular signal unique to the photographer. Based on the history data, the prediction means performs prediction by locally generating a linear image projection function and statistically processing the fluctuation component and measurement error component of the signal. That is, the prediction means performs "linearization" and "vectorization" processing based on the generated camera shake signal to obtain a map as an attractor, thereby performing "local linear prediction" to predict the camera shake signal.

Effect 2: By accumulating the detection data, it is possible to provide a camera shake prediction device with improved prediction reliability.

(3) It further comprises storage means for storing pre-measured time-series camera shake signals, wherein the predicting means stores a plurality of points in the multidimensional space generated by the trajectory generating means, and the storage means. Characterized by predicting a point to be arranged next in time series in the multidimensional space based on a plurality of points in the multidimensional space generated from the output from the means.
The camera shake prediction device described in (1).

Action 3: The storage means accumulates and records irregular signals unique to the photographer as history data. The predicting means predicts the point to be arranged next based on the time-series points in the multidimensional space by using the detected signal and the accumulated history data.

Effect 3: By combining the storage means and the prediction means, it is possible to provide the camera shake prediction apparatus for learning the characteristics of the individual camera shake signal of the photographer.

(4) A smoothing means for smoothing the output from the detecting means is further provided, and the trajectory generating means generates points in the multidimensional space based on the output from the smoothing means. The camera shake prediction device according to any one of (1) to (3).

Action 4: The high frequency component is removed by using the filter for smoothing the detection data by the moving average of the camera shake signal.

Effect 4: Prediction error due to slight signal fluctuation is reduced.

(5) A smoothing means for smoothing the output from the storage means is further provided, and the predicting means includes a plurality of points in the multidimensional space generated by the trajectory generating means,
The camera shake according to (1), characterized in that a point to be arranged next in time series in the multidimensional space is predicted based on the point in the multidimensional space generated from the output from the smoothing means. Prediction device.

Action 5: High frequency components are removed by using a filter for smoothing history data by moving average of camera shake signals.

Effect 5: A prediction error due to a slight signal fluctuation is reduced.

(6) Comparing means for comparing the output from the predicting means with the output from the detecting means for the output, and the dimension control for changing the predetermined number so that the output from the comparing means is minimized. The camera shake prediction device according to any one of (1) to (5), further comprising:

Action 6: The dimension control means controls the difference output by the comparison means to be a minimum by changing a predetermined parameter. That is, the dimension of linearization is changed to match the sequentially predicted value and the actually measured value.

Effect 6: By controlling so that the predicted value matches the actually measured value, it is possible to substantially suppress the shake caused by the shake vibration.

(7) Comparing means for comparing the output from the predicting means and the output from the detecting means with respect to the output, and the output from the trajectory generating means so that the output from the comparing means is minimized. The camera shake prediction device according to any one of (1) to (6), further comprising: a unit for controlling the number of a plurality of points in the multidimensional space input to the prediction unit.

Action 7: The means for controlling the number of points adjusts a plurality of points in the multidimensional space to the number of points such that the difference output from the comparing means is minimized, and the predicted value and the measured value are obtained. To match.

Effect 7: By controlling so that the predicted value matches the actually measured value, it is possible to substantially suppress camera shake due to camera shake vibration.

(8) The camera shake prediction device according to any one of (1) to (7), wherein the prediction means makes a prediction based on an algorithm of the Jacobi method.

Action 8: The state of blurring after a predetermined time can be derived by the method of local linearization regarding the time series of camera shake vibration using the Jacobi method algorithm.

Effect 8: The long-term prediction of camera shake vibration is
The application of this well-known Jacobi algorithm facilitates this.

(9) The prediction means makes a prediction based on an orthogonal normalization algorithm (1)
The camera shake prediction device according to any one of (6) to (6).

Action 9: The state of blurring after a predetermined time can be derived by the method of local linearization regarding the time series of camera shake vibration using the orthogonal normalization algorithm.

Effect 9: The long-term prediction of camera shake vibration is
The application of this well-known orthogonal normalization algorithm facilitates this.

[0196]

As described above, the present invention as a whole has a function of accumulating data concerning camera shake vibration peculiar to the user of the image capturing apparatus, and further has a predetermined algorithm for prediction which has been conventionally established and irregular vibration. By applying it for the prediction, it is possible to easily predict the camera shake vibration for a long period of time, and it is possible to provide a highly reliable camera shake prediction device with further improved prediction accuracy.

If the predicted value is converted into an electric signal and processed based on an operation signal to an actuator or the like for mechanically correcting the optical axis of the image pickup apparatus, the actuator moves the optical axis due to camera shake. It is also possible to perform camera shake correction by adjusting so as to perform minute correction, and thus it is possible to realize a more reliable camera shake prevention mechanism than ever before.

[Brief description of drawings]

FIG. 1 is a functional block diagram showing a configuration of a camera shake prediction device according to a first embodiment.

FIG. 2 is a functional block diagram showing a configuration according to a modification of the first embodiment.

FIG. 3 is a functional block diagram showing a configuration according to another modification of the first embodiment.

FIG. 4 is a functional block diagram showing a configuration of a camera shake prediction device according to a second embodiment.

FIG. 5A is a graph showing displacement in the x-axis direction of a camera shake signal in time series, and FIG. 5B is an attractor that draws a trajectory of xi.

FIG. 6 Jaco showing a method for approximating a Jacobi matrix
bi law conceptual diagram.

7 (a) to (d) are conceptual diagrams illustrating a method of simplifying and predicting the Jacobi method.

FIG. 8 is an explanatory diagram showing changes in an orthonormal basis due to mapping.

9A to 9E are conceptual diagrams illustrating a prediction method by local linearization by the Jacobi method.

FIG. 10 is a flowchart of a main routine relating to operation procedure 1 of the embodiment.

FIG. 11 is a flowchart of a subroutine of S6 of operation procedure 1.

FIG. 12 is a flowchart of a subroutine of S7 of the operation procedure 1.

FIG. 13 is a flowchart of a subroutine of S8 of operation procedure 1.

FIG. 14 is a flowchart of a main routine relating to operation procedure 2 of the embodiment.

FIG. 15 is a flowchart of the subroutine of S6 of operation procedure 2.

FIG. 16 is a flowchart of a subroutine of S7 of operation procedure 2.

FIG. 17 is a flowchart of a main routine relating to operation procedure 3 of the embodiment.

FIG. 18 is a flowchart of the subroutine of S7 of operation procedure 3;

FIG. 19 is a flowchart of a subroutine of S8 of operation procedure 3.

FIG. 20 is a flowchart of a subroutine of S9 of operation procedure 3.

FIG. 21 is a flowchart of a main routine relating to operation procedure 4 of the embodiment.

FIG. 22 is a flowchart of the subroutine of S8 of the operation procedure.

FIG. 23 is a flowchart of a main routine relating to operation procedure 5 of the embodiment.

[Explanation of symbols]

101 ... Angular velocity sensor, 102 ... Signal recording unit, 103 ...
Prediction calculation unit, 104 ... Correction signal calculation unit, 105 ... Actuator, 106 ... Imaging optical system, 107 ... Smoothing processing unit, 108 ... Error calculation unit, 201 ... Electronic imaging device, 20
2 ... Motion vector detection unit, 203 ... Signal recording unit, 204
Prediction calculation unit 205 Pixel correction signal calculation unit 206
Electronic image sensor, S1 to S10 ... Main processing routine, S
61 to S68 ... S6 subroutine, S71 to S79 ...
S7 subroutine, S81 to S87 ... S8 subroutine, S91 to S96 ... S9 subroutine.

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] July 26, 1995

[Procedure Amendment 1]

[Document name to be amended] Statement

[Name of item to be corrected] 0022

[Correction method] Change

[Correction content]

O (2) represents a second-order or higher-order term with respect to δx. In the first-order approximation, it is assumed that the value of δx is sufficiently small, and the higher-order terms higher than the second-order are considered to be zero. Therefore,
If the Jacobi matrix can be obtained approximately, then equation (2)
Can be approximately represented.

[Procedure Amendment 2]

[Document name to be amended] Statement

[Name of item to be corrected] 0032

[Correction method] Change

[Correction content]

According to the definition of Equation 1 above, 1 of x _{NOW + m}
The th element is the predicted value of the scalar variable m steps after the present.

[Procedure 3]

[Document name to be amended] Statement

[Correction target item name] 0043

[Correction method] Change

[Correction content]

[0043]

(Equation 13)

[Procedure amendment 4]

[Document name to be amended] Statement

[Correction target item name] 0053

[Correction method] Change

[Correction content]

As a method of canceling a minute fluctuation of the measured value , there are a method of taking a moving average numerically, a method of using a filter for electrically band limiting, and the like. Here, a method that can be added to the above-mentioned "local linearization method" will be described.

[Procedure Amendment 5]

[Document name to be amended] Statement

[Correction target item name] 0063

[Correction method] Change

[Correction content]

According to the first embodiment, the signal recording section 10
2 records the state up to the present of the time-series signal of camera shake vibration detected by the camera shake signal detection unit 101. For the prediction calculation unit 103, the time of prediction, the dimension of the linearization matrix when performing local linearization, and the number of past data samples required for linearization are set in advance. The prediction calculation unit 103 selects a past history close to the current state from the recorded signal and performs "local linearization". (However, the detailed method of the actual "local linearization" will be described later.) The blurring state after a predetermined time from the present is predicted by the local linearization method. The correction signal calculation unit 104 converts the predicted value into an electric signal and generates an operation signal to the actuator 105 that mechanically corrects the optical axis of the imaging optical system 106 of the image capturing apparatus. With the above operation, this actuator adjusts the optical axis so as to correct the movement due to the camera shake, and performs the camera shake correction control.

[Procedure correction 6]

[Document name to be amended] Statement

[Correction target item name] 0087

[Correction method] Change

[Correction content]

In steps S75 to 77 after this procedure, the d-dimensional norm is calculated, and if the value does not exceed the reference value ε, the point is added to the list of points near x ₀ .

[Procedure Amendment 7]

[Document name to be amended] Statement

[Correction target item name] 0110

[Correction method] Change

[Correction content]

In S66, a list is created in which the indices of the candidates of the x _n neighboring points are associated with the norms. When the number of search points exceeds q, the search is terminated, the norm is sorted (S67), and a list is created with d points in ascending order of the norm. Let this be x _pi | i = 1,2,, d.

[Procedure Amendment 8]

[Document name to be amended] Statement

[Correction target item name] 0114

[Correction method] Change

[Correction content]

At S72, the deviation vector ξ _i between the neighboring point and the center of gravity R is calculated.

[Procedure Amendment 9]

[Document name to be amended] Statement

[Correction target item name] 0117

[Correction method] Change

[Correction content]

Further, in S75, the signal recording units 102, 20
The centroid R ^* of the point x ^* _pi after m steps of the points near _xn is calculated with reference to the past signal history recorded in 2.

[Procedure Amendment 10]

[Document name to be amended] Statement

[Correction target item name] 0118

[Correction method] Change

[Correction content]

In S76, the deviation vector ξ _i ^* between the center of gravity R ^* and x ^* _pi is calculated.

[Procedure Amendment 11]

[Document name to be amended] Statement

[Correction target item name] 0130

[Correction method] Change

[Correction content]

At S71, the smoothed time series data sa
Compute the sort sequence from (i) and the elements of x _n .

[Procedure Amendment 12]

[Document name to be amended] Statement

[Name of item to be corrected] 0138

[Correction method] Change

[Correction content]

In the following S85 to 87 , the d-dimensional norm is calculated, and if the value does not exceed the reference ε,
Add that point to the list of points near x ₀ .

[Procedure Amendment 13]

[Document name to be amended] Statement

[Name of item to be corrected] 0164

[Correction method] Change

[Correction content]

In this way, the linearization dimension d is changed and the difference between the successive prediction value and the actual measurement value is calculated. As a result, the characteristics of the user's hand movement can be stored as a dimension of linearization.

[Procedure Amendment 14]

[Document name to be corrected] Drawing

[Name of item to be corrected] Fig. 12

[Correction method] Change

[Correction content]

[Fig. 12]

[Procedure Amendment 15]

[Document name to be corrected] Drawing

[Correction target item name] Fig. 16

[Correction method] Change

[Correction content]

FIG. 16

Claims (9)

[Claims] 1. A camera shake prediction apparatus for a photographing apparatus, wherein a detecting means for detecting camera shake vibration and converting it into an electric signal, and a predetermined number of signals from the time-series signals from the detecting means are extracted, and each of them is extracted. By using each component of the coordinates of the dimensional space, a trajectory generation unit that associates the time-series signal with a point in the multidimensional space, and a time series of a plurality of points in the multidimensional space generated by the trajectory generation unit. And a prediction means for predicting a point to be arranged next in time series in the multidimensional space. 2. A storage means for storing a signal from the detection means, further comprising: a plurality of points in the multidimensional space, and the plurality of points in the multidimensional space, which are earlier in time series than the plurality of points. The point to be arranged next in time series in the multidimensional space is predicted based on the point in the multidimensional space generated from the signal stored in the storage means. The described camera shake prediction device. 3. A storage means for storing pre-measured time series camera shake signals is further stored, wherein the prediction means includes a plurality of points in the multidimensional space generated by the trajectory generation means, and the storage means. The camera shake according to claim 1, wherein a point to be arranged next in time series in the multidimensional space is predicted based on a plurality of points in the multidimensional space generated from the output from. Prediction device. 4. Further comprising smoothing means for smoothing the output from said detecting means, said trajectory generating means generating points in a multidimensional space based on the output from said smoothing means. The camera shake prediction device according to any one of claims 1 to 3. 5. The method further comprises smoothing means for smoothing the output from the storage means, and the predicting means includes a plurality of points in the multidimensional space generated by the trajectory generating means and the smoothing means. The camera shake prediction device according to claim 1, wherein a point to be arranged next in time series in the multidimensional space is predicted based on the point in the multidimensional space generated from the output. 6. A comparing means for comparing an output from the predicting means with an output from a detecting means for the output, and a dimension control means for changing the predetermined number so as to minimize the output from the comparing means. The image stabilization apparatus according to any one of claims 1 to 5, further comprising: 7. The comparing means for comparing the output from the predicting means with the output from the detecting means for the output, and the predicting means for outputting the output from the trajectory generating means so that the output from the comparing means is minimized. 7. The camera shake prediction apparatus according to claim 1, further comprising: a unit that controls the number of a plurality of points in the multidimensional space input to the unit. 8. The camera shake prediction device according to claim 1, wherein the prediction unit makes a prediction based on an algorithm of the Jacobi method. 9. The camera-shake prediction device according to claim 1, wherein the prediction unit performs prediction based on an orthogonal normalization algorithm.