JPH0926473A - Measurement device using spatio-temporal differentiation method - Google Patents

Abstract

(57) 【Abstract】 PROBLEM TO BE SOLVED: To accurately extract a velocity distribution of a moving body by using a space-time differential method. SOLUTION: This is a measuring device using a spatiotemporal differential method in which the azimuth angle of a pair of sound collecting devices 111, 112 is controlled to localize a measurement sound source, and an acoustic signal output from the measurement sound source is a pair of sound collecting devices. A time difference calculation means 120a for calculating the time difference until reaching the devices 111 and 112 at predetermined time intervals, and a self-evaluation amount calculation means 120b for calculating a self-evaluation amount that serves as a reference for determining the effectiveness of the calculated time difference. Then, based on the calculated self-evaluation amount, the determination unit 140 that determines the validity of the calculated time difference, and the azimuth angle control unit 113 that changes the azimuth angle when it is determined that the validity is low.
And a sound source localization means for locating the measurement sound source based on the time difference determined to have high effectiveness.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for generating a sound source to be measured from a result obtained by processing a temporally and spatially changing signal obtained with respect to the measurement object, for example, an image or sound, based on a spatio-temporal differentiation method. The present invention relates to a measurement device that extracts a time difference for localization.

[0002]

2. Description of the Related Art In recent years, various methods for extracting motion and stereoscopic information from a continuous image series have been proposed as researches of computer vision. In particular, a flow of an image formed by projecting a movement of a measurement target onto an image plane is called an optical flow, and is used for separating an object from a background, determining a three-dimensional structure and arrangement, and the like. This problem of optical flow determination has wide application not only in computer vision but also in the field of pure pattern measurement. From this viewpoint, when considering the conventional methods, they are roughly classified into two types. One is a correspondence search type method, which is executed by sufficiently closely repeating a procedure for associating one feature point with another feature point with respect to two continuous images. This is a complicated and uncertain procedure called a so-called correspondence problem, and various attempts have been made to introduce a stepwise search method for gradually improving the resolution of an image and to introduce various constraints.

Another method is based on the spatio-temporal differentiation of an image. This was first used to detect small movements of a TV image, and then formalized and analyzed using Lagrange differentiation.

[0004] Considering a system for observing an object moving in a plane with a fixed image pickup body, a velocity field near a point of interest (x, y) is set as (u, v). The time change factor of the image is limited to only the movement, and two images f ₁ (x,
Considering observing y) and f ₂ (x, y), these are locally expressed as f ₂ (x, y) = f ₁ (x−uΔt, y−vΔt) (1) Meets If the shift amount (uΔt, vΔt) is small and can be approximated to be locally constant, the right side of the equation (1) is Taylor-expanded around the point of interest, and f ₁ (x−uΔt, y−vΔt) ≒ f ₁ (X, y) −uΔtf ₁ x (x, y) −vΔtf ₁ y (x, y) (2) Where f ₁ x
(X, y) and f ₁ y (x, y) are respectively f ₁ (x, y)
X, y partial derivatives of That is, (1) bonded to the _{formula, f 1 (x, y)} -f 2 (x, y) ≒ uΔtf 1 x (x, y) + vΔtf 1 y (x, y) ... the relationship of (3) can get. When both sides are divided by -Δt, the left side shows the time derivative f ₁ t (x, y). f ₁ x (x, y), f ₁
Since y (x, y) is a derivative of the field, it can be calculated immediately on the image data and can be treated as a known quantity. Therefore, equation (3) can be regarded as a linear equation including these as coefficients and unknowns u and v. it can. This is the principle of the spatiotemporal differentiation method.

[0005] The method of determining the velocity field using this principle includes: 1) limited to one-dimensional motion; 2) assuming the smoothness of the velocity field, and estimating the roughness without contradicting equation (3). To find a velocity field that minimizes
Looking at this as a measurement method, 1) clarity and high speed that can form a velocity distribution only by numerical calculation, 2) versatility and objectivity without the need for prior knowledge of the object, 3) high resolution, 4 ) There is an advantage in that it is hardly affected by pattern deformation due to the use of a small deviation and the possibility of serious errors is small.

[0006]

However, the former method of the correspondence search type has the following disadvantages as a measurement method: 1) a serious error due to a correspondence error is mixed; 2) processing takes a very long time. There are many. In particular, unlike the computer vision system, the problem 1) is very difficult to solve in a general measurement system that does not require the introduction of preliminary knowledge.

Further, the latter spatiotemporal differentiation method has a weak point that is susceptible to noise, and its application is limited when the deviation is very small, so that it has been difficult to put it to practical use. Such a problem applies not only to the case where the measurement target is moving but also to the case where a still solid is photographed by two imaging bodies placed on the left and right and a stereoscopic image is extracted from a pair of images. This also applies to a case where sound from a sound source is collected by a pair of microphones, and a time difference between the pair of acoustic signals is extracted to localize the sound source. Further, in addition to image processing using an optical image, ultrasonic waves, N
The image processing by MR (nuclear magnetic resonance) and X-ray is widely applied, and in any field, the processing by the spatiotemporal differential method, which has a great many advantages from the above-mentioned problems, has not been put to practical use. is there.

An object of the present invention is to provide a measuring apparatus using a spatiotemporal differentiation method for accurately localizing a measured sound source.

[0009]

FIGS. 1 and 2 are conceptual explanatory diagrams of the present invention. As shown in FIG. 1, the present invention processes a signal related to an object to be measured by a spatiotemporal differentiation method. Calculation means 5 for calculating the physical quantity and the self-evaluation quantity of the measurement object
00 and determining means 501 for determining the validity of the calculated physical quantity based on the self-evaluation quantity. More specifically, according to the present invention, as shown in FIG. 2, a sensor 502 for detecting a signal related to a measurement target is provided.
And a calculating means 503 for processing a detection signal obtained by the sensor 502 by a spatiotemporal differentiation method to check a physical quantity to be measured and a self-evaluation quantity.
Determination means 504 for determining the validity of the calculated physical quantity
And a control unit 505 that changes the measurement condition of the sensor 502 for the measurement target to obtain a detection signal again and restarts the calculation unit 503 and the determination unit 504 when there is a physical quantity having a low degree of effectiveness. .

The present invention includes such means. The present invention will be described with reference to FIG. 33. When the azimuth of the pair of sound collecting devices 111 and 112 is controlled to localize the measured sound source, A measuring device using the spatial differentiation method, wherein an acoustic signal output from a measuring sound source is paired with a pair of sound collecting devices 111, 1
A time difference calculating means 120a for calculating a time difference until the time reaches twelve every predetermined time; a self-evaluation amount calculating means 120b for calculating a self-evaluation amount serving as a reference for determining the effectiveness of the calculated time difference; Determining means 140 for determining the validity of the calculated time difference based on the calculated self-evaluation amount; azimuth control means 113 for changing the azimuth when the validity is determined to be low; The above object is achieved by providing the sound source localization means for localizing the measurement sound source based on the time difference determined as above.

According to the present invention, the time difference until the acoustic signal output from the measurement sound source reaches the pair of sound collectors 111 and 112 is calculated at predetermined time intervals, and the effectiveness of the calculated time difference is determined. The reference self-evaluation amount is calculated. Then, the validity of the calculated time difference is determined based on the calculated self-evaluation amount, and when it is determined that the validity is low, the azimuth angle is changed and sound is collected again. Then, finally, the measurement sound source is localized based on the time difference determined to have high effectiveness.

[0012]

Embodiments of the present invention will be described below with reference to the drawings. First, as a premise of the present invention, a case where stereoscopic information is reproduced by binocular stereoscopic vision and a case where a moving image is reproduced will be described. Next, a case of time difference extraction used for localizing a sound source according to an embodiment of the present invention will be described.

First, a mode of reproducing stereoscopic information by binocular stereovision will be described. As shown in FIGS. 3 and 4, consider the irregular surface OB directly opposite to the height H, 2 pieces of imaged IS _L at intervals D, and systems IS _R is located. Right and left image sensing unit IS _L, are offset to each origin of the image plane of the IS _R captures the origin of the xy plane. Assuming that the erect image is taken at the same magnification for simplicity, the left and right images are f _L (x,
y), coordinates (x, y) on the object side such as f _R (x, y)
Can be expressed as a function of The target object is a simple uneven surface whose height from the xy plane is represented by z = h (x, y) (1.1). F as a function of (x, y)
Let it be expressed as (x, y).

[0014] At this time, the left and right image sensing unit IS _L, because IS _R captures video is something that the projection of the gray distribution of the surface in the xy plane, unevenness h (x, y) the influence of f (x, y) to A shift occurs. The deviation _{(Δ R x, Δ R y} ), (Δ L x, Δ L
y) from FIGS. 3 and 4

[Equation 1] And the left and right images f _R (x, y), f
_L (x, y) is _{f R (x, y) =} f (x + Δ R x, y + Δ R y) ... (1.3a) f L (x, y) = f (x + Δ L x, y + Δ L y) ... It is expressed as (1.3b). Using equations (1.2a) to (1.2c),

[Equation 2] Therefore, the coefficient of the unevenness h (x, y) is also affected. As can be seen from the mechanism of the imaging system, the image is enlarged as the object approaches the lens. The x, y coefficients {H-h (x, y)} / H in these equations represent the enlargement of this image. Rather than the contrast of the uneven surface function of the absolute coordinates (x, y), because it may consider that contain advance this scale enlarged as shading distribution of an object, shading f (x,
y),

(Equation 3) Can be written. Now, the change of the unevenness h (x, y) is sufficiently gradual, changes in height in the degree of the deviation delta _RL x is negligible, that is, the x-direction differential of h (x, y), hx (x, y) << 2H / D (1.8) Assuming that: _ΔRL x can be locally ignored.
On the other hand, the change in shading f (x, y) is smooth but h
Assuming that the change is steeper than the change in (x, y), Taylor expansion is performed in the x-axis direction around the point (x, y) as follows, and f _R (x, y) = f (x + Δ _RL x , y) = f (x, y) + Δ RL x f x (x, y) ... (1.9a) f L (x, y) = f (x-Δ RL x, y) = f (x, y _{) -Δ RL x f x (x} , y) is approximated ... and (1.9b). Taking the sum and difference of these two equations, f _R (x, y) + f _L (x, y) = 2 f (x, y) f _R (x, y) −f _L (x, y) = 2Δ _{RL x f x (x, y} ) , and the by substituting the first equation into the second equation by differentiating f (x,
y)

(Equation 4) Is obtained. Given the definition of differentiation, it is clear that this relationship holds when Δ _RL x is infinitely small.
This equation is considered to hold almost everywhere in the image.

Here, f _R (x, y) and f _L (x, y)
It is imaged images of the target three-dimensional IS _L, since an amount obtained by IS _R, (1.10) equation, and these coefficients,
It can be seen as a linear equation with Δ _RL x as an unknown.
That is,

(Equation 5) Is approximately determined by The sampling interval of the image may be used as Δx. However, the expression (1.12) is obviously affected greatly by noise because it is an expression composed of image differentiation and difference. Also

(Equation 6) Is not enough to cope with the bad condition that happens to be zero.

Therefore, when extracting a three-dimensional image based on the spatio-temporal differentiation method, it is necessary to reduce the amount of calculation and to make it strong against noise. First, assuming that “the height of the uneven surface can be approximated to be substantially constant in the vicinity of the point of interest (hereinafter, the size of this area is referred to as the neighborhood size)”, one height is calculated from many data of the approximate point. Ask for it. The size of the neighborhood size is determined by the balance between the spatial fineness of the shading f (x, y) and the resolution of the unevenness measurement (for example, a 5 × 5 to 7 × 7 area). This neighboring area is denoted by Γ (see FIG. 5). Δ in Γ
_{Since RL} x is assumed to be coincident, all points (x,
y)

(Equation 7) Holds. on the other hand,

(Equation 8) Assuming that 変 化 changes in 方程式, equations different by the number of pixels in Γ are established. These may be solved the Δ _RL x by coalition.

When the equation (1.14) is solved by the following least square method, the sum of squares of the left side and the right side, which represents the goodness of the equation (1.14), within Γ is minimized.

[Equation 9] Differentiating this with respect to Δ _RL x and setting it to zero,

(Equation 10) Is obtained. S with suffix is

[Equation 11] Represents the integral of the product of the differentials calculated by Substituting these values into equation (1.16) gives Δ _RL x

(Equation 12) Can be expressed as Therefore, the amount of deviation _ΔRLx and the unevenness h
From the relation (1.7) of (x, y),

(Equation 13) And the height of the unevenness is estimated.

Here, in order to stably obtain the expression (1.18), the denominator of the expression (1.18) needs to be sufficiently larger than zero. That is, for the least square method to be effective, (1.1
6) Determine the goodness of the normal equation

[Equation 14] Needs to be large enough. this

(Equation 15) That is, when the change in the X-axis direction of the pattern is uniformly zero in the vicinity area Γ, and there is no pattern locally,
Since the amount of deviation cannot be determined, it is necessary to select the size of 近 傍 such that the shading in the neighboring region 大 き く changes as much as possible.

The evaluation function J in the equation (1.15) is obtained under the optimum condition.

(Equation 16) I can write Since the value of this equation represents the residual error, the magnitude of the noise included in the image, the validity of the assumption of the equal elevation in the neighborhood Γ, and the change in the pattern due to the difference in the viewpoint (when the surface is glossy) It can be used as a criterion for determining the possibility.

Here, since the linear relationship (1.10), which holds when the deviation is small, is used as a basis, if the deviation is increased due to the increase in the unevenness, image processing cannot be performed properly. The main part of such an error is that the image with the displacement is Ta
Approximated by the first order term of ylor expansion (1.9a, b)
Produced by the formula. In order to estimate the range in which this approximation is valid, the equation (1.9a, b) is expanded to the second-order term.

[Equation 17] Becomes At this time, the condition for the size of the second-order term to be sufficiently smaller than that of the first-order term is expressed by using their variances.

(Equation 18) Is obtained. This relationship indicates that the upper limit of the amount of shift that can be detected is determined by the characteristics of the density of the target surface. That is, since the measurement range becomes wider as the average energy of the first-order differentiation of the shading is larger than the average energy of the second-order differentiation, a shading pattern in which a low spatial frequency component is main is more preferable. On the other hand, for an object having the same property, the condition to be satisfied by the imaging system is determined from this equation. (1.1
9) By substituting the equation, the measurement range of unevenness is estimated.

[Equation 19] Thus, it can be seen that it is better to make the distance D between the left and right imaging elements smaller than the imaging distance H in order to increase the measurement range of the unevenness. This is the opposite of the requirement in triangulation. However, there is a practical limit to reducing the size because it is susceptible to noise.

Next, the error given to the unevenness measurement result when it is assumed that noise is mixed in the pixel value is estimated. Assuming that noises added to the pixel at the coordinates (x, y) of the left and right images are n _L (x, y) and n _R (x, y),

(Equation 20) Assuming that the equation (1.12) is applied as it is to the calculation of the deviation ignoring the presence of noise,

(Equation 21) Given in. That is, the influence of noise is 1) smaller as the gray scale fx (x, y) of the image is larger, 2) the influence of the component common to the left and right images in the noise increases in proportion to the deviation, and 3) the difference component thereof. It can be seen that the effect of is independent of the deviation. Further, in 2), the common component of the noise is divided by the sampling interval Δx of the image. 4) When the noise of the pixel is independent, it is also shown that the error increases as the sampling interval becomes smaller.

On the other hand, in the local least square method, the error term is calculated by the same calculation as follows.

(Equation 22) Is required. The effect of noise in this equation is 1)
The larger the sum of squares in the neighborhood Γ of the density gradient fx (x, y) of the image, the smaller the sum of the squares. It can be seen that the influence of the components is independent of the deviation. Such a property is basically the same as that of the expression (1.24), but when the correlation of the noise is small, the noise component is averaged in Γ and the error is relatively reduced. Further, in 2), the common component of the noise is divided by the sampling interval Δx of the image. 4) If the noise of the pixel is independent, the error generally increases as the sampling interval decreases, but 5) the density gradient of the image. fx
If the change of (x, y) is gentle within Γ (note that this is different from the flatness of the image), the integrand of the molecule is canceled out, so that the error is significantly smaller than in the case of equation (1.24). Become.

As is apparent, (1.24) and (1.2)
Equation 5) represents an error when measuring the amount of deviation regardless of the arrangement parameters D and H at the time of imaging. Therefore, the error in measuring the unevenness is multiplied by a factor of 2H / D when converting the deviation into a height by the equation (1.19).

As described above, the unevenness h
(X, y) indicates that it can be obtained at any point on the image. It can be seen that this operation is a very organized operation method, assuming a parallel processing structure over the entire image. That is, for the input image f

(Equation 23) Are applied stepwise, and by multiplying by the coefficient 2H / D of the expression (1.19), the unevenness h (x, y) is determined simultaneously in a pattern. The processing at each individual stage is simple, and parallel processing is extremely easy in principle.

Next, an embodiment for extracting a three-dimensional image based on the above-described embodiment will be described. 6 to 12 show the embodiment. In FIG. 6, 11, 12
Are two solid-state imaging device cameras whose optical axes intersect at a convergence angle θ on a reference plane RP having a target stationary solid SOB. Here, a 128 × 128 pixel image sensor (MC manufactured by RETICON, Inc.) is used. -9000). The cameras 11 and 12 are controlled by a pixel clock and a synchronizing signal from the imaging body driving circuit 13. Also,
The optical axes of the cameras 11 and 12 are rotated by the convergence angle control device 20, and can shift the reference plane RP forward and backward (up and down in the drawing). That is, as shown in FIG. 7, both cameras 11, 12 are installed on a base 21, and
Each of 12 is connected to step motors 24 and 25 via connecting mechanisms 22 and 23. These step motors 24 and 25 are rotated by a signal from a drive circuit 26. That is, a convergence angle control signal from a self-determination / control logic unit 54 described later is supplied to the drive circuit 26, and the step motors 24 and 25 are rotated by a predetermined number of steps, whereby the optical axes of the cameras 11 and 12 are rotated. And the convergence angle θ is changed, and the intersection of the two optical axes is changed to the above-described reference plane R.
P can be changed before and after.

The analog image signals L and R from the cameras 11 and 12 are converted into digital signals L and R by the A / D converter 5 and input to the relative altitude / self-evaluation amount calculator 30.
Relative altitude and self-evaluation amount J _DET , J _ERR described later
Is calculated and output to the stereoscopic synthesis reproduction unit 50.

The three-dimensional composition reproducing unit 50 performs an operation as described below, and outputs an elevation map and an error map.
The three-dimensional information of the target object SOB is extracted from the elevation map.

Next, the relative altitude / self-evaluation amount calculation unit 30 and the three-dimensional composition reproduction unit 50 will be described in detail. (1) Relative altitude / self-evaluation amount calculation unit 30 This is a relative elevation calculation unit 30a and a self-evaluation amount calculation unit 30
b. FIG. 8 shows an example of the calculation unit 30. The image signals L and R are input to the sum and difference circuit 31, and the sum image signal R
+ L and a difference image signal RL are calculated. The sum image signal R + L is input to the X differentiating circuit 32.

(Equation 24) This summing circuit 33 may be omitted. The X differential value is supplied to the squaring circuit 34 and the multiplying circuit 35, and the average value is supplied to the squaring circuit 3
6 and the multiplication circuit 35.

The squaring circuit 34

(Equation 25) Is squared, and the result is input to the smoothing circuit 37. The smoothing circuit 37 integrates the input signal with a predetermined neighborhood size (for example, 5 × 5 pixels), and uses the output Sxx as the self-evaluation amount J _DET as the stereo synthesis reproduction unit 50 and the ratio circuit 4.
0, 43.

(Equation 26) The result is input to the smoothing circuit 38. This smoothing circuit 38 outputs an output S obtained by integrating the input signal with the above-mentioned neighborhood size.
xd is output to the ratio circuit 40 and the multiplication circuit 41.

[Equation 27] The result is input to the smoothing circuit 39. Smoothing circuit 3
Reference numeral 9 outputs an output Sdd obtained by integrating the input signal in the neighborhood size to the subtraction circuit 42.

[Equation 28] And outputs a relative altitude value Δ _RL x. The multiplication circuit 41 multiplies the output Δ _RL x of the ratio circuit 40 by the output Sxd of the smoothing circuit 38 to obtain a result Δ _RL x × Sxd,
Output to the subtraction circuit 42. The subtraction circuit 42 receives the input signal Δ
_RL x × Sxd, relative Sdd, performing an operation of _{Sdd-Δ RL x × Sxd =} J RES ... (1.27), and outputs a self-evaluation value J _RES. This equation (1.27) corresponds to equation (1.20). Ratio circuit 4
3 is the output J _RES of the subtraction circuit 42 and the output J _DET (Sxx) of the smoothing circuit 37,

(Equation 29) And outputs a self-evaluation amount _JERR . That is, the relative altitude / self-evaluation amount calculation unit 30 outputs the self-evaluation amounts J _DET and J _ERR and the relative altitude value Δ _RL x to the three-dimensional composition reproduction unit 50.

(2) Stereoscopic synthesis reproducing section 50 As shown in FIG. 6, the stereoscopic synthesis reproducing section 50 is composed of the following elements. Relative elevation value classification unit 51 This is realized in the form of software according to the processing procedure shown in FIG. 9, and classifies 128 × 128 pixels for each neighborhood size based on the input self-evaluation amounts J _DET and J _ERR. Is performed. That is, in the procedure P1, the self-evaluation amount J _DET from the relative altitude / self-evaluation amount calculation unit 30 is calculated as _follows :
The magnitude is compared with a predetermined threshold value SS1, and if J _DET ≧ the threshold value SS1, the procedure proceeds to step P2, where the self-evaluation amount J _ERR from the relative altitude / self-evaluation amount calculation unit 30 is compared with the predetermined threshold value SS2. Compare. If J _ERR ≦ threshold value SS2, in step P3, the label of the effective area is set to the center pixel of the neighborhood size, which is the target of calculation of the self-evaluation amount J _ERR .

If a negative determination is made in step P1, in step P4, a label of an invalid area is set for the center pixel of the above-described neighborhood size. If a negative determination is made in the procedure P2, a label of the excessive elevation area is similarly set in the procedure P5. That is, the self-evaluation amount J _DET determines the goodness of the normal equation (1.16) because the least squares method is effective, and must be sufficiently large. Also,
Self-evaluation amount J _ERR is the cause of the error is assumed only pixel noise, proportional to the noise variance delta ^2, can be used as an amount which predict that it contains what extent previously measured error, better smaller .

The load elevation / load accumulator 52 which is realized in the form of the circuit shown in FIG. 10, the relative elevation delta _RL x input, self-evaluation value J _ERR (used as a load), described later altitude bias Then, the cumulative load altitude and the cumulative load are calculated. That is, the relative altitude and the altitude bias are input to the summing circuit 521 for each neighboring area classified as an effective area, and the sum of both inputs is calculated by the ratio circuit 522.
Is input to The self-evaluation amount J _ERR is also input to the ratio circuit 522.

[Equation 30] Is calculated, and the output is input to the sum circuit 523. This accumulation circuit 523 is followed by an accumulation memory 524 via a switch 528, and the output of the accumulation memory 524 is input to the accumulation circuit 523. Therefore, the sum circuit 523
Is

(Equation 31) , New data is added to the past accumulated data, and the result is output. With the above circuit configuration, the cumulative load altitude of each pixel is output from the cumulative memory 524.

Further, the self-evaluation amount J _ERR is _calculated by a reciprocal circuit 525.
And 1 / J _ERR is output from the reciprocal circuit 525 to the sum circuit 526 at the next stage. The summing circuit 526 is followed by a cumulative memory 527 via a switch 529, and the output of the cumulative memory 527 is input to the summing circuit 526. Therefore, the sum circuit 526 is

(Equation 32) , New data is added to the past accumulated data, and the result is output. With the above circuit configuration,
The cumulative weight of each pixel is output from the cumulative memory 527.
The switches 528 and 529 described above are turned on / off by an accumulated control signal from a self-determination / control logic unit 54 described later.

This is realized in the form of a circuit shown in FIG. 10, and is based on the cumulative load altitude and the cumulative load input from the preceding load altitude / load accumulator 52 and an altitude map and an error map. Is output. That is, the cumulative load altitude and the cumulative load are input to the ratio circuit 531.

[Equation 33] Is calculated, output to the self-determination / control logic unit 54 described later, and taken out as final output data. On the other hand, the cumulative load is input to the reciprocal circuit 532, and the reciprocal thereof is calculated.

(Equation 34) Is output to the self-determination / control logic unit 54, and is extracted as final output data.

Self-determination / control logic unit 54 This is realized in the form of software according to the processing procedure shown in FIG. Based on the classification map from the relative altitude value classifying unit 51, the relative altitude from the relative altitude calculating unit 30a, the altitude map and the error map from the altitude reproducing unit 53, an altitude bias, a cumulative control signal, and a convergence angle. A control signal, an imaging camera activation signal, and a relative elevation operation unit activation signal are computed and output.

That is, in FIG. 11, the cumulative area and the convergence angle are initialized in a procedure P11, and an effective area is extracted in a procedure P12 according to the classification map. In step P13, an accumulated confirmed area is extracted according to the error map. This is to extract a region where the error map ≦ the threshold SS3 with respect to a preset threshold SS3. Then, the procedure P
At 14, an overlap area of both areas extracted at each of the procedures P12 and P13 is extracted. Then, the process proceeds to step P15, in which the average value of the relative altitudes in the overlap area, that is, the altitude bias (average altitude difference) is extracted and input to the sum circuit 521 in FIG.

Instead of the above procedures P12 to P15, as shown in FIG.
It is also possible to calculate the altitude bias by calculating the amount of forward and backward movement of.

Next, in step P16, the accumulation control signal is output to close the switches 528 and 529 shown in FIG. 10, and as described above, the altitude bias obtained in step P15 and the relative position of each pixel in the neighboring area are determined. The altitude and the altitude are added, and thereafter, the accumulated load altitude is stored in the accumulated memory 524 as described above, and the accumulated load at that time is stored in the accumulated memory 527. Thereafter, the process proceeds to steps P17 and P18 in order to extract the altitude load area and the altitude uncertain area, respectively. Here, the altitude uncertainty area refers to the previous procedure P13
Is an area other than the cumulative determined area extracted in. Further, in a procedure P19, an overlap area between these two areas, that is, an unmeasurable measurable area is extracted as a remaining area. Thereafter, the process proceeds to step P20, where the presence or absence of a remaining area is determined.
A convergence angle control signal is supplied to the step motor drive circuit 26 in order to control a predetermined amount of 1 and 12. For example, the reference plane R
The convergence angle is controlled so that P is shifted 2 ° to the front side. When the vergence angles of the cameras 11 and 12 are controlled, in step P22, an image pickup body activation signal is supplied to the image pickup body drive circuit unit 13 to instruct the image pickup of the target object SOB by the cameras 11 and 12, and the relative altitude is calculated. The unit 30a is activated by supplying the activation signal. If there is no remaining area in the procedure P20, the above processing ends. Although not shown in the figure, this process is also terminated when the convergence angles of the cameras 11 and 12 exceed the limit value.

FIGS. 13 to 15 show examples of experimental results when a cube having a random dot pattern on the surface shown in FIG. 7 is to be measured. 13 and 14 are elevation maps based on effective data obtained at different convergence angles θ, and FIG. 15 is a final result obtained by combining a plurality of such elevation maps. In addition, FIGS. 17 to 19 show experimental results when wedges are measured as shown in FIG. Each figure is an elevation map based on effective data obtained by making the convergence angle θ different. In the measurement, a random dot pattern was formed on the wedge.

As described above, the three-dimensional extraction according to this embodiment can be performed by: 1) an image (differential image, difference image) in which the spatial differentiation and the difference between the left and right images are pixel values from an image sequence obtained for a predetermined reference plane RP; Forming seeds; 2) forming two types of images (product images) having their products as pixel values; 3) smoothing them with a point spread function of the size of the neighborhood Γ (smoothed product images); ) Calculate the ratio of their sum of products over the entire image. Therefore, it can be realized by a multi-stage parallel operation circuit as shown in FIGS. 8 and 11, and three-dimensional information can be extracted in real time. The step 4) may be further disassembled.

Further, the three-dimensional information extracted by the self-evaluation amounts J _DET and J _ERR calculated in the process of extracting the three-dimensional information is evaluated, and its validity including validity / invalidity is determined, and the validity is determined. Weight J for the three-dimensional information of the
The image of the target solid is reproduced by performing weighting (weighting according to the degree of validity) by _ERR , and the region determined to be invalid is located before and after the reference plane RP, that is, between the imaging body and the target solid. The same processing is repeated by changing the distance H. Therefore, it is possible to accurately reproduce a stereoscopic image by the spatiotemporal differentiation method, which has conventionally been considered to be difficult to be realized because of being largely affected by noise and insufficiently coped with under bad conditions. Here, J _DET is an evaluation value indicating measurability, and the larger the evaluation value, the higher the effectiveness. J _ERR is an evaluation value indicating an estimated value of the measurement error, and J _RES is an evaluation value indicating validity of the measurement condition.

Although the random dot pattern is given to the target object in the above description, the random dot pattern may be projected on the target object in the project. In addition, for those having no pattern on the surface, it is necessary to project a random dot pattern by a project to provide a shading pattern on the surface. Further, the accuracy is further improved by changing the convergence angle for each measurement and projecting a random pattern every time the measurement is performed in order to change the shading pattern of the surface of the target object.

Next, a mode for extracting a moving image based on the spatiotemporal differentiation method will be described. As described in the related art, when extracting a moving image based on the spatiotemporal differentiation method,
It is necessary to reduce the amount of calculation and to be strong against noise. To eliminate noise, it is necessary to reduce the degree of freedom of the solution and increase the amount of data that substantially contributes to the solution.
Simply, it is assumed that "the velocity field can be approximated to be substantially constant in the vicinity of the point of interest", and one velocity is obtained by simultaneously combining data in the vicinity. The size of the neighborhood is determined based on the balance between the fineness of the density of the object and the spatial resolution of the velocity field measurement. If this region is denoted by Γ, the following must be established: ufx (x, y) + vfy (x, y) + ft (x, y) = 0 (2.1) Now, the goodness (badness) of the uniform establishment in Γ of this equation is evaluated by the square integration of the left side,
If this is minimized for each neighborhood,

(Equation 35) However, the description of the variable (x, y) is omitted. U, v
And normalizing it as zero, uSxx + vSxy + Sxt = 0 ｛(2.3) uSxy + vSyy + Syt = 0 is obtained. S with suffix is

[Equation 36] Represents the integral of the product of the differentials calculated by By substituting these values and solving equation (2.3), the velocity vector (u, v)
Is

(37) Is obtained as follows.

This operation has a structure suitable for parallel processing of the entire image. That is, Sx including a large amount of product-sum calculation
x, Sxy, Syy, Sxt, Syt, and Stt are calculated based on the entire input image f by: 1) a spatiotemporal differential image fx, fy, ft 2) a self-intersection image fx ² , fxfy, fy ² , fxft, fyft, ft ² 3) The moving average image Sxx, Sxy, Syy, Sxt, Syt, and Stt are replaced by parallel calculations. The moving average corresponds to the integral over Γ. Since there is no special processing in each stage, the processing can be performed by general-purpose image processing hardware.

In moving image processing using the spatio-temporal differentiation method, it is important to limit the application when the deviation is small, beyond the sensitivity to noise described above. The solution (2.5a, b) is completely meaningless unless the approximation of equation (2) shown in the prior art is established, but there is a risk that it cannot be determined from the solution. Therefore, it is necessary to understand the range within which the target speed can provide practically sufficient measurement accuracy.
In addition, it is necessary to determine whether the target speed is actually within the range.

Also, as a problem unique to this image processing,
There is a need to select an appropriate size of the neighborhood to avoid unnecessary degradation of resolution and to increase velocity uniformity within the neighborhood. Therefore, evaluation amounts and judgment criteria relating to these points will be described below.

A) Evaluation amount of measurability In order for equation (2.3) to be solved stably, the determinant of the coefficient matrix needs to be sufficiently large. Now, if we put this as J _DET ² ,

(38) Therefore, J _DET ² becomes zero according to the condition of the Schwartz inequality when fx (x, y) = Cfy (x, y) (C: arbitrary constant) throughout Γ. This equation indicates that the shading is uniform in one direction (x + y / c = constant straight line). That is, for the stability of the solution, it is necessary that the shading in Γ changes in two directions with a degree of freedom. This guides the selection of the size of Γ, and the degree to which 評 価 actually satisfies this condition can be evaluated by J _DET ² . In this sense, J _DET can be considered to represent the measurability at the point of interest.

B) Evaluation amount of validity of assumption Substituting equation (2.3) into evaluation function J and calculating residual error J _RES under optimal conditions,

[Equation 39] Is obtained. Since the value of this equation generally serves as an index of the inclusion of a factor outside the model, it serves as basic information for determining whether the assumption of a small deviation is not satisfied, the presence of pixel noise, the speed fluctuation in Γ, and the like.

C) Evaluation amount of measurement error variance As can be seen from the definition of the equation (2.2), assuming that the error is included only in the time derivative fx, the residual error J _RES roughly represents the area of this variance. I have. In fact, the first approximation error of Taylor expansion, which becomes dominant as the image shift increases, can be regarded as an error equivalently added to ft. Now, to investigate the propagation of this error to the solution, this is represented by white noise ε (x, y) satisfying the following equation.

(Equation 40) Is required. Since J _ERR indicates the dispersion of the solution expected from the degree of non- _satisfaction of the assumption and the shading near the point of interest, it has a meaning of the self-evaluation amount of the measurement error.

For the measurement to be meaningful, J _DET must be greater than the value that would inevitably occur due to pixel noise. Assuming that Γ is sufficiently large and the number of autocorrelations of pixel noise is Ψ (x, y), the differential theorem of the number of correlations gives

[Equation 41] The degree value can be taken by the noise-only contribution. Assuming that pixel noise is independent and pixel spacing is normalized to 1,
Assuming that the noise is white and the variance is σ ² within the Nyguist band of sampling at interval 1, the autocorrelation number is

(Equation 42) Given in. Substituting this (2.11) where the value of Motomari and (πσ) ^4/4, as a condition for measurable J _DET

[Equation 43] Is obtained. If σ ² = 0.5 quantization unit ² and Γ = 5 × 5 pixels, the right side becomes ≒ 62.

Since J _ERR is an estimate of the measurement error variance,
It is considered that the measurement point at which the expression (2.13) is satisfied represents the magnitude of the measurement error as it is. Therefore, the absolute permissible absolute accuracy A _ABS or the relative accuracy A _REL is given in advance, and the measurement points that do not satisfy J _ERR ≦ A _ABS ² or J _ERR ≦ A _REL ² (u ² + v ² ) (2.14) Is excluded from the final result.

Using the equations (2.13) and (2.14), in the present embodiment, the velocity field on the object is determined by combining the evaluation amounts of J _DET and J _RES. 2) A region where a speed that can be obtained is obtained. 2) A measurement is considered possible due to the presence of a pattern, but a region where reliable measurement was not performed due to excessive speed. 3) A measurement was not performed because there was no appropriate pattern on the object. It can be classified into three types of possible areas. Another possible cause of 2) is non-uniformity of the speed in the vicinity. This classification diagram is used not only for output with the final result but also for setting measurement conditions. That is, when the region 1) is sufficiently large and there is a margin in the value of J _DET , the size of the neighboring region is reduced to improve the resolution and the speed uniformity. When a large number of excessive speed regions occur in 2), the frame interval is shortened to suppress the image shift amount, and when the region in 3) is too wide, the size of the neighboring region is increased.

Next, an embodiment for extracting a moving image based on the above-described embodiment will be described. 20 to 3
1 shows an embodiment thereof. In FIG. 20, 61 is
This is a solid-state image sensor camera for photographing a target moving object MOB at predetermined frame intervals.
Pixel image sensor (MC-91 manufactured by RETICON)
28) is used. The camera 61 can freely set the imaging rate in a range of 10 to 500 frames per second according to the cycle of the pixel clock from the imaging body driving circuit 62. The control of the pixel clock is performed by a clock control signal from a self-determination / control logic unit 90 described later.

When the translation component to be measured is not zero, the image pickup camera 61 is moved by the tracking mechanism 63 to the moving object MO.
It is configured to follow the movement of B at an angle θ, thereby expanding the measurement range.

An analog image signal of two frames adjacent to the imaging camera 61 is converted into a digital image signal by the A / D converter 5 and input to the speed distribution / self-evaluation amount calculation unit 70,
A speed vector map and an error map (self-evaluation amount) J _ERR to be described later are extracted as a final output, and the self-evaluation amounts J _DET and J _ERR are output to a speed distribution data classification unit 8 to be described later.
Input to 0. The velocity distribution data classifying unit 80 classifies 128 × 128 pixels for each neighborhood size based on the input signals J _DET and J _ERR , and determines the effective area and the velocity for the central pixel of each neighborhood area. Set the labels for the excess area and the invalid area. In this manner, the information of each pixel on which each label is set, the velocity vector map from the velocity distribution / self-evaluation amount calculation unit 70, and the self-evaluation amount J
_ERR is input to the self-determination / control logic unit 90, which controls enlargement and reduction of the neighborhood size and increase and decrease of the pixel clock frequency, which will be described later.

Next, each of the above elements will be described in detail. (1) Speed distribution / self-evaluation amount calculation unit 70 FIGS. 21 and 22 show an example of the calculation unit 70. This is composed of a speed distribution calculator 70a shown in FIG. 21 and a self-evaluation calculator 70b shown in FIG.

Speed distribution calculation unit 70a In FIG. 21, this calculation unit 70a first takes in image data of two adjacent frames by the imaging camera 61 and forms differential images fx, fy, and ft in real time. That is,
The input image f is divided into an X differentiating circuit 71, a Y differentiating circuit 72, and T
The signals are input to the differentiating circuits 73, and the spatiotemporal differential images fx, fy, and ft are output from the respective circuits. Differential image f
x, fy, and ft are squaring circuits 74, 76, and 79, respectively.
, And multiplied by each other in multiplication circuits 75, 77, 78.
^{x 2, fxfy, fy 2,} fxft, fyft, ft 2 is output. These self-cross product images fx ² , fxf
y, fy ² , fxft, fyft, and ft ² are input to the next-stage smoothing circuits 80 to 85, respectively, and each of the smoothing circuits 8
Moving average images Sxx, Sxy, Syy, S from 0 to 85
xt, Syt, Stt are output. Here, moving average images Sxx, Sxy, Syy, Sxt, Syt, Stt
Corresponds to an integral value on a predetermined neighborhood size Γ, and the integration area is changed by a neighborhood size control signal calculated and determined by a self-determination / control logic unit 90 described later.

The outputs Sxx of the smoothing circuits 80 and 82,
Syy is input to the multiplication circuit 86, and as a result, SxxSy
y is output to the difference circuit 92. The output Sxy of the smoothing circuit 81 is input to the squaring circuit 87, and as a result, Sxy
² is output to the difference circuit 92. Therefore, the difference circuit 92
Performs an operation of SxxSyy−Sxy ² = J _DET (2.15) on input signals SxxSyy and Sxy ² , and outputs a self-evaluation amount J _DET . This equation corresponds to equation (2.6).

The outputs Sxx of the smoothing circuits 80 and 84,
Syt is input to the multiplication circuit 88, and as a result, SxxS
yt is output to the difference circuit 93. The outputs Sxy and Sxt of the smoothing circuits 81 and 83 are input to the multiplication circuit 89, and the result SxySxt is input to the difference circuit 93.
Therefore, the difference circuit 93 performs an operation of SxySxxt-SxxSyt (2.16) on the input signals SxxSyt and SxySxt. Then, the output J _DET of the difference circuit 92 and the output SxySxt−SxxSyt of the difference circuit 93 are input to the ratio circuit 95,

[Equation 44] Is output. Here, u represents a velocity vector.

On the other hand, the outputs S of the smoothing circuits 82 and 83
yy and Sxt are input to the multiplication circuit 90, and as a result, Sy
ySxt is output to the difference circuit 94. The outputs Sxy and Syt of the smoothing circuits 81 and 84 are input to the multiplication circuit 91, and as a result, SxySyt is output to the difference circuit 94. Therefore, the difference circuit 94 outputs the input signal SySxt
And SxySyt, the following operation is performed: SxySyt−SySxt (2.18). Then, the output J _DET of the difference circuit 92 and the output SxySyt−SySxt of the difference circuit 94 are input to the ratio circuit 96,

[Equation 45] Is output. Here, v represents a velocity vector.

FIG. 23 shows an example of the smoothing circuits 80 to 85. The smoothing circuit 80 for the image Sxx will be described below. The smoothing circuit 80 includes an Nг-stage shift register group 201 corresponding to the maximum neighborhood size Nг, an adder 202,
It comprises a switch group 203 for connecting the Nг × Nг area of the register and the adder 202. When the image Sxx is scanned, data is sequentially input to the shift register group 201. Now, the neighborhood size is N 近 傍 by the control signal.
When the number is set to × N ×, all the switches 203 are turned on, so that the data in the Nг × Nг area of the register group 201 is added by the adder 202 and output as the output of the smoothing circuit 80. On the other hand, when the neighborhood size becomes 5 × 5 by the neighborhood size control signal, each switch of the switch group 203 is turned on / off correspondingly, and, for example, 5 × 5 area data of the register group 201 is added by the adder 202. Output.

Self-evaluation amount calculation unit 70b In FIG. 22, this calculation unit 70b is a speed distribution calculation unit 7
Following 0a, the self-evaluation amount J _ERR is calculated. The speed vector u and the moving average image Syt are input to the multiplication circuit 97, and the multiplication circuit 97 outputs u × Syt. On the other hand, the velocity vector v and the moving average image Sxt are input to the multiplication circuit 98, and the multiplication circuit 98 outputs v × Sxt. The outputs u × Syt, v × Sxt from the multiplication circuits 97 and 98 and the moving average image Stt from the preceding operation unit 70a are added to the sum circuit 9.
9 is input. The sum circuit 99 performs an operation of u × Syt + v × Sxt + Stt = J _RES (2.20) on these input signals, and outputs a self-evaluation amount J _RES to the ratio circuit 101. The self-evaluation amount J _DET obtained by the preceding operation unit 70a is also input to the ratio circuit 101, and J _RES / J
_DET is calculated, and the result is output to the multiplication circuit 102 in the next stage. The moving average images Sxx and Syy are input to the sum circuit 100 from the operation unit 70a at the preceding stage.
x + Syy is output to the multiplication circuit 102. Multiplication circuit 1
02 for both input signals

[Equation 46] And outputs the self-evaluation amount J _ERR .
Equation (2.21) corresponds to equation (2.9).

Next, the speed distribution data classifying section 80 will be described with reference to FIG. This is realized in the form of software. Based on the self-evaluation amounts J _DET and J _ERR that are input, classification is performed for each 128 × 128 pixel for each neighborhood size. That is, in step P31, the self-evaluation amount J _DET from the preceding operation unit 70a is compared in magnitude with a predetermined threshold SS4, and if J _DET ≧ the threshold SS4, the process proceeds to step P32 and the operation from the operation unit 70b. The self-evaluation amount J _ERR is compared in magnitude with a predetermined threshold value SS5. J _ERR
If ≤ threshold value SS5, in step P33, the self-evaluation amount J
The label of the effective area is set for the center pixel of the neighborhood size that is the calculation target of _ERR .

If a negative determination is made in step P31, step P3
In step 4, the label of the invalid area is set for the center pixel having the above-described neighborhood size. If a negative determination is made in the procedure P32, a label for the excessive speed area is set in the same manner in the procedure P35. These classification maps are output to the self-judgment / control logic unit 90 at the next stage.

FIG. 25 shows the self-judgment / control logic unit 90 realized in the form of software. This self judgment /
The control unit 90 performs an operation described below in accordance with the input classification map, and controls the enlargement / reduction of the neighborhood size and the increase / decrease of the clock frequency, which is the frame interval of the camera 61.

In step P41, the area of the invalid area is calculated, and in step P42, the area of the excessive speed area is calculated. In step P43, the average value of the self-evaluation amount J _{ERR and} the average value of √ (u ² + v ² ) in the effective area are calculated. Next, in a procedure P44, it is determined whether or not the invalid area is excessive. If an affirmative determination is made, the process proceeds to step P48, where the neighborhood size control signal (FIG. 21) is used to enlarge the neighborhood.
) Of the smoothing circuits 80 to 85 are changed. If a negative determination is made in step P44, the process proceeds to step P45, in which it is determined whether the excessive speed region is excessive. If an affirmative determination is made, the process proceeds to step P50, in which the clock frequency is increased to reduce the frame interval.

If a negative determination is made in step P45, step P46 is reached.
To determine whether the average J _ERR has room. If an affirmative determination is made, the process proceeds to step P49, where the neighborhood size control signal is changed to reduce the neighborhood size. If the procedure P46 is negatively determined, the procedure proceeds to the procedure P47, where the average √ (u ² + v ² )
Is determined to be too small. If a positive determination is made, the procedure P51
And reduce the clock frequency to increase the frame interval. If a negative determination is made, this series of procedures ends.

FIGS. 26 to 28 show the results of real-time measurement of the flow velocity distribution of the fluid according to this embodiment. When a flat plate is inserted into the water surface on which the aluminum powder is floated by mixing the ink and moving along the water surface, the generation of a vortex is easily observed behind the flat plate. This pattern was measured in real time as a velocity vector distribution. The values of the speeds u and v and the evaluation amounts J _DET and J _ERR obtained on the 32 × 32 grid points are stored in the extended memory of the computer every 0.15 seconds, and are displayed after a series of measurements. The display can be performed at the same time, but the measurement interval becomes much longer. 26 to 28,
Marks with squares indicate areas where the speed range was appropriate and correct measurement was performed, and marks with only line segments indicate areas where there was a pattern on the target and measurement was possible, but measurement was not possible due to excessive speed. The mark of only the center point indicates an area where there is no pattern on the target and measurement is impossible. The blank area at the bottom represents a flat plate, and it is measured that the water surface is dragged from above as it moves downward, and the water is dragged. It can also be seen that the portion near the center of the vortex is judged to have excessive velocity or high non-uniformity in velocity.

FIGS. 29 to 31 show other experimental results. The object to be measured is a randomly sprayed black spot pattern several mm in size on a 32 cm-diameter white disk attached to the rotating shaft. The rotation speed of the disk is reduced to 0.0833 rotation / sec by the gear head, and the clock of the camera is set to 0.25 MHz,
By changing to 0.5 MHz and 1 MHz, an image shift equivalent to three stages of object speed was equivalently generated. The corresponding frame intervals are about 68ms, 34ms, 17ms respectively
At this time, the maximum deviation between adjacent frame images is 2.27 pixels, 1.14 pixels, and 0.5 pixels at the edge of the disk, respectively.
7 pixels. The change in sensitivity when the frame interval was changed was compensated for by the illumination intensity. Image data (8 bits for each pixel) always takes in two adjacent frames continuously and forms a differential image in real time by digital hardware.
For mechanical reasons at this time, the subsequent frame is used for forming a spatial differential image, and the previous frame is used only for temporal differentiation. The central difference method is used in the spatial derivative calculation to avoid introducing spatial anisotropy. In the experiment, 5 × 5 pixels were mainly used in the region near the smoothing, and 32 × 32 sampling was used while allowing one pixel overlap. Since the subsequent processing is sequential processing, this thinning calculation method has a great effect on speeding up. Furthermore, in this experiment, when the center point was designated, the above six types of sum of products were calculated at the same time by using a hardware mechanism for speeding up. The final output is the velocity component u
And v, an evaluation amount J _DET indicating the measurability, and a measurement error evaluation amount J _ERR in four 32 × 32 arrays. Under the above conditions, the time required from the capture of the pixel to the final output is about 0.15 seconds.

The measurement results shown in FIGS. 29 to 31 are plotted display, and the frame intervals are 17 ms, 34 ms,
For 68 ms, the direction and length of the line segment indicate the velocity vector (u, v) at that point (tip is downstream), and the presence or absence of the grid at the center point is determined by the measurement error evaluation amount J _ERR . It indicates whether it is within the range. The former specified value is 0.
25 (pixel length / frame interval) ² , the latter specified value is 64
(Brightness quantization unit) ² is used. In the result of FIG. 29, almost correct velocity distribution is obtained for the entire disk,
In FIG. 30 and FIG. 31, abnormal measurement results are spreading from the peripheral portion where the speed is high. Also, the range of measurement points where the J _ERR indicated by the square is below the specified value is also narrowed.

As can be seen from the above experiment, the present method uses the self-evaluation amounts J _ERR and J _DET to determine the area on the target object, 1) the area where the speed range is appropriate and the correct measurement was performed, It can be classified into two types: 2) a region where the target has a pattern and measurement is possible, but measurement was impossible due to excessive speed; and 3) a region where the target did not have a pattern and measurement was impossible. They are 1) with squares, 2) only line segments,
3) Only the center point is displayed as a grid point set.
In the case of 2), it is expected that the measurement range can be adaptively expanded by introducing dynamic control such as shortening the frame interval when a large number of overspeed regions occur.

The speed vector map and the error map are input to the synthesizing unit 110 shown by a broken line in FIG.
Data as shown in FIG. 31 can also be synthesized. Such a combination is effective for those in which the stationary pattern moves, such as the movement of the earth's cloud captured from an artificial satellite.

As described above, the moving image extraction according to this embodiment includes the following three steps: (1) an image (differential image) having pixel values of the spatial differentiation and the time differentiation from the image sequence of two adjacent frames obtained at a predetermined frame interval. Forming seeds; 2) forming five types of images (product images) having their products as pixel values; 3) smoothing them with a point spread function of the size of their neighboring region （(smoothed product images); And 2) calculating the ratio of the sum of the products over the entire image. Therefore, it can be realized by a multi-stage parallel operation circuit as shown in FIGS. 21 and 22, and a moving image can be extracted in real time. Step 4) may be further disassembled.

The extracted moving image information is evaluated based on the self-evaluation amounts J _DET and J _ERR calculated in the process of extracting the moving image, and the validity including the validity / invalidity determination is determined. The frame interval is increased or decreased based on the determination result, and the neighborhood size is enlarged or reduced to make it fit under ideal measurement conditions, so it is not affected by noise,
Further, it is possible to realize a moving image processing by a spatio-temporal differentiation method capable of adaptively expanding a speed range of a measurement target.

Next, an embodiment in which a spatio-temporal differentiation method is applied when extracting a time difference for localizing a sound source according to the present invention will be described. As shown in FIG. 32, a sound source is indicated by S, a pair of left and right microphones installed at an equal distance d from the center point 0 is indicated by ML and MR, an angle between the center point 0 and the sound source S is θ, and a distance is indicated by θ. D, the acoustic signals received by the microphones ML and MR are f _L (t) and f _L (t), respectively.
When f _R (t) and a virtual signal at the center 0 are represented by f ₀ (t), f _L (t) = f ₀ (t−τ) (3.1) f _R (t) = f ₀ (t + τ) (3.2) Here, τ represents a time difference between f _L (t) and f _R (t). Here, if τ is small enough,
Formulas (3.1) and (3.2) are Taylor-expanded to 1
Approximating with the next term,

[Equation 47] Becomes

Here, when obtaining the time difference τ by the spatio-temporal differentiation method, it is necessary to reduce the amount of calculation and to be strong against noise. Therefore, it is assumed that the time difference τ is constant within a certain time (t _{1 to} tm), and the solution is stabilized by the least square method. That is,

[Equation 48] Is the time difference τ1 to be obtained. Here, in order to stably obtain the expression (3.9), the denominator of the expression (3.9) needs to be sufficiently larger than zero. Further, the evaluation function J in the equation (3.8) is obtained under the optimum condition.

[Equation 49] I can write Since the value of this equation represents a residual error, it can be a criterion for determining the magnitude of noise included in the audio signal.

The above embodiment of the present invention will be described below. 33 to 35 show the embodiment. FIG.
3, reference numerals 111 and 112 denote sound sources AO to be measured.
The microphone is a pair of left and right microphones directly facing B at an azimuth angle θ, and the azimuth angle θ is controlled by the angle control unit 113. The sound signals collected by the microphones 111 and 112 are input to the time difference calculation / self-evaluation amount calculation unit 120, and the time difference τ1 and the self-evaluation amounts J _DET , J _ERR and J _RES are calculated. These signals are input to the self-judgment / control logic unit 140, and the arithmetic unit 1 is calculated based on the self-evaluation amounts J _DET and J _ERR.
The operation at 20 is evaluated, and if the predetermined evaluation is not reached, the azimuth angle θ is controlled to increase the resolution so that a predetermined evaluation can be obtained. Then, the time difference τ1 at that time is output as final data.

Next, the time difference calculation / self-evaluation amount calculation unit 12
0 will be described with reference to FIG. Microphone 11
The signals f _L (t) and f _R (t) from
31 and 132 respectively. Amplifier 131, 1
32 is input to a subtractor 133 and an adder 134, respectively.

[Equation 50] Is calculated and output as the time difference τ1, and the signal is input to the multiplication circuit 144. The multiplication circuit 144
Also receives the signal Std,

(Equation 51) Is input to the difference circuit 145.

[Equation 52] And outputs the self-evaluation amount J _RES . The self-evaluation amount J _RES is also input to the ratio circuit 146. Ratio circuit 1
46, a signal Stt = J _DET is also input.

(Equation 53) Is calculated, and a self-evaluation amount J _ERR is obtained.

Next, the self-judgment / control logic unit 140 will be described. This is realized in the form of software shown in FIG. First, in procedure P61, the self-evaluation amount J _DET
It is determined whether or not ≧ threshold SS6. If an affirmative determination is made, the process proceeds to step P62 where it is determined whether J _ERR ≤ threshold value SS7.
If an affirmative determination is made, a time difference τ1 is output in step P63. When the procedure P61 or the procedure P62 is negatively determined, the process proceeds to a procedure P64 to drive the azimuth control unit 113 to control the azimuth θ. The time difference τ1 obtained as described above
The sound source AOB can be localized based on

In the embodiment of the present invention described above, the time difference τ1 can be extracted in real time by the multi-stage parallel operation circuit shown in FIG. Further, the time difference information extracted by the self-evaluation amounts J _DET and J _ERR calculated in the extraction process is evaluated to determine _whether the time difference information is valid or invalid, and only the valid time difference τ1 is extracted. Therefore, inaccurate information due to noise or the like is eliminated, and the time difference can be extracted by the spatiotemporal differentiation method.

The present invention has been described above.
When the separation of the background from an individual object by the X-ray fluoroscopic image is realized by the spatio-temporal differentiation method, it can be sufficiently applied to the field of image processing not based on an optical image, such as image extraction by ultrasonic waves or NMR.

[0082]

According to the measuring device of the present invention, the time difference until the sound signal output from the measurement sound source reaches the pair of sound collecting devices is calculated for each predetermined time, and the effectiveness of the calculated time difference is calculated. Calculate a self-evaluation amount as a reference for determination, determine the validity of the time difference based on this self-evaluation amount, change the azimuth angle if it is determined that the validity is low,
When the effectiveness is determined to be high, the measurement sound source is localized based on the time difference, that is, the effectiveness of the time difference until the acoustic signal output from the measurement sound source reaches the pair of sound collection devices is determined. Since the azimuth angles of the pair of sound collection devices are controlled to be higher, the measurement sound source can be localized with high accuracy.

[Brief description of drawings]

FIG. 1 is a conceptual explanatory diagram of the present invention.

FIG. 2 is a conceptual explanatory diagram of the present invention.

FIG. 3 is a top view of a positional relationship between a photographing object and a target object.

FIG. 4 is a view from the left side of FIG. 3;

FIG. 5 is a diagram illustrating a neighborhood size.

FIG. 6 is an overall configuration diagram showing an embodiment of a mode for reproducing stereoscopic information.

FIG. 7 is a perspective view showing an imaging camera and a target object.

FIG. 8 is a block diagram illustrating an example of a relative elevation / self-evaluation amount calculation unit.

FIG. 9 is a flowchart illustrating an example of a relative elevation value classification unit.

FIG. 10 is a block diagram showing an example of a load altitude / load accumulating unit and an altitude reproducing unit;

FIG. 11 is a flowchart illustrating an example of a self-determination / control logic unit.

FIG. 12 is a diagram illustrating another example of calculating an altitude bias;

FIG. 13 is a diagram showing an experimental result of stereoscopic image extraction.

FIG. 14 is a diagram showing an experimental result of stereoscopic image extraction.

FIG. 15 is a diagram showing experimental results of stereoscopic image extraction.

FIG. 16 is a diagram showing a wedge from which a stereoscopic image is extracted.

FIG. 17 is a diagram showing experimental results of stereoscopic image extraction.

FIG. 18 is a diagram showing an experimental result of stereoscopic image extraction.

FIG. 19 is a diagram showing an experimental result of stereoscopic image extraction.

FIG. 20 is an overall configuration diagram showing an embodiment of a mode for extracting a moving image;

FIG. 21 is a block diagram illustrating an example of a speed distribution calculation unit.

FIG. 22 is a block diagram illustrating an example of a self-evaluation amount calculation unit.

FIG. 23 is a block diagram illustrating an example of a smoothing circuit.

FIG. 24 is a flowchart illustrating an example of a velocity vector classification unit.

FIG. 25 is a flowchart illustrating an example of a self-determination / control logic unit.

FIG. 26 is a view showing experimental results of moving image extraction;

FIG. 27 is a diagram showing experimental results of moving image extraction;

FIG. 28 is a diagram showing an experimental result of moving image extraction.

FIG. 29 is a view showing experimental results of moving image extraction;

FIG. 30 is a view showing experimental results of moving image extraction;

FIG. 31 is a diagram showing experimental results of moving image extraction;

FIG. 32 is a diagram showing a positional relationship between a sound source and a microphone in a mode in which the present invention is applied to time difference extraction used for sound source localization.

FIG. 33 is an overall configuration diagram showing an embodiment of the present invention.

FIG. 34 is a block diagram illustrating an example of a time difference / self-evaluation amount calculation unit.

FIG. 35 is a flowchart illustrating an example of a self-determination / control logic unit;

[Explanation of symbols]

 111, 112 sound collecting device 113 angle control unit 120 time difference calculation / self-evaluation amount calculation unit 140 self-judgment / control logic unit 500 calculation unit 501 determination unit 502 sensor 503 calculation unit 504 determination unit 505 control unit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display location // G01S 3/782 9061-5H G06F 15/70 410

Claims (1)

[Claims] 1. A measurement device using a spatiotemporal differentiation method for localizing a measurement sound source by controlling an azimuth angle of a pair of sound collection devices, wherein an acoustic signal output from the measurement sound source is a pair of the sound collection devices. A time difference calculating means for calculating a time difference until reaching the device every predetermined time; a self-evaluation amount calculating means for calculating a self-evaluation amount serving as a reference for determining the effectiveness of the calculated time difference; Determining means for determining the validity of the calculated time difference based on the calculated self-evaluation amount; azimuth control means for changing the azimuth when the validity is determined to be low; And a sound source localization means for localizing the measurement sound source based on the time difference determined to be high.