JPH06138137A - Moving-object extraction apparatus - Google Patents

Abstract

(57) [Summary] [Object] An object of the present invention is to provide a moving object extraction apparatus capable of stably extracting the contour shape of an moving object and its movement. A moving object extracting apparatus of the present invention is provided with a contour extracting unit that detects a moving object from the supplied image information and extracts the contour thereof, and a contour extracted by the contour extracting unit. It comprises a plurality of contour tracking means and a moving object extraction means for obtaining the contour shape and movement of the object from the shapes and movements of the plurality of contour tracking means.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION The present invention is
The present invention relates to a moving object extraction device that extracts a contour shape of a moving object and its movement simultaneously and stably.

[0002]

2. Description of the Related Art In recent years, a technique for extracting a moving object (hereinafter, simply referred to as a moving object) from an image is an important elemental technique as preprocessing for recognizing the moving object. Various animal body extraction methods have been proposed so far. The basic idea is to detect the movement of the moving body,
An attempt is made to extract the moving object area from the information on the detected movement of the moving object or from the information combined with the object area information extracted by other features (edge, color, texture, etc.). It was a thing.

For example, a typical method for detecting motion is a method using inter-image correlation (Tachikawa, Inaba, Inoue: Vision system with high-speed correlation calculation function (2nd report: application to real-time tracking processing). ), Proceedings of the 9th Annual Conference on Robotics in Japan (1991)), and other methods (Fukui, Kubota, Fukui, Kubota,
Mizoguchi, Ishikawa: Visual processing of balloon beanbag robot, 5th Symposium of image sensing technology in industry, Proceedings, pp.9
1-96 (1990)), optical flow method (Chiba,
Ozawa: Detection of optical flow from noisy images, IEICE Transactions, Vol.j73-D-II, No.12,
pp.1952-1959 (1990)), and further, the method by difference, the method of expanding in space and time (Mase: Estimation of object velocity in non-epipolar plane image, Computer Vision Research Group,
CV-71, pp1-8 (1991)) and the like.

Further, in order to accurately extract the moving body region, it is necessary to detect this motion information accurately and densely. However, in an actual image, originally, when an object having three-dimensional image information is projected on a two-dimensional surface, information loss and noise mixing occur, and further, the detected motion information includes an error. In many cases Further, the extraction of other features may not always be successful. For example,
With a simple method such as a Sobel operator, an edge may be intermittently extracted or an edge of the background may be extracted depending on the amount of lighting and the condition of the background. Therefore, the extracted region is also inaccurate.

Therefore, there is a method of fitting the contour of an object by applying the energy minimization problem in a dynamic system to a dynamic contour model prepared in advance, for example, a snake curve.
Proposed by Kass et al. (M. Kass, A. Witkin, D. Terzop
ouls, "Snakes: Active ContourModels", In Proc. 1st In
t.Conf.on Computer Vision, pp.259-268 (1987)).

This is the Active Contour Models (Snakes)
It is capable of extracting the contour shape and movement of a moving object at the same time. That is, this method defines an energy function that minimizes on the contour in the image, and converts the contour extraction problem from the object in the image into the energy function minimization problem using the relaxation method. It is a method of solving. This is an effective method for extracting the moving body region. Further, the energy function E at this time is defined as follows.

E = Σ (E _int (vi) + E _image (vi) + E _ext (vi)) (1) Here, E _int (vi) is the force (internal energy) that the snake curve itself tries to make smoothly. , E _image (vi) is a force (image energy) attracted to an image feature (line, edge, color, etc.), and E _ext (vi) is a forcing force (external energy) from the outside.

From another point of view, the Snake model can be considered as a method for obtaining a spline function under constraint conditions. Snake is a discrete point sequence of n points V _i = (X _i , Y
_i ) (i = 0, n). In terms of physical expression, it is likened to a process in which a wire having a certain degree of rigidity is gradually deformed by a force attracted to an edge. Energy is minimized when the attractive force and the restoring force of the wire itself are balanced. The external energy is a restraining force applied from the outside, but if such a force is not applied from the outside, it is not considered.

Since this snake solves the energy function minimization problem by the relaxation method, a very large amount of calculation is required. A method of solving this calculation by dynamic programming has been developed, but it is processed in real time. ,It's not easy. Further, the degree to which the snake curve itself tends to be smooth (hereinafter referred to as rigidity) is defined by a parameter in the internal energy function, and therefore it is necessary to optimally determine this. The stiffness parameter needs to be changed when extracting a fine uneven contour and when extracting a rough uneven contour, which further increases the amount of calculation.

Therefore, in order to solve these problems,
B-Spline snake (R. Cip), which introduces B-spline curves and Bezier curves that are often used to express curves in the field of computer graphics.
olla, A. Blake, “The dynamic Analysis of Apparent
Conours ”In Proc. 3st Int. Conf.on. Computer Vis
ion, pp616-623 (1990)) was developed.

Compared to the conventional snake, the B-sprain snake does not perform the convergent repetitive calculation, so the processing is light and the real-time processing is possible. FIG. 1 shows a schematic view of B sprain snake.

The concrete processing consists of the following five steps.

Step 1. Several control points are given as initial values.

Step 2. Coordinate X on the Bezier curve (or B-spline curve) represented by this control point
(S) and Y (s) are calculated by the following equations. Here, s is a parameter defined on the curve.

[0015]

[Equation 1] Note that n is the total number of control points, Q _ix is the X coordinate of the control point i, Q _iy is the Y coordinate of the control point i, and B _i (s) is the nth-order mixing function or B spline basic function.

[0016]

[Equation 2] Step 3. The strongest edge in the vertical direction of the curve at each sample point (the area with the largest concentration gradient)
To explore. The distance between itself and this edge is taken as the movement distance, and is calculated for all sample points.

Step 4. Amount of movement of each sample s point (d
_sx , d _sy ), the minimum of Q _ix and Q _iy that minimizes the following equation is 2
The multiplication is applied to obtain the amount of movement to the new control point i.

[Equation 3] Here, m is the total number of sample points on the curve.

Step 5. Control points only 3. Move by the amount of movement obtained in.

Step 6. Step 2. Return to.

[0020]

However, the conventional methods including B spline snake have the following three problems to be solved when used in an actual image.

First, when the background image is complicated and has many edges, only the outline of the object cannot be extracted well. This is because Snake simply searches for and extracts the edge closest to itself (the area with the largest density gradient) in the image. For a simple background image that does not include edges that are stronger than the contour of the object, the contour of the moving object can be extracted well, but if there are edges with a density gradient greater than the contour of the moving object in the vicinity, The edges of will be mistakenly extracted. This is a big problem in tracking the contour of a moving object.

Second, depending on the situation, the rigidity is not appropriate, and even a minute movement is easily affected by noise.

As described above, the B spline snake expresses the rigidity by the Beyer curve or the spline curve. Therefore, although it is not necessary to adjust the parameters related to rigidity, Snake reacts sensitively to such changes in situations where, for example, a part of the object becomes invisible, the shape changes abruptly, or noise in the image is received. Then, the contour shape is suddenly deformed.

Therefore, in some cases, there is a problem that the snake starts to vibrate and finally oscillates, making it difficult to maintain the contour shape of the moving body. Regarding this problem, it is reported that there are multiple vibration modes similar to the vibration analysis of a rigid body (Inagawa, Pentland, Sclar.
off, Correspondence determination method in shape estimation using active model: Correspondence in “3D Shape Reconst
ruction for ActiveModels ”, PRU91-58 (1991)).

Third, the amount of calculation may be larger than that of the conventional snake. B-Sprain Snake has a smaller amount of calculation than the conventional Snake, but it still needs to increase the number of sample points and control points when extracting a complicated shape faithfully in detail, and the amount of calculation is accordingly increased. The point is that the number increases and real-time processing becomes impossible.

The present invention has been made in view of the above-mentioned problems, and the contour shape of the moving object in the input image is irrespective of the background image, the rigidity of B spline snake, and the shape of the target moving object. It is an object of the present invention to provide an animal body extracting device capable of extracting both and its movements simultaneously and stably.

[0027]

In order to achieve the above object, the first invention of the present application is a contour extracting means for detecting a contour of a moving object from the supplied image information and a contour thereof, and the contour extracting means. It has a plurality of contour tracking means provided for the contours extracted in 1. and a moving object extracting means for obtaining the contour shape and movement of the object from the shapes and movements of the plurality of contour tracing means. .

The second aspect of the present invention is provided with a contour extracting means for detecting a moving object from the supplied image information and extracting the contour thereof, and a contour extracted by the contour extracting means. , A plurality of contour tracking means each responding only to a movement of the contour in a specific direction, and at least two contour tracking means reacting to a movement in a different specific direction among the plurality of contour tracking means are paired with each other. It is summarized that it has an interpolating means for interpolating information relating to movements in different specific directions, and a moving object extracting means for obtaining the contour shape and movement of the object from the shapes and movements of the plurality of contour tracking means.

Further, the third invention of the present application is provided with a contour extracting means for detecting a moving object from the supplied image information and extracting the contour thereof, and a contour extracted by the contour extracting means. , A plurality of contour tracking means each responding only to a movement of the contour in a specific direction and at least two contour tracking means reacting to a movement in a different specific direction among the plurality of contour tracking means are paired with each other. The gist is to have a restraint means for restraining the movement of the contour tracing means, and a moving object extracting means for obtaining the contour shape and the movement of the object from the shapes and movements of the plurality of contour tracing means.

Further, the fourth invention of the present application is provided with the contour extracting means for detecting a moving object from the supplied image information and extracting the contour thereof, and the contour extracted by the contour extracting means. A plurality of contour tracking means, an arbitrary point setting means for setting an arbitrary point in an area surrounded by the plurality of contour tracking means, an arbitrary point set by the arbitrary point setting means, and the plurality of contour tracking means Restraining means provided between the contour tracing means for restraining the movement of the contour tracing means, and a moving body extracting means for obtaining the contour shape and the movement of the object from the shapes of the plurality of contour tracing means and their movements. It is a gist to have.

[0031]

The moving object extracting apparatus according to the first invention of the present application detects a moving object from the supplied image information and provides a plurality of contour tracing means for the extracted contours. The unique reaction of a part is prevented from affecting other parts, and the contour shape and movement of the object are obtained from the shapes and movements of the plurality of contour tracking means. Further, since a plurality of contour tracking means are used, it is possible to perform parallel processing, and since each contour tracking means is small, the processing load can be reduced, and therefore real-time processing is possible.

The moving object extracting apparatus according to the second aspect of the present invention includes at least two contour tracking means that respond to movements in different specific directions among a plurality of contour tracking means that respond only to movements of the contours in a specific direction. As a pair, all contours can be stably extracted by interpolating information relating to movements in different specific directions.

The moving object extracting apparatus according to the third invention of the present application comprises at least two contour tracking means which respond to movements in different specific directions among a plurality of contour tracking means which respectively respond only to movements of the contour in a specific direction. As a pair, the movements of the contour tracking means are mutually constrained to be robust against disturbance.

In the moving object extracting apparatus according to the fourth aspect of the present invention, since the restraint means is provided between the set arbitrary point and each of the contour tracing means of the plurality of contour tracing means, the movement of the contour tracing means can be prevented. It was made to be restrained and robust against external disturbances.

[0035]

First, the basic concept of the present invention will be described. In order to solve the first problem described above, the present invention preferentially extracts moving edges.
Difference processing between consecutive images to extract this edge,
Threshold processing is performed to detect a change area where the brightness has changed. Next, the absolute value of this amount of change is taken as the amount of change.
When performing the difference processing between two consecutive images at the video rate, it is considered that this changed region substantially corresponds to the contour of the moving object. The edge search having the maximum gradient is performed in this brightness change area. This makes it possible to prevent the edge in the background image from being accidentally extracted.

Further, although the accuracy is somewhat deteriorated, it is possible to search for a portion having a maximum change amount in the change region as a contour.
When the object is stationary and no change area is detected, the closest edge, that is, the point having the maximum brightness gradient is extracted.

Further, in order to solve the above-mentioned second problem, it is avoided to extract the entire contour of the moving body in one process, and the entire contour is divided into appropriate small parts for processing. . Further, each partial process is performed while being restrained by being connected to each other by a virtual spring or a damper as restraining means. Each partial processing is the following 2
Realize in two ways.

As a first method, in each partial processing, edges are searched and extracted in different directions. Each partial contour extraction process limits the edge search to a specific direction (for example, horizontal or vertical). Each partial contour extraction process has the selectivity of reacting only the contour that moves in a specific direction.

Therefore, compared with the case where all contours are extracted at once, a part of the contours is extracted, and the influence of disturbance can be small. Furthermore, in the direction perpendicular to the search direction,
Since it does not change the length of the contour, it is robust against disturbances. In each partial contour extraction process, the motion information and the position information which the self lacks are received from other partial contour extraction processes. As a result, it is possible to stably extract all contours while avoiding the problem that the two-dimensional movement cannot be dealt with only by the partial contour extraction processing.

The second method does not limit the edge search to a specific direction, but configures the edge search process in two stages. Figure 3
Shows the flow of processing. First, in the first process, the contour model (for example, a snake curve) extracted before the temporary point indicated by A is subjected to affine transformation such as parallel movement and rotation movement while keeping its shape, and moved to the vicinity of the edge.

Subsequently, in the second processing, an edge is searched for in a narrow range to allow the contour model to be deformed, and the edge model is completely matched (fitted) with the edge.

The affine transformation coefficient in the first process is obtained from the moving amount of each sample point of the contour model using the least square method. Also, only the calculated movement amount and rotation angle,
Affine transform the contour model. With this method, the shape of the contour model is maintained, and the contour model is moved to the vicinity of the edge at the current point in time, and minute deformations are allowed after that. Can be solid.

Further, the above-mentioned third problem can be solved by the means described above. Since each part of the whole contour is extracted by a plurality of different processes, each process can be distributed and parallelized, and real-time processing can be dealt with.
Further, even if the number of sampling points and control points increases, it is possible to easily cope with them by increasing the number of parallel processings.

Next, one embodiment according to the present invention will be specifically described with reference to the drawings. Extracting the face area from the image is an essential elemental technique in the human I / F technique using the image. In the present embodiment, in order to make it more specific, a case of extracting the face contour will be described as an example.
FIG. 1 is a block diagram showing the configuration of a moving body extracting apparatus according to the present invention.

As shown in FIG. 1, the moving object extraction apparatus of this embodiment has an image input unit 1, a tracking contour determination unit 2, a contour tracking unit group 3, a host computer 4, an image output unit 5, and a position. And a decision unit 6.

Further, the image input section 1 is provided with a camera 11 and an A / D.
The tracking contour determination unit 2 configured by the converter 13 and connected to the A / D converter 13 via the image bus B _I includes an image memory 21, a difference circuit (edge extraction circuit) 23, and a binarization circuit 25, It is composed of a labeling circuit 27 and a circumscribed rectangle calculation circuit 29. Similarly, the contour tracing unit group 3 connected to the A / D converter 13 via the image bus B _I is a local module, specifically, n contour tracing units 31a,
31b to 31n. Further, the host computer 4 includes a contour tracking unit management unit 41, a binding force calculation unit 43,
It is composed of an entire contour calculation unit 45 and a three-dimensional position calculation unit 47. The image output unit 5 includes a three-dimensional position calculation unit 47 and an output monitor 51 connected to the image bus B _I. Further, the position determination unit 6 is further defined as the mouth position determination unit 6
1, a both-eyes position determination unit 63 and a nose position determination unit 65.

Next, a detailed description will be given with reference to FIGS. As the camera 11 of the image input unit 1, for example, an ITV camera is used. Here, the image input from the camera 11 is referred to as image 1. The input image 1 is digitized by the A / D converter 13.

In the tracking contour determining section 2, first, in step S1
In the difference circuit 23, the difference between the current image captured by the camera 11 and the image data of one frame before stored in the image memory 21 or the background image data in which an object previously input does not exist is obtained. In the case of the difference from the background image, the binarization circuit 25 binarizes the data, the labeling circuit 27 labels the data, and the circumscribed rectangle calculation circuit 29 obtains the circumscribed rectangle coordinates for the moving object (step S3). The information such as the circumscribed rectangular coordinates for the moving object that has been calculated is used as the contour tracking unit management unit 4 of the host computer 4.
It is output to 1.

The contour tracking unit management unit 41 uses the contour tracking unit group 3
Are arranged in a rectangular area including each side of the circumscribed rectangle. The position where the contour tracking unit group 3 is arranged and the size of the contour model are
Determined by this circumscribed rectangle. Specifically, when four contour tracing units are arranged in the circumscribed rectangle extracted as shown in FIG. 4, horizontal contour tracing units L _HL and L _HL are provided on the vertical sides.
_HR , the vertical contour tracking units L _VU and L _VD are arranged on the side,
Further, as shown in FIG. 5, when eight contour tracking units are arranged, horizontal contour tracking units L _HLU , L _HRU , and L are arranged on the vertical _sides .
_HLD and L _HRD are vertical contour tracing units L _VUL and L on the horizontal side.
_VDL , L _VUR , and L _VDR are arranged respectively (step S5).

Alternatively, it is conceivable that the tracking contour determination unit 2 and the contour tracking management unit 41 perform the following processing. The difference between consecutive images is calculated, and threshold processing and binarization processing are performed to extract the contour area of the object. Usually, this contour area is often divided and extracted. Therefore, the plurality of extracted partial areas are once connected and divided again. The division is performed based on the rotation angle about the center of gravity G of the contour. For example, in the case of four divisions as shown in FIG. 6, the horizontal contour tracing units L _HL and L _HR and the vertical contour tracing units L _VU and L _VD are arranged in a rectangular area including this line segment at intervals of 90 degrees. To do. In case of 8 divisions,
Divide into 45 degree intervals.

Further, each of the contour tracking portions L may be arranged interactively by an inputting means such as a mouse or a keyboard while a person looks at the image on the output monitor 51.

Each of the arranged contour tracking units L executes step S
In step 7, edge search processing is performed in each specific direction. For example, assuming that the specific direction is the horizontal / vertical direction in the image, the contour tracking group 3 includes a plurality of contour tracking units 31 as shown in FIG.
a, 31b, ..., 31n.
Horizontal contour tracking unit L set by an arbitrary contour tracking unit 31k
_{In H} (L _HLU , L _HRU , L _HLD , L _HRD ) the edge search direction is limited to the horizontal direction. Other arbitrary contour tracking unit 3
Vertical contour tracking unit L _V (L _VUL , set by 1 l,
In L _VDL , L _VUR , and L _VDR ), the edge search direction is limited to the vertical direction. That is, the horizontal contour tracking unit L _H responds only to the horizontal movement of the contour. Similarly, the vertical contour tracking unit L
_V responds only to vertical movement of the contour.

Immediately after the arrangement as described above, in the search processing of step S7, only the inside of the circumscribed rectangle is edge-searched in order to attach the contour model to the object. At this stage, each contour tracking unit proceeds independently. However, as it is, each contour tracking unit can only track one-dimensional movement in a specific direction, and thus cannot trace the entire contour of the object. Therefore, the lacking position information is complemented by the contour tracking units.

Hereinafter, with reference to FIG. 8, description will be given of a case where the four contour tracking units are used. Vertical contour tracking section L
1 and the vertical contour tracking unit L3 obtain the lacking horizontal position (the center of gravity position of the contour model) from the center of gravity positions of the horizontal contour tracking unit L2 and the horizontal contour tracking unit L4. On the contrary, the horizontal contour tracking unit L
2 and the horizontal contour tracking unit L4 determine the insufficient vertical position (the center of gravity position of the contour model) from the center of gravity positions of the vertical contour tracking unit L1 and the vertical contour tracking unit L3.

In the case of being composed of eight contour tracking units,
It shows in FIG. The vertical contour tracking unit L1 obtains the missing horizontal position from the barycentric positions of the horizontal contour tracking units L2 and L7. The three positional relationships are expressed by x1 = x7 + (x2-x7) * (2/3). The vertical contour tracing section L8 also has a horizontal contour tracing section L.
It is calculated from the position of the center of gravity of 2 and L7.

X8 = x7 + (x2−x7) × (1/3) Similarly, the horizontal contour tracking section L2 finds the missing vertical position from the center of gravity of the vertical contour tracking sections L1 and L4.

Y2 = y4 + (y1−y4) × (2/3) Obtained from the barycentric positions of the horizontal contour tracing section L3 and the vertical contour tracing sections L1 and L4.

Y3 = y4 + (y1−y4) × (⅓) Similarly, other contour tracking units L are similarly obtained from insufficient positions. The enlargement / reduction of the size of the object is dealt with by changing the size of the contour model of each contour tracking unit described later.

The contour tracking unit 31 is processed by a plurality of local modules which can be controlled by a host computer (Kubota, Fukui, Ishikawa, Mizoguchi: “Vision processor concept and prototype model for object recognition and discrimination. Development ”, Confidential Techniques, PRU89-07 (1990)). The local module includes a local processor 75 as shown in FIG. 10, and various parallel processes can be realized by software.
The local processor 75, the bus interface 71 via the local bus B _L, the address decoder 73,
It is connected to the window controller 77 and the window memory 79.

A rectangular area of an appropriate size including the line segment at the center is defined and its position (address) is set in the register of the window controller. The local module loads the image of that portion at the present time (time t) into the window memory 79 from the image bus B _{I at the} video rate. This area is called a tracking window W (see FIG. 7). The tracking window W is set to a size including the contour model. Next, at the next time point (time t + Δ), the image of the same portion is fetched from the window memory 79. First, the local processor 75 performs a difference process between the two continuous images to detect a change area. Furthermore, edge search processing is performed on the captured image data in a specific direction within the change area to extract and track edges.

The processing of the contour tracking section 31 will be described later in detail. The extracted edge position information is transferred to the contour tracking unit management unit 41 on the host computer 4 via a parallel port or the like at any time.

The contour tracking section management section 41 obtains the constraint force "Force" from the contour position information sent, and calculates the position of each contour tracking section again (step S9). In addition, Size of the contour model is obtained and sent back to each contour tracking unit together with the position. The binding force is realized by a virtual spring (wavy line) and a virtual damper (solid line) in the figure, and is obtained by the following formula.

Force = force _sp + force _dp (6) force _sp = Const 1 * (Len _ij −Len _natural ) (7) force _dp = Const 2 * (Dlen _ij / D _t ) (8) where Len _ij is the contour Tracer i, distance between, Len
_natural is the natural length of the virtual spring, Const1 is the spring constant, and Const2 is the damping constant.

The size of the contour model of each contour tracking unit changes as the size of the object increases or decreases. That is, if there is a gap between the contour tracking units, fill it in, and if there is a gap, shorten it. As a standard when changing,
If only the size of some contour models increases, the calculation time will increase accordingly and the overall processing time will also increase.
Each size is determined so that the length is as uniform as possible.

An example of the vertical contour tracking section L1 will be described with reference to FIG. The horizontal size of the contour model of the vertical contour tracking unit L1 is determined by the X coordinates of both ends, Xs1, new and Xe1, new.
Here, Xs1, new is determined by the interval between the X coordinates Xes, old and Xs1, old of the end point of the adjacent horizontal contour tracing section L8 and the horizontal length Leng1 of itself. Here, the subscript new is the coordinate of the newly calculated end point, and old is the coordinate before the calculation.

X _{s1, new} = X _{s1, old} − (X _s1 −X _e8 ) * rat (9) rat = Leng8 / (Leng1 + Leng8) (10) where Leng8 is the horizontal length of the horizontal contour tracking unit L8. Is.

The X coordinate Xe1, new of the other end point is obtained by the following equation.

X _{e1, new} = X _{e1, old} − (X _s2 −X _e1 ) * rat (11) rat = Leng2 / (Leng1 + Leng2) (12) Here, Leng2 is the vertical length of the horizontal contour tracking unit L2. Is.

Similarly, the vertical size of the contour model of the horizontal contour tracing section L2 is determined by the Y coordinates of both ends, Ys2, new and Ye2, new. Here, Ys2, new is the adjacent vertical contour tracking unit L1.
It is determined by the interval between the Y coordinates Ye1, old and Ys2, old of the end points of and the vertical length Leng2 of itself.

Y _{s2, new} = Y _{s2, old} − (Y _e1 −Y _s2 ) * rat (13) rat = Leng1 / (Leng1 + Leng2) (14) The Y coordinate Ye2, new of the other end point is expressed by the following equation. I want it.

Y _{e2, new} = Y _{e2, old} − (Y _e2 −Y _s3 ) * rat (15) rat = Leng3 / (Leng2 + Leng3) (16) where Leng3 is the vertical length of the horizontal contour tracing section L3. Is.

The sizes of the contour models of the other contour tracking units are also
The same is obtained. As a result, even if the face is scaled up or down depending on the distance in the image, the contour model size of each contour tracking unit changes optimally, and the contour can be extracted and tracked successfully.

In the total contour calculating section 45, step S11
Then, the entire contour shapes are calculated by connecting the obtained partial contour shapes. If the connection of the partial contours does not succeed, connect the smoothness again by adding the constraint of the smoothness of the shape. Also,
The portion where the contour shape is not obtained is interpolated by a spline curve or the like.

Next, in steps S13 to S17,
The mouth position determining unit 61, the both eye position determining unit 63, and the nose position determining unit 65 determine the positions of the mouth, both eyes, and nose, respectively.

That is, first, based on the contour shape information of the extracted face area, the approximate areas of both eyes, nose, and mouth are limited based on the face model registered in advance. Further, the positions of eyes, nose and mouth are determined in detail within this area. This is realized by matching with an image mask pattern prepared in advance. The mask pattern is enlarged / reduced in proportion to the size of the face. Alternatively, edge extraction and Hough transformation are performed to match with a shape registered in advance. For example, a circle area is found and used as the eye position. If it is a nose, the triangular area is the nose.

In the three-dimensional position calculator 47, step S1
At 9, the three-dimensional position of the face is obtained from the contour shape of the face and the relative positional relationship between the eyes and the nose or the mouth. The relationship between the size of the contour of the face and the distance from the camera is obtained in advance. As a result, the distance is obtained from the size of the extracted face contour. The orientation of the face is obtained from the geometrical relationship between the eyes and the nose or the mouth based on the face model.

The image output unit 5 outputs the extracted partial contour shape to the output monitor 51 (step S21).

Next, the processing of the contour tracking section will be described in detail. Each local processor 75 extracts and tracks contours in the captured image data. The flow of processing is shown below.

Step 11. The tracking window W is arranged in the rectangular area including the extracted partial contour.

Step 12. Some, for example four, control points are given on the bisector of the tracking window W.

Step 13. Coordinates X (s) on the Bezier curve or B spline curve represented by these control points,
Y (s) is calculated by the following formula. Here, s is a parameter defined on the curve.

[0082]

[Equation 4] Here, n is the total number of control points, Q _ix is the X coordinate of the control point i, Q _iy is the Y coordinate of the control point i, and B _i (s) is the nth-order mixing function. Alternatively, it may be replaced with a B-spline basic function.

[0083]

[Equation 5] Step 14. Difference processing is performed between consecutive images, threshold processing and absolute values are calculated, and a change area is detected.

Step 15. As shown in FIG. 12, at each sample point s, an edge area is searched for in a specific direction within the change area. Here, the normal direction to the curve of each sample point will be described. The distance between each sample point and the searched edge area is taken as the movement distance. This moving distance is obtained for all sample points.

Step 16. From the moving amount (d _sx , d _sy ) of each sample s point, Q _ix and Q _iy that minimize the following equations are obtained by applying the least square method, and the moving amount to a new control point i is obtained.

[0086]

[Equation 6] Here, m is the total number of sample points s on the curve.

Step 17. Step 1 for each control point only
The amount of movement obtained in 3 is moved.

Step 18. The obtained curve is written in the local image memory and the contour is displayed on the output monitor 51 of the image output unit 5.

Step 19. Next, the local window is moved by the average movement distance Distance of all control points. At the same time, each control point is moved by -Distance.

Step 20. Return to step 13.

Incidentally, in the processing of step 15, in the change area, although the accuracy is slightly lowered, instead of the edge search, a portion having the maximum change amount may be searched as an edge.

Next, with reference to FIG. 13, the result of horizontal motion contour tracing using an actual image will be described. In FIG. 13, the upper solid line W1 represents the conventional B-spline snake, and the lower solid line W2 represents the contour tracing means with new difference information. The rectangle indicates a local window. The two means are placed in the same initial position (equal to the X coordinate). Along with the horizontal movement of the moving body (stick), the stick is tracked, and it is tracked halfway. However, the boundary line W1 detects an edge with a strong background,
Tracking has failed. On the other hand, the boundary line W2 is successfully tracked to the end.

FIG. 14 shows the result of extracting the contour of the face in real time. It can be seen that each contour tracking unit fits the face. The contour tracking unit consists of two horizontal contour tracking units and one vertical contour tracking unit. FIG. 15 shows the configuration of each contour tracking unit. In FIGS. 13 and 14, the rectangle indicates the tracking window W of the local module. The boundary lines W1 and W2 in the window are the horizontal contour tracking portions L _HL and L
_HR and the boundary line W3 indicate the vertical contour tracking portion L _V.

The constraint force between the contour tracing units can also be realized by a method using a physical template. In order to realize this restraint force, a plurality of virtual springs and virtual dampers as shown in FIG. 16 are placed at the center of gravity g position and the total contour center of gravity G of each contour tracking unit, and each contour tracking unit is constrained in the radial direction. Add force. Further, a virtual coil spring is placed at the center of gravity G of all contours to realize the restraining force in the rotational direction. The spring constant of each spring and the initial value of the damper constant are initially set to zero.

When the contour tracing succeeds in the preset predetermined processing step time, the natural length of the spring is set to the distance between the centers of gravity at this time. These values grow over time. In each spring, the value of potential energy is calculated and this time change is monitored. When the time change becomes larger than the set value,
The natural length of the spring is reset to that length.

The addition of the rigidity described above can be realized by means of a two-step process other than the means for performing edge search in the specific direction described above.

The basic idea is to divide the whole process into two steps. The first process is called freeze and the second process is called relax. First, in the freeze processing, from the motion information of each searched sample point,

[Outer 1] Is calculated by the least squares method.

[0098]

[Equation 7] All control points are rotated and moved in parallel by the calculated parameters. Here, all control points undergo the same transformation. This means rotating and translating without changing the shape. In the next relaxation process, a range narrower than the search range of the freeze process is searched for at each sample point to obtain a motion. At this stage, allow the shape deformation,
Move all control points. In this way, the two-step processing provides rigidity even when the moving body moves suddenly or changes its shape, and is strong against disturbance.

Next, a moving object tracking camera as a second embodiment will be described.

In this embodiment, a moving object area is extracted from an image, and the movement of the moving object area is tracked to perform pan, tilt, zoom,
The present invention relates to a camera that automatically controls focus. Figure 17
1 is an overall configuration diagram of an embodiment, which is an image input unit 1A, a tracking contour determination unit 2A, a contour tracking unit group 3, a contour tracking unit management unit 41, a total contour calculation unit 45, and a three-dimensional position calculation. Part 4
7, a camera control unit (not shown), and a camera platform control unit 8
And an image output unit.

Image input unit 1A, tracking contour determination unit 2A, contour tracking unit group 3, contour tracking unit management unit 41, total contour calculation unit 4
Reference numeral 5 has substantially the same operation as that described in the first embodiment. The camera control unit includes a focus control unit, a zoom control unit 85, and an aperture control unit.
The focus control unit adjusts the focus from the input image using the focus control circuit. Zoom control unit 85
Then, the zoom amount is adjusted based on the size of the object calculated by the three-dimensional position calculation unit 47.

The zoom control amount is feedback loop controlled so that the ratio Ratio of the area of the object region in the image is always constant.

Ratio = α1 * (s / s0) (2
1) Here, s is the area of the object in the image, and s0 is the area of the entire image.

The camera platform controller 8 has a pan controller 81.
And a tilt control section 83. The pan / tilt control units 81 and 83 are configured to constantly capture the object in the image.
The pan and tilt angles of the platform 15 are changed. This controller forms the field back loop control. Position (x, y) of the object in the image, pan Pan, tilt Til
The control amount of the t angle is obtained by the following formula.

Pan = α1 * (x−x0) + β1 * vx (22) Tilt = α2 * (y−y0) + β2 * vy (23) where (x, y) are the barycentric coordinates of the object, and (x0, y0) is the image center coordinate, and (vx, vy) is the velocity of the object on the image. α1, α2, β1 and β2 are optimum constants determined by the control theory.

Here, the problem is that when the camera itself is moving, it cannot be tracked well by the above contour tracking method. Because, in the difference, the edge of the background also moves, so it is detected. To solve this, when the camera is moving, edge extraction is simply performed, and when the camera is stationary, the edge search is combined with the above-described difference processing to solve the problem.

Next, a workstation WS with a moving object extracting function as a third embodiment will be described.

This embodiment relates to a workstation WS equipped with the above-mentioned moving object extracting device. FIG. 18 is an overall schematic view of the apparatus of the embodiment, which is the A / D conversion unit 13B.
And an image memory 21B, an image input unit, an object extraction processing unit 91, a display change calculation unit 93, an object tracking calculation unit 95.
A central processing unit (CPU) 9, a monitor turntable control circuit 97, a camera platform control circuit 98, a camera control circuit 99, a graphic controller 101, and an image output unit 51Bs. FIG. 19 shows an overview of this workstation. The image input unit 11B is the first
What is described in the embodiment of is used. Small IT camera 11B for input with the platform 15B installed on the monitor 51B
Has the object tracking function described in the second embodiment and tracks the face area of the user. The monitor 51B is placed on a turntable 97a having two degrees of freedom (pan, tilt), and the direction of the monitor is changed toward the user based on the above-mentioned tracking result.
The character size displayed on the monitor 51B is changed according to the distance to the user.

The processing in the local processor described in the above embodiments is all performed by software using the CPU. All the result displays are displayed in real time in windows opened on the monitor of the workstation WS.

[0110]

As described above, the moving object extracting apparatus of the present invention can be expected to have a great effect in practical use such as realizing an apparatus for simultaneously extracting the contour and movement of the moving object from the input image. .

[Brief description of drawings]

FIG. 1 is a block diagram showing an overall schematic configuration of an embodiment of the present invention.

FIG. 2 is a flow chart schematically showing the overall flow of processing of the moving object extraction apparatus shown in FIG.

FIG. 3 is a diagram showing a processing procedure when the edge search is performed in a two-step process.

FIG. 4 is a diagram showing a state in which four contour tracking units arranged in a circumscribed rectangle are initially arranged.

FIG. 5 is a diagram showing a state in which eight contour tracking units arranged in a circumscribed rectangle are initially arranged.

FIG. 6 is a diagram showing a manner in which a contour tracking unit is arranged based on a rotation angle about a center of gravity of a contour.

FIG. 7 is a diagram showing an edge search process in a specific direction by each contour tracking unit and a relationship between each contour tracking unit.

FIG. 8 is a diagram for explaining calculation of position information when there are four contour tracking units.

FIG. 9 is a diagram for explaining calculation of position information when there are eight contour tracking units.

10 is a block diagram showing a configuration of a local module shown in FIG.

FIG. 11 is a diagram for explaining determination of each size of a contour model.

FIG. 12 is a diagram for explaining a search for an edge area in a change area of each sample point.

FIG. 13 is a diagram showing a result of contour tracking in which a horizontal motion is performed using an actual image.

FIG. 14 is a diagram showing a result of extracting a contour of a face in real time.

FIG. 15 is a diagram showing a configuration of each contour tracking unit.

FIG. 16 is a diagram for explaining a restraining force realized by using a plurality of virtual springs and virtual dampers.

FIG. 17 is a block diagram showing a schematic configuration of a camera that is automatically controlled following the movement of a moving object region.

FIG. 18 is a block diagram showing a schematic configuration of a workstation equipped with the moving object extracting device shown in FIG. 17.

19 is a perspective view showing the external appearance of the workstation shown in FIG.

FIG. 20 is a schematic diagram for explaining B spline snake in a conventional example.

[Explanation of symbols]

 1 Image Input Section 2 Tracking Contour Determining Section 3 Contour Tracing Section Group 4 Host Computer 5 Image Output Section 6 Position Determining Section 8 Camera Platform Control Section 9 Central Processing Unit (CPU) 11 Camera 13 A / D Converter 21 Image Memory 23 difference circuit (edge extraction circuit) 25 binarization circuit 27 labeling circuit 29 circumscribing rectangle calculation circuit 31 contour tracing unit 41 contour tracing unit management unit 43 binding force calculation unit 45 total contour calculation unit 47 three-dimensional position calculation unit 51 output monitor 61 Mouth position determination unit 63 Both eye position determination unit 65 Nose position determination unit 71 Bus interface 73 Address decoder 75 Local processor 77 Window controller 79 Window memory 91 Object extraction processing unit 93 Display change calculation unit 95 Object tracking calculation unit 97 Monitor turntable control Circuit 98 Camera pan head control circuit 99 Camera control Road 101 Graphics Controller

Claims (4)

[Claims] 1. A contour extracting means for detecting a contour of a moving object from the supplied image information and extracting the contour thereof, and a plurality of contour tracing means provided for the contours extracted by the contour extracting means. A moving object extracting device comprising moving object extracting means for obtaining the contour shape and movement of the object from the shapes and movements of the plurality of contour tracking means. 2. A contour extracting means for detecting a contour of a moving object from the supplied image information and extracting the contour thereof, and the contours extracted by the contour extracting means are provided as targets, and the respective specific directions of the contour are provided. Of a plurality of contour tracking means that respond only to the movement of a plurality of contour tracking means and at least two contour tracking means that respond to a movement of a different specific direction among the plurality of contour tracking means An animal body extraction device comprising: an interpolation means for interpolating information; and an animal body extraction means for obtaining the contour shape and movement of the object from the shapes and movements of the plurality of contour tracking means. 3. A contour extraction means for detecting a contour of a moving object from the supplied image information and a contour thereof, and a contour extracted by the contour extraction means as a target. Of a plurality of contour tracking means and at least two contour tracking means of the plurality of contour tracking means which react to a movement in different specific directions are paired, and the movements of the contour tracking means are mutually restricted. And a moving body extracting means for obtaining the contour shape and movement of the object from the shapes and movements of the plurality of contour tracking means. 4. A contour extracting means for detecting a contour of a moving object from the supplied image information and extracting a contour thereof, and a plurality of contour tracking means provided for the contour extracted by the contour extracting means. An arbitrary point setting means for setting an arbitrary point within an area surrounded by the plurality of contour tracing means; an arbitrary point set by the arbitrary point setting means and each contour tracing means of the plurality of contour tracing means. The present invention is characterized by further comprising: restraining means provided between the restraining means for restraining the movement of the contour tracing means, and a moving body extracting means for obtaining the contour shape and movement of the object from the shapes of the plurality of contour tracing means and their movements. Animal body extraction device.