JP3401512B2 - Moving object tracking device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving object tracking device, and more particularly, to a moving object tracking device for tracking a moving object using multi-viewpoint images obtained asynchronously.

[0002]

Stereo measurement, which obtains depth information in a scene from a plurality of images obtained from different viewpoints, occupies an important position in computer vision research whose restoration of three-dimensional information from an image is one of the main subjects. ing.

Correspondence between images is particularly important in stereo measurement, and feature amounts are appropriately selected to reduce erroneous correspondence (Marr D. And Poggio
TA theory of human stereo vision. In Roy. Soc. L
ondon, pp. 301-328.1979), a method for effectively utilizing more viewpoints (Ramakant Nevatia. Depth
measurement by motion stereo. Computer Graphics an
d Image Processing vol. 5, pp.203-214, 19
76) has been developed. These are mainly for static scenes, and there are already many reports about the error of the stereo method in this case (Jeffrey J. Rod
riguez and JK Aggarwal. Stochastic Analysis of
stereo quantization error. IEEE Pattern Anal. Mach
ine Intell., Vol.12, No.5, pp.467-470,
5 1992).

[0004] In the research targeting scenes with motion,
Robot navigation based on stereo observation (Larr
y Matthies and Steven A. Shafer. Error Modeling in
stereo navigation. IEEE Rebotics and Automation,
vol. RA-3, No.3, pp.239-248, 1987),
Tracking known shapes with one camera (Akio Kos
aka and Goichi Nakazawa. Vision-based motion track
ing of rigid objectsusing prediction of uncertaini
es. In Proc. of International Conferenceon Robotic
s and Automation, pp.2637-2644,195
5), tracking of known shape by stereo measurement (Ge
m-Sun J. Young and Rama Chellappa. 3-d motion esti
mation using a sequence of noisy stereo images: Mo
dels, estimation, and Uniqueness results. IEEE Pat
tern Anal. Machine Intell., Vol.12, No.8, pp.7
35-759, 1990), Tracking of multiple objects based on stereo observation (Jae-Woong Yi and Jun-Ho Oh. Recursive
resolving algorithm for multiple stereo and motion
matches. Image and Vision Computing, Vol.15, pp.
181-196, 1997) and the like have been proposed. These problem settings, which track multiple objects while associating image features with the object model of interest, are "motion associations."
Known as (Motion Correspondence) (Ingem
ar J. Cox. A review of statical data association te
chniques for motion correspondence. International
Journal of Computer Vision, Vol.10: 1, pp.53
-66, 1993).

[0005]

By the way, all of the above-mentioned conventional methods are based on the premise that observations from a plurality of viewpoints are made simultaneously at fixed intervals, or that they are still scenes (equivalent to simultaneous observations). If the observations are made simultaneously,
Since the same physical characteristics will be observed, it becomes easy to associate the movements.

However, even in the conventional method of performing simultaneous observation, "movement correspondence" between different times is still necessary in order to track an object in time series.

Further, since many conventional systems using multi-view images are based on the premise that observations are simultaneously made between the viewpoints in order to acquire position information at the time of tracking, they are obtained at each viewpoint. When processing an image, there is a problem that the overall processing speed is limited by the slowest processing. Further, in such a system, means for synchronizing the viewpoints in addition to the exchange of image information is required separately. The problems of the conventional system based on these synchronizations become more remarkable as the number of viewpoints used increases.

Therefore, the present invention has been made in order to solve such a problem, and provides a moving object tracking device capable of performing highly accurate asynchronous tracking by integrating information observed at different times. Is to provide.

A further object of the present invention is to provide a moving object tracking device capable of efficiently tracking an object that changes in a simplified system configuration.

[0010]

A moving object tracking device according to claim 1 is a moving object tracking device for tracking a moving object moving in a scene, and a plurality of moving object tracking devices for photographing the scene from a plurality of viewpoints. And a plurality of observing means provided corresponding to each of the plurality of photographic means and operating asynchronously independently of each other. Each of the plurality of observing means extracts a characteristic point from the image received from the corresponding photographing means, and a characteristic point extracting means for extracting a characteristic point from the image received from the corresponding photographing means and at least one attribute value unique to the moving object to be tracked. The attribute value extracting means for extracting is associated with at least one tracking target moving object and the extracted feature points, and is associated with the first observation information including the attribute value regarding the associated feature points. And feature point associating means for outputting the second observation information regarding the feature points that have not been obtained. The moving object tracking device integrates, using a Kalman filter, first observation information that is asynchronously supplied from a plurality of observing means that have extracted feature points from among a plurality of observing means, and thus moves at least one tracking target. The apparatus further includes a tracking unit that predicts the state of the object based on the tracking model of the moving object. The tracking means includes attribute value extraction means for extracting an attribute value from the supplied first observation information,
Includes state estimation means for estimating the state of the moving object that is the tracking target based on the extracted attribute value, and tracking model determination means for determining the tracking model of the moving object according to the estimated state of the moving object. . The moving object tracking device predicts by performing a Kalman filter update process in chronological order with the second observation information that is asynchronously and sequentially supplied from the plurality of observation means that have extracted the feature points among the plurality of observation means. Further, there is further provided a finding unit that calculates a trajectory, detects an estimated position of a moving object that is a tracking target newly based on the calculated predicted trajectory, and transmits the estimated position to the tracking unit as an initial found position. The feature point associating unit detects a moving object as a tracking target corresponding to the extracted feature point, based on the predicted state of at least one moving object as a tracking target transmitted from the tracking unit.

In the moving object tracking device according to a second aspect, the attribute value according to the first aspect is at least one of backrest information, action information and color information of the moving object.

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION A moving object tracking device 1 according to an embodiment of the present invention will be described with reference to FIG. In the following description, the same reference numerals in the drawings indicate the same or corresponding parts. FIG. 1 is a block diagram showing the overall configuration of a moving object tracking device 1 according to an embodiment of the present invention.
Referring to FIG. 1, moving object tracking device 1 includes camera 2 #.
1, 2 # 2, ..., 2 # n, observation units 4 # 1, 4 # 2 ,.
4 # n, a discovery unit 6 and a tracking unit 8.

Observation units 4 # 1, 4 # 2, ..., 4 # n are provided corresponding to cameras 2 # 1, 2 # 2, ..., 2 # n, respectively (hereinafter collectively referred to as cameras). 2 and observation section 4, respectively). Observation units 4 # 1, 4 # 2 ... 4 # n,
Each of the finding unit 6 and the tracking unit 8 can operate independently of each other. For example, these are composed of different computers, and each computer is connected to the local area network LAN.

In the figure, the symbol A0 indicates the observation section 4 to the tracking section 8.
The observation information of the corresponding point (tracking target) sent to
Reference numeral 1 denotes the observation information of uncorresponding points (points that cannot be traced to the tracking target) transmitted from the observing unit 4 to the discovering unit 6 by the symbol A2.
Is the predicted position information transmitted from the tracking unit 8 to the observation unit 4, the symbol A3 is the position information (initial value) of the new person transmitted from the discovery unit 6 to the tracking unit 8, and the symbol A4 is the tracking unit. 8
The position information (after updating) transmitted from the to the discovery unit 4 is represented.

The observing unit 4 performs a feature extraction process based on the input image obtained from the corresponding camera 2. Each observation section 4
Works independently. The feature amount (the center of gravity point and the distance conversion value) obtained by the observation unit 4 is associated with the tracking target based on the predicted position information A2 transmitted from the tracking unit 8 described later, and then the observation time information is obtained. It is sent together with the tracking unit 8. The feature that could not be matched is the observation information A of the uncorresponding point.
1 is transmitted to the discovery unit 6.

The finding section 6 detects a person (new person) newly appearing in the scene by using the transmitted observation information A1 of uncorresponding points. The detection result of the new person, that is, the position information A3 of the new person is acquired by the tracking unit 8
Sent to. As a result, the new person is added as a new tracking target. Then, the transmitter 8 starts tracking.

The tracking unit 8 updates the position information of the person using the Kalman filter with the position information A3 of the new person as the initial value and the observation information A0 as the input value, and further predicts the position based on the observation model. The predicted position information A0 is transmitted to the observation unit 4. The location information (after updating) A4 is transmitted to the finding unit 6 as described later.

The human body model used in the embodiment of the present invention will be described with reference to FIG. Figure 2
It is a figure for explaining a human body model used in an embodiment of the invention. In the figure, X, Y, and Z indicate the three axes of the world coordinate system. The person MAN is modeled by an elliptic cylinder h. The center axis Xh of the person model h is the rotation axis of the person, and the normal axis (the axis in the minor axis direction perpendicular to the rotation axis Xh)
The angle r between the X-axis and the X-axis is the posture angle of the person. It is assumed that the rotation axis Xh of the person MAN is perpendicular to the floor surface (the plane formed by the X axis and the Y axis). The camera 2 is arranged perpendicular to the rotation axis Xh of the person, that is, the Z axis.

Next, the observation section 4 shown in FIG. 1 will be described. FIG. 3 is a diagram for explaining the outline of the configuration of the observation unit 4. Referring to FIG. 3, the observation unit 4 includes an area dividing circuit 10, a pixel value calculating circuit 12, a center of gravity point selecting circuit 14,
And a feature point correspondence circuit 16.

The area division circuit 10, the pixel value calculation circuit 12, and the center-of-gravity point selection circuit 14 perform the feature extraction processing based on the inputted person image. The feature point association circuit 16 associates the extracted feature points with a tracking target (model).

The area dividing circuit 10 divides the input image into a person area and a background area. For example, "Image segmentation for person tracking by hierarchical adaptation based on continuous images" Computer Vision and Pattern Recognition (CVPR'9
8), Journal of IEEE Computer Society, p. 911 to
In 916, a method of area division is described.

More specifically, the mask image is generated by binarizing the difference between the image of the current frame captured by the camera and the images of the preceding and following frames, and superimposing some frames of the difference image (that is, Extract a rough body region by coarse division). When the size of the mask image is within a certain range, further estimate the distribution of the parameter by luminance information, color information, or pixel information, texture information and any combination thereof, and further by the estimated parameter distribution, The probability that the moving object region exists for each pixel on the newly obtained image is calculated (that is, precise division is performed). As a result, the person area is cut out from the input image captured by the camera.

Subsequently, the pixel value calculation circuit 12 performs distance conversion on the obtained person image. Specifically, a pixel value indicating the shortest distance from each of the pixels forming the person area to the boundary of the person area is calculated. The center-of-gravity point selection circuit 14 selects a point having the maximum pixel value in the person area as a center of gravity (that is, a feature point) of the person area. The pixel value at the center of gravity is used as the distance conversion value. The center-of-gravity point and the distance conversion value of the center-of-gravity point (the center-of-gravity point distance conversion value) are used as the feature amount.

Specifically, the area dividing circuit 10 outputs a binarized image as shown in FIG. The pixel value calculation circuit 12 generates a distance conversion image shown in FIG. 4B for the binarized image shown in FIG. Figure 4
In (b), a pixel with a larger distance conversion value is represented in black,
Pixels with smaller distance conversion values are shown in white. The color of the pixel becomes darker as the distance from the contour of the human body increases. The center-of-gravity point selection circuit 14 selects the pixel corresponding to the symbol “X” in FIG. 4B as the center of gravity point. The detection of the center of gravity by the distance conversion is characterized by being less susceptible to changes in the pose of a person.

Next, the feature point associating circuit 16 for associating the extracted barycentric point with the already-discovered tracking target will be described. FIG. 5 is a diagram for explaining the feature point association processing. Symbol 2 in the figure
0 # 1, 20 # 2, ..., 20 # l, ..., 20 # m ,.
Each of 20 # n has a camera 2 # 1, 2 # 2, ..., 2
The image planes obtained by # 1, ..., 2 # m, ..., 2 # n are respectively represented (hereinafter, generically referred to as image plane 20).

At the time ta, the time tb is obtained by the camera 2 # l.
It is assumed that the observation is performed by the camera 2 # m (note that ta <tb). At time ta,
It is assumed that there are two persons h0 and h1 and the person h2 first appears in the scene at time tb.

The symbol x attached to the image plane 20 indicates the center of gravity detected in each image. In this case, time t
Two barycentric points (feature points) are detected from the image plane 20 # l at a, and three barycentric points (feature points) are detected from the image plane 20 # m at time tb.

For each of the persons h0 and h1, an inquiry is made to the tracking unit 8 to obtain the predicted position at time tb based on the observation result up to time ta. As will be described later, in the tracking unit 8, the motion of the person is assumed to be a uniform velocity motion, and the predicted position of the person hj at a certain time t is represented by a two-dimensional Gaussian distribution. The position of the person hj in the world coordinate system at time t is _defined as the position X _{hj, t} , the average of the two-dimensional Gaussian distribution is represented as / X _{hj, t} , and the covariance matrix is represented as / S _{hj, t} .
The average and covariance matrix are transmitted from the tracking unit 8 as the predicted position information A0.

In the formula, / X _{hj, t} is _expressed as a symbol X _{hj, t with} a "-" added, and / S _{hj, t} is denoted by a symbol S _{hj, t} on ". It is expressed as "-".

Predicted position distribution N (/ X _{hj, t} , / S _{hj, t} )
Is weakly perspectively projected onto the image 20 # i (i = 1, 2, ..., N), the one-dimensional Gaussian distribution n (/ x _{hj, t, i} , / s _{hj, t, i} is obtained, which indicates the existence probability of a person in the image 20 # _i, where the symbol x is the person position X in the world coordinate system.
Is projected on the image plane, the symbol / x is the average / X in the world coordinate system projected on the image plane, the symbol / s
Represents the projection of the covariance matrix / S in the world coordinate system on the image plane. In the formula, /
x is expressed as a symbol x with "-" added,
/ S is expressed as a symbol s with "-" added.

[0037]

[Equation 1]

The feature point that maximizes the probability expressed by equation (1) is taken as the observed value corresponding to the person hj at the observation time, and the feature point is labeled hj. The observation information A0 of the labeled feature points is transmitted to the tracking unit 8. However, for the feature points that can be associated with multiple people,
It is determined that occlusion has occurred at the time of observation, and transmission is stopped.

After these processes, the feature points that are not associated are assumed to belong to an unknown person, that is, a new person, and the position and time are transmitted to the finding unit 6 as uncorresponding observation information A1. It Note that in FIG. 5, the observation information A1 about the person h2 is transmitted to the discovery unit 6.

Next, the tracking unit 8 shown in FIG. 1 will be described. The tracking unit 8 updates the position / direction of the tracking target person based on the observation information A0 sent from each of the observation units 4.

FIG. 6 is a diagram for explaining the outline of the configuration of the tracking section 8 shown in FIG. Referring to FIG. 6, tracking unit 8 includes position estimation circuit 22 and direction angle estimation circuit 24.
including. The position estimation circuit 22 estimates the predicted position based on the position of the center of gravity obtained from the observation unit 4, and outputs the estimation result to the direction estimation circuit 24. The direction angle estimation circuit 24 follows the estimation result output from the position estimation circuit 22,
The direction angle is estimated based on the converted value of the distance to the center of gravity obtained from.

First, the position estimation processing in the position estimation circuit 22 will be described in detail. The position of the person being tracked is updated by using the feature information (centroid position) associated with each observation unit 4. FIG. 7 is a diagram for explaining the position estimation process. Referring to FIG. 7, the camera 2
The distance between #i and the image plane 20 # _i is the distance l _i , and the camera 2 #
The distance between i and the person hj is referred to as a distance L _{hj, i} . The angle formed by the epipolar line and the Y axis is denoted by w _{hj, i} .

In the position estimation processing, it is assumed that the person is moving at a constant velocity. The state of the person hj at time t is represented by equations (2) and (3) on world coordinates (X, Y). However, the initial state is determined by the information of the new person (model) transmitted from the discovery unit 6. The “′” attached to the matrix represents transposition.

Let ∧X _{hj, t-1 be} the estimated value of the person position X _hj at time (t-1), and let ∧S _{hj, t-1} be the estimated value ∧X.
_{Assuming that hj, t-1 is} the covariance matrix, the state at time t is given by equation (4).
And equation (5). In the formula, ∧X
_{Express hj, t-1} as a symbol X _{hj, t-1} with "∧" added, and ∧S _{hj, t-1} on the symbol S _{hj, t-1} with "∧" It is expressed as the one with.

The transition matrix F is expressed by equation (6) (Δt: t-1 → t). The symbol Q represents the covariance matrix at the transition.

[0046]

[Equation 2]

Here, it is assumed that the observation section 4 # i has made the first observation. This observation can be expressed by equations (7) to (9) based on the position information sent from the observation unit 4 # i.

H = [1 0 0 0] (7)

[0049]

[Equation 3]

The symbol C _i indicates the position of the camera, and the symbol R indicates
_{hj, t, i} represents the clockwise rotation of the angle w _{hj, t, i} formed by the epipolar line and the Y axis. The symbol e represents an observation error, and has an average of 0 and a standard deviation σ _{hj, t, i} . The standard error σ _{hj, t, i} is considered to increase as the distance of the camera increases, and is expressed as in equation (9).

Here, the camera position C _i and the person (X
_Since the distance L _{hj, t, i} from ( _{hj, t} ) is an unknown number _, the predicted position of X _{hj, t} / the distance calculated by X _{hj, t} / L _{hj, t, i} is used as an approximate value ( In the formula, / L _{hj, t, i} is the symbol L
It is expressed as _{hj, t, i} with "-" added.

In the observation equation of equation (8), the left side represents the observation information, and the right side represents the result of projecting the person position on the image.

The position estimation circuit 22 constitutes a Kalman filter by the above observation model and updates the state of the person hj.

[0054]

[Equation 4]

For each camera, equations (10) and
The update process according to (11) is performed to predict the state.
The state prediction of the person hj at time (t + 1) is the average.
/ X _{hj, t + 1}, The covariance matrix / S_{hj, t + 1}Gaussian distribution
Given in. The result of the state prediction meets the request of the observation unit 4.
Correspondence of feature points as described above
Used for. A person who has moved outside the detectable range of the camera
Delete the physical model and stop tracking the person.

Next, the processing of the direction angle estimating circuit 24 will be described. In the direction angle estimation circuit 24, the posture angle r of the person
(See FIG. 2) is estimated using the centroid distance conversion value that does not cause occlusion. This estimation is performed based on the human body observation model (ellipsoidal model) shown in FIG. Figure 8
With reference to, the center-of-gravity point distance conversion value is used with the width of the ellipsoid on the image captured by the camera 2 as s. Assuming a weak perspective transformation, the observed center-of-gravity point distance conversion value s follows Equation (12), where θ is the angle formed by the optical axis and the normal to the rotation axis of the ellipsoidal model (person). Here, the formula (1
The symbol L in 2) indicates the distance from the rotation axis of the ellipsoidal model to the camera, and the symbols A and B are constants. Assuming that the observation involves a Gaussian error, the probability P (s | θ) that the transformed value s of the barycentric point distance is observed is given by the equation (13).
It is expressed as. The constants A and B and the variance σ of the error
_s is determined in advance based on the learning data.

[0057]

[Equation 5]

The posture of the human body is the Euler angle (a, e, r)
Can be expressed by Where r is the azimuth angle (posture angle)
, A represents an azimuth angle, and e represents an elevation angle.
Since the azimuth angle a and the elevation angle e are determined by the rotation axis direction of the human body, only the rotation angle r around the rotation axis is an unknown number.

The normal vector N of the human body is expressed by equation (14). Therefore, the optical axis vector C of the camera 2 # i
The angle θ _i (r) formed between the normal vector N of the human body and the normal vector N of the human body is expressed by Expression (15).

N = R _Z (a) R _Y (e) R _X (r) e (14) θ _ci (r) = cos ⁻¹ · N ^T · C (15) In the formula (14), , R _Z , R _Y , and R _X represent rotation matrices about the Z, Y, and X axes, respectively, and e _Z is Z.
A unit vector in the axial direction is shown. Also, equation (15)
In, the symbol N ^T indicates a transposed vector of the normal vector N.

The probability P that the set W (s ₁ , s ₂ , ..., Sn) of the barycentric point distance conversion values is observed by the _n cameras 2 # ₁ to ₂ # _n is given by the equation (13). It is expressed by equation (16). A value r that maximizes the probability P (W | r) in Expression (16) is used as the estimated value of the posture angle.

[0062]

[Equation 6]

The azimuth angle estimation is updated every time observation is performed, like the position estimation described above. Note that the observation model shown in FIG. 8 has the input as the characteristic information (the center-of-gravity point distance conversion value s) transmitted from the observation unit 4 _, and has the state r _{hj, t} expressed by the equation (17). The presence / absence of occlusion is determined at the time of associating feature points, and is not used as observation information when occlusion occurs.

Next, an outline of the finding section 6 shown in FIG. 1 will be described. The finding unit 6 detects a person (new person) who newly appears in the scene, and adds a corresponding model to the tracking unit 8.

Since the observation information is acquired asynchronously, ordinary stereo correspondence cannot be applied as it is. Therefore, the following correspondence (discovery) method using time series information is used.

First, of the observation information of uncorresponding points sent from each of the observing units 4, the observation information at different four times is selected one by one (denoted by γ). Observation time t1, t
The observation viewpoints are C ₁ , C ₂ , C ₃ , and C ₄ for 2, 2, and t ₄ , respectively. With observation information as input,
The above-described Kalman filter update processing (equation (8)) is performed. However, the initial variance / S _{hj, t} = 0.

By this operation, the predicted orbits in the four observations can be obtained. Using this predicted trajectory, position estimation is performed using equation (4) (where Δt = t4-ti: i
= 1, 2, or 3). The set of position estimation results is [∧
_{X t1, ∧X t2, ∧X t3} , and ∧X _{t 4].}

As described above, in the observation formula of the formula (7),
The left side shows the observation information, and the right side shows the result of projecting the person position on the image. The error evaluation function f (γ) using the Mahalanobis distance shown in Expression (17) is defined as the difference between the observation information and the result of projecting the person position on the image.

[0069]

[Equation 7]

A combination in which the value of the evaluation function shown in Expression (17) is within a certain threshold is a set of feature points belonging to a new person, and the estimated position at the latest observation time (here, t4) is the initial found position. It transmits to the tracking unit 8.

Next, the effectiveness of the present invention will be clarified.
Therefore, we conducted the following two simulation experiments.
It was Figure 9 shows a simulation for checking the accuracy of position tracking.
FIG. 6 is a diagram for explaining a solution experiment. In Figure 9
In the simulation experiment shown, two cameras (2 #
1, 2 # 2) and a circular orbit with a radius of 100 has a cycle of 10 ×
t _fThe object h0 to be tracked which moves in a uniform circular motion was observed. t_f
Is the shooting interval of the camera. The Kalman filter has
A model of constant velocity linear motion is given. Observation assumes parallel projection
However, in the experiment, each of the two cameras 2 # 1 and 2 # 2
There are four types of angles θ formed by these optical axes (0 °, 30 °, 45
, 90 °). Under each condition,
The time difference Δt between the camera 2 # 1 and the camera 2 # 2 is set to 0.
To t_f/ 2, and t from the observation of camera 2 # 1_f
Later (for the direction parallel to the shooting surface of camera 2 # 1)
The position prediction error was recorded.

FIG. 10 is a diagram for explaining the experimental results for FIG. In FIG. 10, the horizontal axis represents the deviation Δt of the photographing time, and the vertical axis represents the prediction error. As shown in FIG. 10, particularly when the angle θ formed by the optical axes is 0 °, 30 °, and 45 °, the prediction error decreases as the shooting interval increases, and the prediction error at Δt = t _f / 2. Has taken the minimum value. From this, it can be seen that when the motion of the tracking target object deviates from the motion model that is the premise of the Kalman filter, a higher followability can be obtained by performing the asynchronous observation, in particular, as compared with the synchronous observation.

FIG. 11 is a diagram for explaining a simulation experiment for clarifying the ability to estimate the movements of a plurality of persons. In the simulation experiment shown in FIG. 11, five cameras 2 # 1 to 2 # 5 are arranged. Each camera is connected to one computer that constitutes the corresponding observation unit. Image processing is performed on these computers. The processing speed is about 1-2 frame / sec.

Each computer is connected to the local area network LAN and has its internal clocks synchronized with each other. Two computers (not shown) that respectively configure the finding unit 6 and the tracking unit 8 are local area network LA.
It is connected to N.

Each camera has been calibrated in advance, and the calibration information of each observing unit 4 is:
It is transmitted to the discovering unit 6 and the tracking unit 8 together with the observation information (observation time, characteristic points). In the experiment, position tracking of two persons h0 and h1 was performed. As shown in FIG. 11, two persons h0 and h1 appear in the scene 55 in order.

12 to 16 are views for explaining the experimental results for FIG. In the simulation experiment shown in FIG. 11, the change over time of the characteristic points obtained by each camera is shown. FIG. 12 shows the camera 2 # 1
13 corresponds to the camera 2 # 2, FIG. 14 corresponds to the camera 2 # 3, FIG. 15 corresponds to the camera 2 # 4, and FIG. 16 corresponds to the camera 2 # 5. 12 to 16, the horizontal axis represents time, the vertical axis represents the tracking position, and the symbol "+" represents the feature point that can be associated with the predicted position.

Referring to FIGS. 12 to 16, at the beginning of tracking,
Since no characteristic points are observed except for the camera 2 # 1, the person is not found. As the first person approaches the center of the experimental environment (scene 55), feature points are obtained from other viewpoints (cameras 2 # 3, 2 # 4, 2 # 5) and tracking is started (about from the start). 20 seconds have passed). Similarly, tracking of the second person is started about 37 seconds after the movement of the second person. From the above experimental results, it can be said that the present invention makes it possible to track a plurality of persons.

In the above-mentioned embodiment, the movement of the person is tracked based on the feature points in the images which are photographed asynchronously from a plurality of viewpoints. However, the attribute value peculiar to the person is extracted next,
An embodiment will be described in which the extraction result is output as observation information, the state of the corresponding person is detected based on the extracted attribute value, and the tracking model of the person is switched according to the detected state. For example, since it is considered that there is a very low possibility that a person moves while seated, the reliability of the position estimation value is increased.

FIG. 17 is a diagram showing an outline of the observation section and the tracking section of the second embodiment of the present invention, and corresponds to FIGS. 3 and 6 of the first embodiment. In FIG. 17, an area dividing circuit 10, a pixel value calculating circuit 12, and a center point selecting circuit 14
Is the same as in FIG. 3, but the observation unit 4 of this embodiment
A silhouette height detection circuit 17 is newly provided. The silhouette height detection circuit 17 detects the height of a person from the binarized image binarized by the area division circuit 10.

FIG. 18 is a diagram showing a binarized image when a person is standing and seated. As shown in FIG. 18A, the silhouette height detection circuit 17 detects the height T1 of the binarized image and the height T of the binarized image shown in FIG. 18B.
2 is detected. For example, if a person is standing, T
1 is 180 cm and T2 is 1 if a person is seated
It is detected as 40 cm. The detection output of the silhouette height detection circuit 17 is given to the feature point correspondence circuit 16.

As described in the first embodiment, the feature point associating circuit 16 makes correspondence between the barycentric point extracted by the barycentric point selecting circuit 14 and the already-discovered tracking target. Based on the detection output of the silhouette height detection circuit 17, the information obtained by measuring the height of the corresponding person is included in the observation information and output to the tracking unit 8.

On the other hand, tracking section 8 includes a height information extraction circuit 30, a back conversion circuit 32, and a state estimation circuit 34, as shown in FIG. 17, in addition to the configuration shown in FIG. The height information extraction circuit 30 extracts height information from the observation information sent from the observation unit 4 to the tracking unit 8 and supplies it to the backrest conversion circuit 32. The backrest conversion circuit 32 converts the height information extracted from the observation information into a backrest.

If the backrest changes from 180 cm to 140 cm, for example, the state estimating circuit 34 estimates that the person is in a sitting state from a standing state.
On the contrary, if the backrest changes from 140 cm to 180 cm, it is estimated that the person is standing up.

The observation unit 4 and the tracking unit 8 operate in the same manner as in FIG. 6 described above, and the tracking unit 8 estimates the predicted position based on the position of the center of gravity point included in the observation information obtained from the observation unit 4, and The direction angle is estimated based on the estimation result, and a specific person among a plurality of persons is tracked.

Among the observation information sent from the observing unit 4 to the tracking unit 8, the height information is extracted by the height information extracting circuit 30, converted into the ignorance by the displacing circuit 32, and the state estimating circuit 34 exemplifies the person. It is estimated whether the person is standing or seated. Tracking unit 8
Then, the person to be tracked is switched according to the estimated state.
That is, if the person being tracked is seated,
The person no longer moves, so he tracks other people.

In the above description, the state in which the person is standing and the state in which the person is seated are estimated. However, the present invention is not limited to this. It is also possible to estimate action information such as a state of stopping from a state of being. In that case,
It may be estimated that the person is moving when the speed of the person becomes a certain speed or more, and it is estimated that the person is stopped when the speed becomes zero. In this case, the above equations (4) to (6) are expressed by the following equation (1
8) to (20). These expressions are time t
_It shows a case where the state of the person h ₁ at time t _b is predicted from the information up to the previous observation made in a.

[0087]

[Equation 8]

In the above equation, Q _x and Δt represent the fluctuation of the motion between Δt, and it is considered that the magnitude thereof changes depending on the current motion state. In the above-described embodiment, it is possible to change this value such that the walking width is large, the walking width is medium, and the sitting width is small. In this way, by changing the motion model, the reliability of the estimation result can be evaluated according to the motion state of the object.

In the second embodiment shown in FIG. 17, the state estimation of whether the person is standing or seated and the state of whether the person is stopped or has started moving are estimated. . However, in associating the feature of each observed image with the tracking model, the attribute value peculiar to the target object that can be observed from the image can be used instead of the position information. Here, the unique attribute value is not limited to the backrest of the target object or the action information described above, and the color of clothes may be considered.

FIG. 19 is a block diagram showing such an embodiment. The observation unit 4 shown in FIG. 19 is provided with a silhouette height / color detection circuit 18 in place of the silhouette height detection circuit 17 shown in FIG.
7, instead of the height extraction circuit 30 and the backrest conversion circuit 32,
A height / color information extraction circuit 36 and a color conversion circuit 38 are provided. The silhouette height / color detection circuit 18 of the observation unit 4 detects the height of the person as described with reference to FIG. 17, and at the same time, the chest portion (FIG. 20) of the image divided into regions as shown in FIG. The average color of the shaded area) is also detected and given to the feature point correspondence circuit 16.

On the other hand, the height / color information extraction circuit 3 of the tracking unit 8
The reference numeral 6 extracts height information and color information from the observation information, the backrest / color conversion circuit 38 converts backrest and color from the height information and color information, and the state estimation circuit 34 estimates the state of the person. . In this case, the color information S _tn obtained in the previous observation and the color information S _{tn + 1} obtained in the current observation are expressed by the following equation.

[0092]

[Equation 9]

As described above, by using the attribute values peculiar to the target object such as the backrest and the color information as the observation information, there is an advantage that the target object is not confused even when the target object is approaching. is there.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

[0095]

As described above, according to the moving object tracking device of the present invention, the object tracking by the Kalman filter is performed based on the image features in the images asynchronously taken from a plurality of viewpoints. It is possible to follow the accuracy.

Further, by making the observation independently for each viewpoint, it becomes possible to simplify the system implementation and improve the processing efficiency.

Furthermore, the local area network LA
By connecting a camera to each of the computers connected to N and configuring the two-dimensional features obtained from the input image to communicate via the local area network LAN, it is possible to easily add or delete observation viewpoints. Becomes

Further, the state of the corresponding moving object is detected based on the attribute values such as the backrest information, the action information, and the color information of the moving object, and the motion model used for tracking is switched according to the detected state. More precise tracking can be performed.

[Brief description of drawings]

FIG. 1 is a block diagram showing an overall configuration of a moving object tracking device 1 according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining a human body model used in the embodiment of the present invention.

FIG. 3 is a diagram for explaining an outline of a configuration of an observation unit 4.

FIG. 4 is a diagram for explaining a feature extraction process in an observation unit.

FIG. 5 is a diagram for explaining a feature point association process.

FIG. 6 is a diagram for explaining an outline of the configuration of a tracking unit 8 shown in FIG.

FIG. 7 is a diagram for explaining position estimation processing.

FIG. 8 is a diagram for explaining processing in the direction angle estimation circuit 24.

FIG. 9 is a diagram for explaining a simulation experiment for confirming the accuracy of position tracking.

FIG. 10 is a diagram for explaining an experimental result for FIG. 9.

FIG. 11 is a diagram for explaining a simulation experiment for clarifying the ability to estimate the movements of a plurality of persons.

FIG. 12 is a diagram for explaining an experimental result for FIG. 11.

FIG. 13 is a diagram for explaining an experimental result for FIG. 11.

FIG. 14 is a diagram for explaining an experimental result for FIG. 11.

FIG. 15 is a diagram for explaining an experimental result for FIG. 11.

FIG. 16 is a diagram for explaining an experimental result for FIG. 11.

FIG. 17 is a diagram showing aspects of an observation unit and a tracking unit according to the second embodiment of the present invention.

FIG. 18 is a diagram showing a binarized image when a person is standing and when a person is seated.

FIG. 19 is a diagram showing aspects of an observation unit and a tracking unit according to the third embodiment of the present invention.

FIG. 20 is a diagram showing a state in which an average color of a chest portion in an image obtained by region division is detected.

[Explanation of symbols]

1 Moving Object Tracking Device, 2 # 1-2 # n Cameras, 4 #
1-4 # n Observation part, 6 Discovery part, 8 Tracking part, 10
Area dividing circuit, 12 pixel value calculating circuit, 14 barycentric point selecting circuit, 16 feature point associating circuit, 17 silhouette height detecting circuit, 18 silhouette height, color detecting circuit,
22 position estimation circuit, 24 direction angle estimation circuit, 30 height information extraction circuit, 32 back conversion circuit, 34 state estimation circuit, 36 height and color information extraction circuit, 38 back support,
Color conversion circuit.

Continuation of the front page (56) References JP-A-5-205052 (JP, A) Daiki Mori, Akira Utsumi, Atsushi Otani, Masahiko Taniuchida, Proposal and construction of a multi-viewpoint system capable of acquiring diverse human information , IEICE Technical Report, Japan, The Institute of Electronics, Information and Communication Engineers, March 24, 1999, VOL. 98 No. 684, pp. 41-48 (Mve98-103) Akira Utsumi, Atsushi Otani, A Study on Human Tracking Method Using Asynchronous Multi-view Image, Proceedings of IEICE Information and Systems Society Conference, Japan, The Institute of Electronics, Information and Communication Engineers, 1998 9 7th, pp. 274 (D-12-52) (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06T 7/20 H04N 7/18 JISST file (JOIS)

Claims (2)

(57) [Claims] 1. A moving object tracking device for tracking a moving object moving in a scene, comprising: a plurality of photographing means for photographing the scene from a plurality of viewpoints; and a plurality of photographing means. And a plurality of observing means that operate independently of each other and asynchronously, each of the plurality of observing means extracts a feature point from an image received from a corresponding photographing means, and At least 1 from the images received from the corresponding photographing means
Attribute value extraction means for extracting an attribute value unique to one tracking target moving object, and the at least one tracking target moving object and the extracted feature points are associated with each other The first observation information including the attribute value and the second observation information regarding the feature point that is not associated with the feature point associating unit, the feature point among the plurality of observing units. By integrating the first observation information supplied asynchronously from the plurality of observing means extracted by using a Kalman filter, the state of the moving object as the at least one tracking target is converted into a tracking model of the moving object. Further comprising: a tracking unit that makes a prediction based on the attribute value extraction unit that extracts the attribute value from the supplied first observation information; State estimating means for estimating the state of the moving object that is the tracking target based on the attribute value obtained, and tracking model determining means for determining the tracking model of the moving object according to the estimated state of the moving object. A prediction by performing a Kalman filter update process in chronological order with the second observation information that is asynchronously and sequentially supplied from the plurality of observation means that extracted the feature points among the plurality of observation means. Further comprising a finding means for calculating a trajectory, and based on the calculated predicted trajectory, detecting an estimated position of a moving object to be newly tracked and transmitting it to the tracking means as an initial found position, The associating unit, based on the predicted state of the at least one or more moving objects to be tracked, transmitted from the tracking unit, a tracking pair corresponding to the extracted feature point. Detecting a moving object is, the moving object tracking device. 2. The moving object tracking device according to claim 1, wherein the attribute value is at least one of backrest information, action information, and color information of the moving object.