JP2001094975A - Moving object tracking method and apparatus - Google Patents

Abstract

(57) [Problem] To track an object moving in a wide area while avoiding an occlusion phenomenon in which tracking becomes difficult due to overlapping of object images. [MEANS FOR SOLVING PROBLEMS] A measurement area (monitoring area) using cameras C1 to C8.
The three assigned areas a1 to a8 are assigned, and a candidate for a moving object m to be tracked for each camera is determined (first, an object extracted from the assigned area). When there are a plurality of candidates to be in charge, an angle between straight lines connecting the camera position and the moving object is calculated, and the candidates are changed based on the angle evaluation (the magnitude of the threshold). For example,
If the angle θ formed by the straight line connecting the camera C5 and the candidate objects m6 and m7 in the area a5 is smaller than the threshold value, it is determined that occlusion may occur, and the unassigned camera C7 outputs m6 and m6.
Substitute m7 measurement. Or camera C4 which is close in distance
Is selected, and if the angle θ formed by the straight line connecting C4 and m6 and m7 is larger than the threshold value, the measurement is substituted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for extracting and tracing an image of a moving object such as a person or a vehicle, and more particularly to continuously tracking individual objects in a wide area where a plurality of objects move at random. Moving object tracking device.

[0002]

2. Description of the Related Art Security management for monitoring suspicious persons in event venues where a large number of people come and go using images from a television camera or the like, or identifying specific or individual players or balls along with the entire movement in a sports competition. The greatest problem in tracking, watching, analyzing, and the like is that the person or object being tracked is intersected and shielded, causing so-called occlusion. When this occlusion occurs, the object image apparently branches or merges, so that it is very difficult to track the object image.

Recently, a distributed cooperative processing using a plurality of cameras has been proposed for solving the occlusion. For example, in the document "Joint gaze of moving objects by multiple swing cameras (JSPS Future Research Scientific Research Promotion Project-Intelligent Information and Advanced Information Processing Research Field, 1997 1997 Research Results Report, March 1998)" Describes a method of tracking the movement of a plurality of objects with a plurality of cameras capable of swinging. In this method, a camera that is in charge of tracking an individual object is called a master, and the camera has authority called master authority. If tracking becomes impossible due to occlusion, the master authority is delegated to cameras at other positions that can be tracked, and the delegated camera takes over and continues tracking. I do.

Japanese Patent Application Laid-Open No. 9-331520 discloses a method in which a tracking range is determined in advance for each camera and another camera takes over when the camera moves beyond the range. A method is disclosed in which, when an object being tracked by a camera causes an overlap on an image and tracking is difficult, the tracking is continued by an auxiliary camera shooting from another direction.

[0005]

In the above-mentioned prior art, the proxy procedure of the camera is automated, but it is difficult to apply this method as it is to processing in a wide area. In the case of a wide area, it is difficult to view the entire target area with one camera. Therefore, the entire area must be divided into several small areas, and a camera in charge must be assigned to each small area. However, if the assigned area is fixed for each camera as described in JP-A-9-331520, occlusion caused by the positional relationship between the object and the camera cannot be avoided. It is an issue to execute the proxy processing of the camera for the purpose of avoiding occlusion after spatially dividing the area in this way.

In the prior art, when occlusion occurs, the standard for selecting a substitute camera is not clear, and it is considered that master authority can be delegated as long as an object can be observed. . However,
Depending on the positional relationship between the camera and the object, an inappropriate camera may be selected, for example, a calculation error is likely to occur due to a long shooting distance. Therefore, when selecting a substitute camera, it is necessary to determine the evaluation criteria of the camera so that the optimal camera is always selected.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems of the prior art, and to track an object moving in a wide area by a plurality of cameras, and to perform processing when occlusion occurs during tracking by a certain camera. It is an object of the present invention to provide an image-based moving object tracking device in which another optimal camera is selected.

[0008]

SUMMARY OF THE INVENTION In order to achieve the above object, the present invention is directed to a moving method for photographing a plurality of objects moving in a measurement area from a plurality of directions with a plurality of cameras and obtaining a moving locus of each object from the photographed images. In the object tracking method, an angle formed by straight lines connecting each camera and each moving object is obtained, and from this angle, a correspondence relationship between the camera and the object in which the image of the object does not overlap is obtained, and based on this correspondence relationship, It is characterized in that the charge of the object measured by the camera is optimized every moment.

Alternatively, the measurement area is divided to set an area in charge of each camera, and a moving object existing in its own area is set as a candidate object to be assigned to each camera. And an angle formed by straight lines connecting the candidate objects and each candidate object. If this angle is smaller than a predetermined value, another camera is substituted for the candidate object. The assigned area is determined by Voronoi division so that each camera is close to itself.

The proxy to the other camera is selected in the order of the camera closest to the candidate object, and the camera at the position where the angle formed by the straight line connecting the selected camera and each candidate object is the largest or the predetermined value It is characterized in that it is performed by a camera at a position where it becomes larger.

[0011] Further, the substitute for the other camera is performed by an unassigned camera in which a moving object does not exist in its own area. In this case, at least one of the zoom amount, the pan amount, and the tilt amount of the substitute camera is changed, and the candidate object is photographed in a large size. Further, the assigned area of the surrogate camera is temporarily changed to the assigned area of another camera.

A moving object tracking apparatus to which the method of the present invention can be applied includes a plurality of cameras for photographing a plurality of moving objects from different directions, and a moving object tracking unit for calculating a trajectory of the moving object from the photographed images. And a display for displaying a tracking result. The moving object tracking unit obtains an angle between straight lines connecting each camera and each moving object, and determines a possibility that an image of the object is superimposed by evaluating the angle. It has an angle detecting means. Further, when the angle detecting means determines that there is a possibility, the image processing apparatus has a responsible camera detecting means for selecting a camera having an angular relationship in which an image of the object does not overlap and acting in charge of the object. .

According to the present invention, a camera in charge of detecting a possibility of occurrence of occlusion in advance from an angle formed by a straight line connecting a plurality of moving objects and each camera and measuring the target object so as to avoid occlusion. Since the relationship can be changed, tracking processing of a moving object in a wide measurement area can be continued with high accuracy.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below in detail with reference to the drawings. FIG. 2 schematically shows a security system which is an application example of the present invention. This system uses a plurality of cameras 1 to photograph a measurement area 3 such as an event venue, extracts a movement trajectory 21 of a moving object 2 (here, a person) by a computer 4, and displays it along with a display model 20 on a display 5. . A video camera is used as the camera 1 for performing moving image processing, and the video signal is taken into the computer 4 in real time, for example, and a tracking process described later is executed in a time zone between video frames to track the moving object 2.

FIG. 3 is an explanatory diagram showing one embodiment of the camera arrangement of the moving object tracking device of the present invention. For example, eight cameras 1 (C1 to C8) are used to track seven moving objects 2 moving in the directions of the arrows. The rectangular area 3 in the figure is a measurement range, and the moving object 2 in this area is a tracking target. Here, the inside of the measurement range is a plane, and the object 2 moves only on this plane. It is also assumed that the position of the camera 1 is fixed, and the camera position, the camera direction, the camera angle of view (the angle of the camera field of view indicated by two thin dotted lines), and the plane direction are all known.

Here, assuming that the camera direction is not parallel to the plane of the measurement range, straight lines starting from the camera position and within the angle of view of the camera all intersect the plane at one point. Intersection coordinates can be determined. That is, with respect to the object appearing in the camera view angle, the coordinate value on the moving plane can be calculated from the coordinate value on the image. In this embodiment, the coordinates of the moving object are calculated from the image using this relationship.

For example, if the coordinate system of the plane is XY, and the coordinates of the screen 5 are UV, the point (u1, v1) projected on the screen 5 becomes the point (x1, y1) on the plane. You can see it exists. Although the original screen is the imaging surface of the camera and is located behind the lens, it is illustrated here as being located in front of the lens for the sake of simplicity.

FIG. 4 is an explanatory diagram showing an embodiment of a method for determining a camera assigned area. 8 cameras c1 to c8
, M1 to m7 denote moving objects, and a1 to a8 denote areas determined by the cameras. For example, the camera 1 at the camera position c1 is responsible for tracking the moving object m1 within the area of the assigned area a1. When measuring with an image, it is appropriate to take charge of the area closest to the camera from the relationship of errors, so the area in charge is determined based on this concept.

In order to determine a region closest to each camera, for example, a technique based on Voronoi division is used. First, not only the measurement range but also the entire range including the camera position is considered, and a vertical bisector between the closest camera positions is calculated. Then, a set of these perpendicular bisectors becomes a Voronoi dividing line. For example, the vertical bisector of c6 and c7 is
A dividing line between a6 and a7 is determined. According to this Voronoi division, it is possible to divide an area with any camera arrangement. As a matter of course, the camera 1 sets its camera direction and angle of view so as to capture the entire area of its own area. Since the position on the screen and the coordinates on the plane correspond one-to-one, the position where the assigned area is projected on the screen can also be calculated.
Therefore, only the moving object in the assigned area is extracted by the coordinates on the screen, and the position coordinates on the plane are calculated.

FIG. 1 is an explanatory diagram showing an embodiment of a method for determining occlusion according to the present invention. In the figure, for example, a straight line connecting the camera position c1 and the moving object m1 is the assigned area a1.
Indicates that the camera c1 is in charge of measuring the position of m1. When there is only one moving object in the assigned area as in a1 to a3, no occlusion occurs because the object images do not overlap in that area. However, when there are a plurality of moving objects such as the assigned areas a4 and a5, it is necessary to determine the presence or absence of occlusion.

For example, in the case of the assigned area a4, a straight line c4
−m4 and the straight line c4-m5 make an angle θ that is sufficiently large,
Since occlusion has not occurred, measurement by c4 camera is possible. On the other hand, in the case of a5, the straight line c
Since the angle θ formed between 5-m6 and the straight line c5-m7 is too small and exists immediately before occlusion occurs, sufficient measurement cannot be performed at the camera position c5. In this case, another camera in which occlusion has not occurred is substituted for measurement.

As described above, in this embodiment, the angle θ between a plurality of straight lines connecting the camera position in the assigned area and each moving object in charge of measurement is calculated, and a predetermined threshold value (θth ) To determine the presence or absence of occlusion. Note that it is also possible to store the angle θ in a frame of an image before this, and determine the presence or absence of occlusion based on a change amount such as an increasing tendency or a decreasing tendency of the angle θ.

FIG. 5 is an explanatory diagram showing an embodiment of a method of selecting a camera to perform measurement. In the present embodiment, the cameras are selected in the order of proximity to the area to be substituted, and the presence or absence of occlusion by the substitute camera is determined. In the case of the occlusion described above (FIG. 1), first, the camera c4 that is next to the area a5 is selected.

That is, it is determined whether or not the objects m6 and m7 which have become difficult to measure by the camera c5 can be substituted by the camera c4 next to c5. At this time, the camera c4 is already m
I'm in charge of measuring 4 and m5, so c instead of c5
Whether or not 4 can be substituted is determined by the camera c4 and the object m4,
Angle θ1 between straight lines connecting m5, m6 and m7
Is calculated, and if each angle is larger than the threshold value θth, it is determined that the proxy can be performed, and the subsequent m6 and m7
Is continued by the camera c4 on behalf of the camera c4. At this time, if an object that does not cause occlusion exists in the area a5, the camera c5 continues to take charge of the measurement. If any of the angles θ1 to θ3 is equal to or smaller than the threshold value θth, the possibility of substitution by the camera c6 which is next to c4 is checked.

Note that the coordinate values of the object are given in three dimensions,
If the z value indicating the depth from the camera viewpoint can be calculated, one of the objects that have caused occlusion, that is, only the object that is hidden behind the shadow and cannot be seen, may be substituted by another camera.

FIG. 6 is an explanatory diagram showing another example of a method of selecting a camera to perform measurement for. A camera not in charge of measurement of the moving object 2 performs measurement on a priority basis. Moving object 2
Is not always uniformly distributed over the entire measurement area 3,
The existing position may be localized. In the example shown,
No moving object exists in the assigned areas a6, a7, and a8. At this time, the cameras c6, c7, and c8 are in a non-assigned state without an assigned object. In the present embodiment, these unassigned cameras are used to avoid occlusion. For example, the angle θ between the straight line c7-m6 and the straight line c7-m7 is calculated, and the threshold θth
If so, the processing is performed on behalf of those that can be measured.

In the case where an unassigned camera is used instead,
Since there is no other object being tracked, only the tracking target objects m6 and m7 can be photographed large by changing the zoom amount of the camera c7 as shown. Furthermore, the direction of the camera can be changed by changing the pan amount and the tilt amount of the camera, and the target object can be photographed in a large angle from the most easily viewable angle. Thereby, it is also possible to obtain detailed information on the operation of each part of the moving object. However, the camera c7 in this case has its own area a
Since the image of No. 7 is not taken, the area a7 is changed so as to be included in the assigned area of another camera in order to cope with the inflow of a moving object from another area during this time. In FIG. 6, the portion that was the assigned area a7 of the camera c7 before the substitution (FIG. 1) is changed to the assigned area a6 of the camera c6 by deleting the Voronoi dividing line.

FIG. 7 is a flowchart showing an algorithm according to an embodiment of the moving object tracking method of the present invention. In step s101, camera conditions such as the number of shots and position coordinates are input. By defining these values, a geometric relationship such as a plane on which the camera 1 and the object 2 move is determined.
In step s102, a responsible area is assigned to each camera by Voronoi division. As shown in FIG. 4, a vertical bisector from the midpoint of the line connecting the adjacent camera positions is calculated, and from each of the bisectors extending to the calculation area 3 and their intersection, a Voronoi dividing line is calculated. I do.

In step s103, the following processing is performed for video 1
This indicates that the operation is repeated for each frame. Step s1
04 indicates that the following processing is repeated for each camera. In step s105, a candidate for the object to be assigned is selected. Here, all the moving objects included in the assigned area are extracted, and these are set as assigned candidates. At this point, it is a candidate only, since another camera may be in charge of the charge later. In order to extract a moving object from an image captured by a camera, a difference from a background image may be calculated, as is generally known.

In step s106, it is determined whether there are a plurality of object candidates in the assigned area. The number of object candidates is obtained by labeling and classifying the image of the moving object extracted in step s105 as is well known, and counting the number. If the number is not plural, occlusion does not occur, and the process proceeds to step s111.

In step s107, as shown in FIG. 5, the angle formed by each straight line connecting the plurality of object candidates and the camera position is
It is determined whether it is smaller than the threshold value θth. If it is not small, occlusion does not occur, so step
Proceed to s111.

In step s108, it is determined whether or not there is an unassigned camera having no object in charge of measurement as shown in FIG. If not, it is assumed that another camera in charge close to the area or the object next takes place as shown in FIG. 5, and the process proceeds to step s111. Step s109 is a case in which an unassigned camera substitutes.In this case, in order to capture the moving object in close-up and improve the accuracy, zoom
Optimum control of pan and tilt amount.

In step s110, the entire area of its own area cannot be photographed at the time of close-up, so the area is changed to another area in charge of the adjacent camera.
In step s111, the responsible object is determined. Only at this point does the moving object for which each camera is in charge of measurement be determined. In step s112, the processing from s105 is repeated until one frame of the video is shot by the camera.

In step s113, the position of the object is calculated from the screen coordinates as shown in FIG. 3, and the position information of the object is updated. In step s114, the position of the display model on the display connected to the computer is updated as shown in FIG. 2 based on the updated position information of the object. Then s103
And the same processing is repeated for the next frame.

In the above description, the agent in charge of the object in which occlusion has occurred gives priority to the camera in charge of the object.
A camera closer in distance may be given priority. Note that when selecting a camera that is close in distance, it is determined whether or not the angle formed by each straight line connecting the plurality of object candidates including the object in which occlusion has occurred and the camera position is smaller than the threshold value θth. It is necessary to judge.

FIG. 8 is a system configuration diagram showing one embodiment of the moving object tracking device of the present invention. The moving object tracking unit 200 is connected to a bus such as a personal computer, and executes a process of tracking a moving object while inputting and outputting data to and from a CPU, a memory, a disk, a mouse, a keyboard, a display, and the like. The moving object tracking unit 200 includes an image processing device and the like, and is configured by, for example, an LSI.

Here, the camera control unit 201 controls the input of a video signal, the amount of zoom, pan, and tilt of the camera. The object image extraction unit 202 detects a moving object image by calculating a difference from the background image. The object position calculator 203 calculates the object position from the screen coordinates. The object number detection unit 204 detects the number of moving object images in the assigned area by counting the number of labels. The angle detection unit 205 detects an angle between straight lines connecting the camera position and the moving object. The assigned camera detection unit 206 detects a camera assigned to an area where no moving object exists.
Alternatively, a camera that is close in distance and does not generate occlusion by proxy is detected. The image memory 207 stores a video signal from the camera as image data in order to perform image processing such as a difference operation.

FIG. 9 is an explanatory diagram of a moving object tracking unit that exchanges information with a computer in real time. When the objects m1 and m2 photographed by the plurality of cameras C1 to C3 are in the illustrated positional relationship, the angle detection unit 205 of the moving object tracking unit 200
Detects the angle between the camera position and each object, and inputs angle detection information of θ1, θ2, and θ3 to the computer 4 corresponding to each camera. In order to prevent the object images from overlapping each other, the computer 4 determines that the camera at the position having the maximum angle is the optimum camera (here, the camera C2), and the moving object tracking unit 20
The camera control information indicating the optimum relationship between the object and the camera is output to 0. The assigned camera detection unit 206 selects a camera from this optimal relationship and continues tracking the moving object.

As described above, the angle information of each object and the camera is passed to the computer for each frame, and the computer is used to determine the optimal charge of the camera and the object for each frame. From this, the object in charge of each camera can be determined immediately and optimally.

[0040]

According to the present invention, since the measurement range of a moving object is divided into areas in charge for each camera, an object having a possibility of occurrence of occlusion is substituted by another camera. There is an effect that the moving object tracking in the range can be continued with high accuracy.

Also, since the possibility of occlusion is determined by the angle between the camera position and the straight line connecting the moving object, the camera to be substituted is quantitatively evaluated by considering the magnitude of this angle. You can always select the best camera.

Furthermore, since the angle information between each object and the camera is calculated for each frame to determine the optimum assignment of the camera and the object, the object assigned to each camera can be determined in real time and optimally.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram showing a method for determining occlusion by a moving object detection device of the present invention.

FIG. 2 is a schematic diagram of a security system to which the present invention is applied.

FIG. 3 is an explanatory diagram showing a camera arrangement and a coordinate relationship in the moving object detection device of the present invention.

FIG. 4 is an explanatory diagram showing a method for determining a region in charge of a camera according to one embodiment.

FIG. 5 is an explanatory diagram of a method of selecting another camera to be substituted when an occlusion may occur.

FIG. 6 is an explanatory diagram of proxy by an unassigned camera when an occlusion may occur.

FIG. 7 is a flowchart illustrating an algorithm of a moving object tracking method according to an embodiment.

FIG. 8 is a configuration diagram of a moving object detection device including a moving object tracking device according to one embodiment.

FIG. 9 is an explanatory diagram of a system for exchanging angle detection information by a moving object tracking device and optimal charge relationship information by a computer.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Moving object, 3 ... Calculation area (monitoring area), 4 ... Computer, 5 ... Display, 20 ... Display model, 21 ... Moving locus, 200 ... Moving object tracking device, 201 ... Camera control part, 202 ... Object image extraction unit, 203 ... Object position calculation unit, 204 ... Number of object detection unit, 205 ... Angle detection unit, 206 ... Camera in charge, 207 ... Image memory, C1-C8 ... Camera position, a1-a8 ... In charge Area, m1 to m7: Moving objects.

Continuation of the front page (72) Inventor Masanori Miyoshi 7-1-1, Omika-cho, Hitachi City, Ibaraki Prefecture F-term in Hitachi Research Laboratory, Hitachi Ltd. 5C054 CF06 CG02 CH10 EA01 FC12 FC15 FD03 FF02 HA18 HA19 HA31

Claims (10)

[Claims] 1. A moving object tracking method in which a plurality of objects moving in a measurement area are photographed by a plurality of cameras from different directions, and a moving trajectory of each object is obtained from the photographed images. Obtain the angle between the straight lines, find the correspondence between the camera and the object in charge that the image of the object does not overlap from this angle, and optimize the responsibility of the object measured by each camera from time to time based on this correspondence A moving object tracking method, comprising: 2. A moving object tracking method in which a plurality of objects moving in a measurement area are photographed by a plurality of cameras from different directions, and a moving trajectory of each object is obtained from the photographed image. Set the camera area,
For each camera, a moving object present in its own area is taken as a candidate object to be assigned, and when there are a plurality of candidate objects, an angle between straight lines connecting the camera and each candidate object is obtained, and this angle is smaller than a predetermined value. In such a case, a moving object tracking method is characterized in that another camera is in charge of the candidate object. 3. The moving object tracking method according to claim 2, wherein the assigned area is a Voronoi division so that each camera is an area close to itself. 4. The method according to claim 2, wherein the proxy to the other camera is selected in the order of cameras closest to the candidate object, and the angle between straight lines connecting the selected camera and each candidate object is maximized. A moving object tracking method, which is performed by a camera at a position. 4. The surrogate for another camera according to claim 2, wherein the camera is selected in the order of cameras closest to the candidate object, and an angle formed by a straight line connecting the selected camera and each candidate object is the predetermined value. A moving object tracking method performed by a camera at a larger position. 5. The moving object tracking method according to claim 2, wherein the proxy for another camera is performed by an unassigned camera having no moving object in its own area. 6. The moving object tracking method according to claim 5, wherein at least one of a zoom amount, a pan amount, and a tilt amount of a surrogate camera is changed to photograph the candidate object. 7. The moving object tracking method according to claim 6, wherein the assigned area of the surrogate camera is temporarily changed to the assigned area of another camera. 8. A moving object tracking system comprising: a plurality of cameras for photographing a plurality of moving objects from different directions; a moving object tracking unit for calculating a trajectory of the moving object from the captured images; and a display for displaying a tracking result. In the apparatus, the moving object tracking unit obtains an angle formed by straight lines connecting each camera and each moving object, and has an angle detection unit that determines a possibility that an image of the object is superimposed by evaluating the angle. Characteristic moving object tracking device. 9. The moving object tracking unit according to claim 8, wherein the moving object tracking unit selects a camera having an angle relationship in which an image of the object does not overlap when the angle detection unit determines that there is a possibility. A moving object tracking device, comprising a responsible camera detecting means for acting for a responsible person. 10. The tertiary moving object according to claim 8, wherein the moving object tracking unit is configured to move the object based on a two-dimensional position where an image of the object captured by the camera is projected on the display. A moving object tracking device having an object position calculating means for calculating a moving trajectory by obtaining a position in an original space.