JP2003030664A - Mobile object extraction method and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and device of extracting a mobile object for separately extracting the individual mobile objects even in the case that that positions of a plurality of the mobile objects overlap between frame images before and after a movement. SOLUTION: The mobile object extraction method is provided with the steps of: generating the background image of the mobile object of an extraction object by obtaining the average value of a plurality of the frame images including the mobile object photographed by a fixed camera; generating the difference image of the frame image and the background image at the moment of extraction; binarizing the difference image and preparing a mask image; and cutting out the area of the mobile object in the frame image at the moment of extraction by using the mask image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting a moving object in a moving image captured by a fixed camera.

[0002]

2. Description of the Related Art Conventionally, a method of utilizing a difference between two frame images is known as a method of extracting a moving object from an image captured by a fixed camera. For details, see "Image Processing Handbook" (supervised by Morio Onoue:
Shokodo (1987)).

The difference between two frame images is the difference between the density values between the two images when the images of the same pixel in both frames are represented by f (x, y) and g (x, y). It is the absolute value of the difference, and the difference image h (x, y) is h (x, y) = f (x, y) − g (x, y) or h (x, y) = | f (x , y) − g (x, y) | When a difference image of two frame images is created, the difference becomes very small in the background part, and a negative region and a positive region are formed before and after the moving object. That is, a negative region is generated in the portion where the moving object was located before the movement, and a positive region is generated in the new position where the moving object is moved. Therefore, if a negative (or positive) area is obtained in the difference image, the moving area can be detected. Since this method can directly detect only a part of the moving object, it is necessary to use this information as the first approximation to detect the entire moving object by another method.

[0004]

By the way, in the above-mentioned conventional method, when there are a plurality of moving objects, the moving area may not be accurately detected if the moving objects overlap each other between the two frame images. . That is, as shown in FIG. 13, vehicles A and B moving on the road 1301 at time t1.
Vehicles A and B are shown in FIG.
Although it was at the position of 3 (a), if it is at the position of FIG. 13 (b) in the frame image taken at time t2, when the difference image is obtained, the vehicle A moves to the position of the vehicle B. (The vehicle A overlaps the original position of the vehicle B), the moving area between the times t1 and t2 is only the position before the movement of the vehicle A, and the moving area of the vehicle B cannot be detected. There is a problem. This means that a preceding vehicle cannot be detected, for example, when detecting a vehicle traveling on a road while maintaining a predetermined inter-vehicle distance.

An object of the present invention is to provide a moving object extraction method and a moving object extraction method which can individually extract each moving object even when the positions of a plurality of moving objects overlap between frame images before and after movement. To provide a device.

[0006]

In order to achieve the above object, the method of the present invention is a method for extracting a moving object from a moving image taken by a fixed camera. A step of generating a background image of the moving object to be extracted by obtaining an average value of the plurality of frame images, a step of generating a difference image between the frame image and the background image at the extraction target time, and the difference image It is characterized by including a step of binarizing to generate a mask image and a step of using the mask image to cut out a region of a moving object in a frame image at an extraction target time.

The device of the present invention is a device for extracting a moving object from a moving image taken by a fixed camera, and by obtaining an average value of a plurality of frame images including the moving object taken by the camera. Means for generating a background image of the moving object to be extracted, means for generating a difference image by the difference between the frame image and the background image at the extraction time, and means for binarizing the difference image to generate a mask image. And using the mask image,
Means for cutting out a region of the moving object in the frame image at the extraction target time.

[0008]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in detail based on illustrated embodiments. FIG. 1 is a block diagram of hardware of a moving object extraction device showing an embodiment of the present invention. The moving object extraction device of this embodiment includes a keyboard 101, a processing device 102, a storage device 107, and a display 112. Keyboard 1
01 inputs a command name and its parameters. The processing device 102 processes a command of the moving object extraction system. More specifically, a background image generation process 103 that generates a background image from a moving image and a difference image generation process 104 that generates a difference image from an arbitrary frame of the moving image and the background image.
And a mask image generation process 105 for generating a mask image from the difference image and a moving object cutout process 106 for cutting out a moving object from the original frame image using the mask image.

In the storage device 107, a moving image 108 that is an input to the processing device 102, a background image 109 that is input / output data for command processing of the processing device 102, and a difference image 110.
And the mask image 111 is stored. Display 112
Is a character string input by the keyboard 101 or the processing device 10.
2 displays the result of command processing.

In this embodiment, moving objects are extracted as follows. (1) A background image is created by taking an average value from a plurality of frame images. (2) A difference image is generated from the difference between the frame image and the background image at the extraction target time. (3) The difference image is binarized to generate a mask image in which the pixel value of the moving object is "1" and the pixel value of the background is "0". (4) Using the mask image, the region of the moving object in the frame image at the extraction target time is cut out.

That is, in the moving image taken by the fixed camera, the background is stationary and the moving object is moving. Therefore, when the frames of the moving image are overlapped, the background is in a fixed position and is emphasized. On the other hand, since the moving object is located in various places in each frame, its image becomes relatively unclear. If enough frames are overlapped, the moving object disappears and a so-called background image consisting of only the background is obtained. This method corresponds to the long exposure in the case of photography. To be precise, the background image is obtained as an image having a pixel value that is an average value obtained by adding the density values of the pixels of each frame and dividing by the number of frames. The moving object at the extraction target time can be easily extracted by obtaining the difference image between the frame image including the moving object at the extraction target time and the background image not including the moving object. In the difference image, the value is relatively large in the area of the moving object and very small in the background. Therefore, a mask image in which the value of the pixel corresponding to the moving object is “1” and the value of the pixel corresponding to the background is “0” is created from the difference image. Next, the partial original frame image in which the value of the mask image is “1” is cut out. Then, the moving object can be easily extracted.

The process of this embodiment will be described below by taking an example of extracting a vehicle from a moving image showing a landscape in which the vehicle is moving. FIG. 2 is an original frame image F1 of a moving image captured by a fixed camera at time t1. Here, the original frame image F1 is a frame image at the extraction target time. The background is two trees 201,
02 and one house 203, and a vehicle AM moving from left to right is on the left side of the front surface thereof. FIG. 3 is a frame image F2 taken at time t2, which is a short time after the original frame image F1 of FIG. In this frame image F2, the vehicle AM is moving to the right of the front of the same background. FIG. 4 is a background image 109 generated by taking the average value of FIGS. 2 and 3. Since the image of the vehicle AM has an average value, it is unclear as indicated by the broken line. On the other hand, the background was the same in both figures, so even if you take the average value,
Not blurred. FIG. 5 is a background image 109 generated by superimposing more frame images. In this image, the image of the vehicle AM has completely disappeared.

FIG. 6 shows the original frame image F1 of FIG. 2 and FIG.
Is a mask image 111 in which the difference image 110 is generated by taking the difference of the background image 109 and binarized to represent the vehicle AM in black and the background in white. FIG. 7 shows the vehicle AM cut out from the frame image F1 of FIG. 2 using the mask image 111.

FIG. 8 is a flow chart for explaining an algorithm for generating an average value image img1 (x, y) from a plurality of frame images f (x, y, t). Here, x and y represent pixel coordinates, and t represents the time when the frame was captured. First, in step 801, the number N of frames to be averaged is calculated. Specifically, N = (t2-t1) / dt + 1 is calculated. Here, t1 is a constant indicating the start time of the first frame and t2 is the start time of the last frame, as shown in FIG. dt is a constant indicating the time between frames. Step 802, step 803 and step 806 form a loop for x. Where x is from x1 to x2
It changes while increasing the value by 1. In step 803
When x exceeds x2, the processing ends. If not, go to step 804. Step 804, Step 80
5 and step 811 form a loop for y. Here, y changes from y1 to y2 by increasing the value by 1. Note that x1 is the first pixel in the x-coordinate direction in one frame as shown in FIG. 8B, x2
Represents the last pixel, y1 represents the first pixel in the y coordinate direction, and y2 represents the last pixel.

When y exceeds y2 in step 805, the process proceeds to step 806, the value of x is incremented by 1, and the process returns to the loop of x. If it does not exceed, the value of the variable v is set to 0 in step 807. Steps 808, 809, and 813 form a loop for t. Here, t changes from t1 to t2 by increasing the value by dt. If t has not reached t2 + dt in step 809, f (x, y, t) is added to v in step 812, and step 8
At 13, the value of t is increased by dt and the loop is executed. If it has reached, the average value v / N is assigned to img1 (x, y) in step 810, the value of y is increased by 1 in step 811, and the process returns to the loop of y.

That is, after the x coordinate is set in step 802 and the y coordinate is set in step 804, the image f (x, y, t) of the corresponding coordinate in each frame from t1 to t2 is set in steps 812 and 813. Is calculated and the average value is calculated in step 810, and if the average value for one line in the y coordinate direction from the frame from t1 to t2 is calculated, the y coordinate is updated by one and the next one line is calculated. The process of obtaining the average value of is executed until the coordinate value in the x coordinate direction exceeds x2. As a result, an average value image is obtained for the N frame images.

FIG. 9 is a flow chart for explaining an algorithm for generating a difference image img2 (x, y) between the frame image f (x, y, t1) and the background image img1 (x, y). This flow chart contains a double loop of x and y. x is from x1 to x2
Up to, y takes a value from y1 to y2, and step 906 is executed. In step 906, the frame f (x, y, t1)
And the background image img1 (x, y) is calculated as the absolute value of the difference, and the difference image
Substitute that value into img2 (x, y).

FIG. 10 is a flow chart for explaining an algorithm for binarizing the difference image img2 (x, y) to generate the mask image img3 (x, y). This flow chart is x
It contains a double loop of y and y. x is from x1 to x2, y
Takes a value from y1 to y2, and step 1006, step 1007 and step 1008 are executed. In step 1006, it is determined whether the difference image img2 (x, y) is larger than a predetermined threshold value T. If so, step 10
At 07, "1" is assigned to the mask image img3 (x, y). If it is not large, the mask image img3 (x, y) in step 1008
Substitute "0" for.

FIG. 11 shows an algorithm for generating an image img4 (x, y) obtained by cutting out a region of a moving object in the original frame image f (x, y, t1) using the mask image img3 (x, y). It is a flowchart explaining. This flow chart is x and y
It contains a double loop. x is from x1 to x2, y is y1
Take values from 1 to y2, step 1106, step 1
107 and step 1108 are executed. In step 1106, it is determined whether the mask image img3 (x, y) is "1". If it is “1”, the image img4 is displayed in step 1107.
Substitute the value of the original frame image f (x, y, t1) for (x, y).
If not "1", the image img4 (x, y) in step 1108
Substitute "0" for. This image img4 (x, y) is output as the extracted image of the moving object.

Therefore, for example, when there are frame images in which two moving vehicles AM1 and AM2 are photographed as shown in FIG. 12A, the average value of a plurality of frame images photographed from the same position is calculated. By taking the above, a background image as shown in FIG. 12B is generated. Therefore, if the difference between this background image and the original frame image of FIG.
A difference image as shown in FIG. 12C is generated. Then, when the difference image is binarized, a mask image as shown in FIG. 12D is generated. Finally, using the binarized image, two vehicles AM1, AM from the original frame image F1
When 2 is cut out, the images of the two vehicles AM1 and AM2 are obtained as shown in FIG. That is, according to the present embodiment, even when the following vehicle is photographed so as to overlap the position of the preceding vehicle over time, each vehicle can be clearly extracted. By the way, in the conventional method, only the vehicle AM1 is extracted and AM2 is not extracted.

Therefore, it becomes possible to easily extract an object moving at a certain interval from a moving image taken by a fixed camera. For example, the moving object can be moved on a remote sensing image such as an aerial photograph or a satellite image. It is extremely useful when applied to a system that counts the number of vehicles inside and observes the traffic volume, a security system that detects suspicious persons with a surveillance camera, and an animal dynamics detection system in animal experiments for the purpose of understanding animal behavior Has a great effect. The mask image does not faithfully reflect the shape of the moving object, and may be wider than the shape of the moving object.

[0022]

As is apparent from the above description, according to the present invention, even if the positions of a plurality of moving objects are overlapped between the frame images before and after the movement, the moving objects are individually separated. Can be extracted.

[Brief description of drawings]

FIG. 1 is a block diagram of hardware of a moving object extraction device showing an embodiment of the present invention.

FIG. 2 is a diagram showing an example of an original frame image F1 of a moving image captured by a fixed camera at time t1.

FIG. 3 is a diagram showing an example of a frame image F2 captured at time t2, which is a time after the original frame image F1 of FIG.

FIG. 4 is a diagram showing an example of a background image generated by taking the average value of FIGS. 2 and 3;

FIG. 5 is a diagram showing an example of a background image generated by superimposing more frame images.

6 is a diagram showing an example of a mask image binarized after the difference image is generated by taking the difference between the original frame image F1 of FIG. 2 and the background image of FIG.

FIG. 7 shows a frame image F1 of FIG. 2 using a mask image.
It is the figure which cut out the moving object from.

FIG. 8 is a flowchart showing an algorithm for generating an average value image from a plurality of frame images.

FIG. 9 is a flowchart showing an algorithm for generating a difference image between a frame image and a background image.

FIG. 10 is a flowchart showing an algorithm for binarizing a difference image and generating a mask image.

FIG. 11 is a flowchart showing an algorithm for generating an image in which a region of a moving object is cut out using a mask image.

FIG. 12 is an explanatory diagram showing a process of extracting a plurality of moving objects that are moving.

FIG. 13 is a diagram illustrating a problem of the conventional method.

[Explanation of symbols]

101 ... Keyboard, 102 ... Processing device, 103 ... Background image generation processing, 104 Difference image generation processing, 105 ... Mask image generation processing, 106 ... Moving object cutout processing, 1
07 ... storage device, 108 ... moving image, 109 ... background image,
110 ... Difference image, 111 ... Mask image, AM ... Moving object.

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Takashi Onoyama             6-81 Onoe-cho, Naka-ku, Yokohama-shi, Kanagawa             Hitachi Software Engineering Stock Association             In-house F term (reference) 5L096 AA06 DA01 EA37 GA08 HA03                       JA09

Claims (2)

[Claims] 1. A method for extracting a moving object from a moving image captured by a fixed camera, comprising: determining an average value of a plurality of frame images including the moving object captured by the camera, Generating a background image, generating a difference image between the frame image at the extraction target time and the background image, binarizing the difference image to generate a mask image, and using the mask image And a step of cutting out a region of the moving object in the frame image at the extraction target time. 2. An apparatus for extracting a moving object from a moving image captured by a fixed camera, wherein the moving object to be extracted is obtained by obtaining an average value of a plurality of frame images including the moving object captured by the camera. A means for generating a background image; a means for generating a difference image based on the difference between the frame image at the extraction target time and the background image; a means for binarizing the difference image to generate a mask image; A moving object extracting apparatus, comprising means for cutting out an area of a moving object in a frame image at an extraction target time.