JPH0855220A - Moving object extraction processing method and method - Google Patents

Abstract

(57) [Summary] (Modified) [Purpose] It is possible to follow changes in the image due to changes in the imaging environment, and by using multiple image features, it is possible to stably separate and extract moving objects. [Structure] When a moving image is input in frame units, the feature extraction unit 101 extracts a plurality of types of feature amounts for each pixel or each small region of the moving image. Each feature amount is input to the image change information extraction unit 102, and it is determined by the time series of the accumulated feature amount that the feature amount of the current frame deviates from the time series change tendency of the above sequence. Is extracted as the difference. Next, the variation amount is calculated, and the normalized feature difference is calculated. Next, all the feature differences are integrated into one value to obtain the feature change amount. Further, the change amount of the feature is converted into the image conversion amount and extracted as the final image change information. The threshold processing unit 104 separates the area of the moving object from the obtained image change amount using a known method.
Extract.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement of a moving object extraction processing method for separating and extracting a moving object region and a background region from a moving image captured by an image pickup device.

[0002]

2. Description of the Related Art Conventionally, as a general method for extracting a moving object area in a moving image, a method called a background subtraction method is known. In this method, an image corresponding to the background is captured and accumulated in advance, and a difference in luminance between both images is calculated by comparing the input moving image with the accumulated image,
When this difference is equal to or larger than a certain set value, it is regarded as an image change due to the moving object, and the moving object is extracted by extracting a region regarded as the image change. As a document describing this method, there is, for example, “Computer Vision” edited by Masahiko Taniuchi.

[0003]

The background subtraction method described above has the advantage that moving objects can be extracted relatively easily. However, the background subtraction method is not always effective when an image of a moving object in an outdoor environment where the imaging environment changes successively depending on the weather or time is targeted for extraction. The reason is that the luminance of the image changes not only in the moving object region in the input image but also in the region corresponding to the background, so that the luminance difference between the input image and the stored image becomes large, and as a result, This is because the background area may be erroneously extracted as the moving object area.

Further, since the actual moving object and the background are various, the brightness values thereof are also different. Especially, when the brightness value of the moving object and the brightness value of the background are close to each other, the image as described above is displayed. However, the method of extracting a moving object focusing only on the luminance has a drawback that the reliability of the extraction is lowered.

Therefore, a main object of the present invention is to provide a moving object extraction processing method capable of following changes in an image due to changes in the imaging environment.

A further object of the present invention is to provide a moving object extraction processing system which can stably separate and extract moving objects by using a plurality of image features.

[0007]

According to the present invention, in a moving object extraction processing method for extracting a region of a moving object from a moving image including a moving object, each pixel or each small region is extracted from the image of each frame of the moving image. A feature amount extraction unit that extracts one or more types of image feature amounts, and a tendency of a time-series change of the feature amount from the feature amount of the past frame extracted by the feature amount extraction unit to determine the change in the imaging environment. By comparing the feature amount estimating means for estimating the feature amount of the considered current frame, the feature amount of the current frame extracted by the feature amount extracting means, and the feature amount of the current frame estimated by the feature amount estimating means. Based on the characteristic change amount calculated by the characteristic change amount calculating unit and the characteristic change amount calculating unit that obtains the difference between the two as the amount of change in the characteristic amount, the region showing the characteristic change amount is And further comprising a determining means for determining whether or not corresponding to the animal body.

In this system, it is desirable that the calculated feature change amount be converted into an image change amount by the image change amount calculation means based on a predetermined distribution function, and the feature change amount be replaced by the determining means. It is configured to use the image change amount.

More preferably, this method extracts a plurality of types of characteristic amounts, estimates a plurality of types of characteristic amounts in the current frame from time-series changes of the plurality of types of characteristic amounts, and It is configured to obtain a difference between the extracted value of the feature amount and the estimated amount and integrate the difference to obtain the feature change amount.

[0010]

In the above structure, by comparing the feature amount extracted from the current frame for each pixel or each small region with the estimated value in the current frame based on the past time series change of the feature amount, both Is calculated as the change amount of the feature amount. Then, based on the feature change amount, it is determined whether or not the region indicating the feature change amount corresponds to the moving object.

According to this method, since the change of the background feature amount due to the change of the image pickup environment is incorporated in the estimated value, the movement is performed based on the difference between the estimated value and the extracted actual feature amount. By extracting an object, it is possible to extract a moving object that follows changes in the imaging environment.

Further, when the feature change amount is converted into an image change amount based on a predetermined distribution function and the moving object is detected using this image change amount, the distribution function is set appropriately. By doing so, the influence of noise included in the feature change can be removed from a probabilistic point of view, and the change of the subject itself can be more accurately captured as the image change amount, which enables more accurate moving object detection. .

Further, when a plurality of types of features are used and the difference between the extracted value and the estimated value of the feature amount is integrated to obtain the feature change amount, the object movement in the image is a certain feature. Even if a change does not appear significantly, it may appear as a change in another type of feature. Therefore, if there is a significant change in at least one type of feature, a significant feature change amount can be obtained. It is possible to more stably extract the moving object as compared with the case of using only the feature of. In particular, this method is effective when the background feature value and the moving object feature value are close to each other for a certain type of feature.

[0014]

Embodiments of the present invention will now be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing the basic principle of an embodiment of a moving object extraction processing system according to the present invention.

As shown in the figure, this system uses the feature extraction unit 10
1, image change information extraction unit 102, feature amount correction processing unit 10
3 and a threshold processing unit 104.

The feature extraction unit 101 extracts the features of the input moving image, and from the moving image input in frame units from an image pickup device (not shown) such as a TV camera,
One or more kinds of required feature quantities are extracted for each pixel or each small area of the moving image. This feature amount is an amount represented by color information or grayscale information of the moving image, or an amount obtained for each pixel or small area of the moving image by applying a known method to these information. It means that.

The image change information extraction unit 102 inputs each feature amount extracted for each pixel or each small region by the feature extraction unit 101, and pays attention to it using a series of feature amounts accumulated from the past to the present. The degree of deviation of the feature amount of the current frame (that is, the feature amount of the current frame; the same applies hereinafter) from the tendency of the time series change of the above series is extracted as the difference between the feature amounts. Next, the variation amount of the feature amount is calculated from the time-series value of each feature amount, the difference between the feature amounts extracted using the calculated variation amount is normalized, and the normalized feature difference is calculated.

Next, the feature difference calculated in this manner is used as the feature change amount. When two or more types of features are used, the feature differences of all the features are integrated into one value to obtain the feature variation amount. Further, the feature change amount is converted using a preset conversion curve so as to associate the change amount of the feature with the possibility of being a moving object, and is calculated as an image conversion amount. The image change amount calculated in this way is corrected using the image change amount in the vicinity thereof and extracted as final image change information. However, the tendency of the feature change described here indicates a global change of the time-series value of the feature at a certain time interval, and the feature change indicates a local change. As described above, by extracting the information of the image change in consideration of the time-series change of the feature amount, it is possible to more clearly extract the change of the image due to the moving object, which is different from the change of the image due to the change in the imaging environment.

The threshold value processing unit 104 separates / extracts the area of the moving object from the image change amount obtained from the image change information extraction unit 102 by using a known method such as a constant threshold value or a recursive threshold value.

On the other hand, based on the image change amount extracted by the image change information extraction unit 102, the feature amount correction processing unit 103 determines whether the change is caused by a moving object or a change in the imaging environment. If there is a high possibility of a moving object, each feature quantity of the frame of interest is deleted or modified and added to the series of each feature quantity accumulated from the past.

By successively performing the above-described image change information extraction and feature amount correction processing on each pixel of each frame, change information can be extracted in accordance with changes in the imaging environment, and stable moving objects can be extracted. it can.

FIG. 2 shows an example of a method of calculating the feature change amount by the image change information extraction unit 102.

In FIG. 2, the horizontal axis represents time, and the vertical axis represents a certain characteristic amount (for example, brightness value), and the symbol Pi (i = TK ... T-). 1) indicates the feature amount in the past frame, and the symbol PT indicates the feature amount in the current frame of interest. Thus, when attention is paid to a certain one feature amount, the obtained feature amount series is distributed like the black dots in FIG. The curve A is the feature value PT-
1 to PT-k is a curve (function) showing a change tendency of the feature amount obtained by a filtering method, a polynomial approximation method, or the like, and a code QT is a feature in the frame of interest estimated by the curve A. It represents the quantity.
The normalized feature change amount is the curve A and the feature amount PT-1 to
The variance σT calculated using PT-k and the difference DT between the feature amount PT and the estimated feature amount QT.

FIG. 3 is a block diagram showing the configuration of the moving object extraction processing system according to this embodiment. This method uses luminance information as one of the features obtained from an image, and extracts a moving object by using it.

As shown in the figure, this system uses the image storage unit 20.
1, image memory 202, feature extraction unit 203, feature amount memory 204, feature amount estimation unit 205, estimated value storage buffer 2
06, statistic calculation unit 207, statistic value storage buffer 20
8, image change amount calculation unit 209, feature change amount memory 21
0, an image change amount memory 211, a change amount correction unit 212, a threshold value processing unit 213, a moving object image memory 214, a moving object output unit 215, and a feature amount correction processing unit 216.

First, a moving image is output from the image input system such as a television camera to the image storage unit 201 for each frame and stored in the image storage unit 201 for each frame. The frame images are stored in the image storage unit 2 for each frame in the order in which they are stored.
01 to the image memory 202. Then, the feature extraction unit 203, which will be described later, about the transmitted frame.
When the series of processing by the feature amount correction processing unit 216 is completed, the image of the next frame is sent from the image storage unit 201 to the image memory 202.

The feature extraction unit 203 reads the image of the current frame sent from the image storage unit 201 from the image memory 202, calculates the brightness value thereof, and stores the calculation result in the feature amount memory 204. The past brightness value accumulated in the feature amount memory 204 is read by the feature amount estimating unit 205, the statistic amount calculating unit 207, and the image change amount calculating unit 209.

The feature quantity estimation unit 205 reads the past brightness values from the feature quantity memory 204, and performs a filtering process or a function approximation to the time series of the brightness values to calculate the tendency of the change of the brightness values. Then, the brightness value in the current frame is estimated.

In the present embodiment, the approximation by the quadratic function is used as the method of calculating the tendency of the change of the brightness value. Here, a method of estimating the brightness value by the quadratic function will be described with reference to FIG. The feature amount estimation unit 205 first uses the past luminance values PT-k to PT-1 shown in FIG. 2 to calculate the coefficients a, b, and c of the quadratic function represented by the following equation (1).
Is calculated by the method of least squares, and the function A shown in FIG. 2 is obtained.

[0031]

[Equation 1] Here, X indicates the coordinate on the horizontal axis and Y indicates the coordinate on the vertical axis. After the function A is obtained in this way, the feature quantity estimating unit 205
Calculates the luminance estimated value QT in the current frame by the following equation (2).

[0032]

[Equation 2] Referring to FIG. 3 again, the estimated luminance value QT obtained by the above equation (2) is stored in the estimated value storage buffer 206 by the feature amount estimation unit 205.

The statistic calculation unit 207 includes a feature quantity estimation unit 20.
When the above-mentioned processing in 5 is completed, the past brightness value is read from the feature amount memory 204, and the function A obtained by the feature amount estimating unit 205 is used to calculate the time T- by the following equations (3) and (4). The variance σT for the function A of the luminance value from K to time T-1 is obtained.

[0034]

(Equation 3) The variance σT obtained by the equations (3) and (4)
Are stored in the statistical value storage buffer 208.

When the above-mentioned processing by the statistic calculation unit 207 is completed, the image change amount calculation unit 209 determines the feature amount memory 20.
4, estimated value storage buffer 206, statistical value storage buffer 2
08, the luminance value PT, the estimated value QT, and the variance σT are read out, respectively, and using these, the following equations (5), (6), and (7)
According to the equation, the difference DT in feature amount (that is, the difference between the brightness value PT and the estimated value QT at the current time T shown in FIG. 2), the normalized difference NT in feature amount, and the feature change amount LT Ask.

[0036]

[Equation 4] The image change amount calculation unit 209 uses this feature change amount LT to obtain the image change amount RT from the curve G shown in FIG.
The feature change amount LT is stored in the feature change amount memory 210, and the image change amount RT is stored in the image change amount memory 211. Here, the curve G is a conversion curve from the feature change amount LT to the image change amount RT and can be expressed by the following equation (8).

[0037]

(Equation 5) Here, X indicates the coordinate on the horizontal axis (X ≦ 0), and Y indicates the coordinate on the vertical axis.

The above-mentioned characteristic change amount LT is expressed by the equation (5),
As is clear from the equations (6) and (7), it is a value that directly represents the change in the feature amount. The region where the amount of change is close to 0 is considered to be mainly due to noise, and when there is no moving object, it is assumed that the amount of change and the frequency of occurrence of change follow a normal distribution. Therefore, a simple change amount is converted into an amount having a probabilistic meaning by converting the feature change amount LT into the image change amount RT by the conversion curve G having the shape of the normal distribution. This makes it possible to probabilistically evaluate how far the obtained image change amount RT is from the normal value, which facilitates the subsequent threshold processing.

After the image change amount is obtained as described above, the change amount correction unit 212 averages the image change amount of the pixel of interest on the image and the image change amounts of the eight surrounding pixels,
By replacing this average value with the image change amount of the pixel of interest, a value greatly different from the surroundings is corrected, and the value of the image change amount memory 211 is rewritten.

The threshold value processing section 213 has a variation correction section 212.
When the above-mentioned processing by is ended, the image change amount memory 211
The image change amount for one frame is read out, and when there is an image change amount equal to or larger than the threshold value set in advance for each pixel, it is determined that the pixel belongs to the moving object region, and the information is stored in the moving object image memory 214. Write. This information is
It is read by the moving object output unit 215.

On the other hand, when the above-mentioned processing by the change amount correction unit 212 is completed, the feature amount correction processing unit 216 outputs the feature change amount L of the current frame from the feature change amount memory 210, and
The image change amount R is read from the image change amount memory 211, respectively. When it is determined that the value of the read characteristic change amount L exceeds the threshold value 1, the characteristic amount correction processing unit 216 does not use the luminance value of the current frame by the statistical amount calculation unit 207 in the processing operation in the next frame. The information for doing so is written in the feature amount memory 204. In addition, the read image change amount R
With respect to, the luminance value P and the estimated value Q of the current frame are used to obtain a correction value H by the following equation (9), and the correction value H is added to the feature amount memory 204.

[0042]

(Equation 6) Here, the fact that the feature change amount L is large means that there is a high possibility that the feature amount is changed by a moving object. Therefore, the current feature quantity is inappropriate as a value for calculating the time-series variance of steady-state image features,
In this embodiment, "1" is set as the threshold because it is necessary to exclude the current feature amount. This “1” corresponds to “1σ” in the normal distribution, and is set because about 90% of the feature change amount is 1 or less if there is no influence of the moving object. In addition, the correction value H is set by the feature amount correction processing unit 216 after the processing of the current image is completed.
It is replaced with the actual feature amount PT indicated by. The reason for calculating the correction value H will be described below.

When the current image change amount detection result is large, there is a high possibility that the actual feature amount PT at that time indicates the feature amount of the moving object. Therefore, in order to obtain the function A indicating the change tendency in the steady state, the above-mentioned PT is an inappropriate value. On the other hand, the feature value QT predicted for the current image is considered to be a reasonable value. Therefore, which of PT and QT is to be used is decided.

That is, when the image change amount RT is small, PT
Is an appropriate value (because the value of PT is due to noise or a gradual change in the environment). On the other hand, as the image change amount RT becomes larger, QT becomes more appropriate than PT (because the value of PT is due to the moving object). As described above, it is necessary to calculate a value that reflects two properties, a property that is suitable when the image change amount RT is small and a property that is suitable when the image change amount RT is large. The formula for the correction value H is a simple description of this property.

The correction value H having such a meaning is obtained by the feature quantity estimating unit 205 in the processing operation in the next frame.
When the function A shown in FIG. 2 is obtained, it is used instead of the brightness value P.

By sequentially executing the processing procedure as described above for input frames, it is possible to use a plurality of image features to follow changes in the imaging environment, and to stably extract moving objects. It will be possible to do.

By the way, in the above embodiment, only one type of image feature, that is, the brightness value is used, but it is also possible to use two or more types of image features. The method of calculating the image change amount in this case will be described with reference to FIG. 5, taking as an example the case of using two types of features.

The image feature amount for two types of features is P
1, P2, the estimated value is Q1, Q2, and the feature amount difference is D1, D2. The normalized feature amount difference N1, N2 uses the respective feature variances σ1, σ2 as follows ( It can be expressed by equations (10) and (11).

[0049]

(Equation 7) The feature change amount L is calculated by a known distance calculation method using the values of the normalized feature amount differences N1 and N2, and the image change amount R is obtained from the graph shown in FIG. For example, if the Euclidean distance is used, the feature change amount L is
It is calculated by the equation (2).

[0050]

[Equation 8] The image features other than the brightness value include, for example, a brightness correlation value, a dispersion value, and a hue.

Here, the luminance correlation value is a value representing the difference in the luminance structure in the area between the current image and the image one time before, and is the same position between the image one time before and the current image. Is calculated by the following equation (13).

Correlation value = σaσb / (σabσab) (13) where σa is the variance of the luminance in the area in the image one time before, and σb is the area in the current image. , Σab is the covariance of the brightness in the regions in both images.

The luminance variance value is a simple luminance variance obtained for a certain area of the current image, and represents the complexity of the luminance structure in the area.

The hue is, for example, a value expressed simply by calculating the ratio of the RGB values.

As described above, by using a plurality of features, a change in a feature that is not detected by only one feature can be detected by another feature, which is reflected when calculating the feature change amount. As a result, it is possible to reduce the trouble of failing to detect change information. For example, since red and gray have similar values in terms of luminance, it is difficult to distinguish them by luminance alone, but since they differ in hue or saturation, they can be distinguished.

If the information indicating that even one feature has changed is extracted in this way, it can be considered that there is a true change in the feature. Therefore, it is effective to use such a plurality of features. Become.

As described above, according to the embodiment of the present invention, it is possible to follow changes in the imaging environment, and it is possible to handle a plurality of image features having different characteristics in an integrated manner. Not only can extraction be performed stably, but by selecting and using image features that match the target moving object, it is possible to improve extraction accuracy as appropriate, for example, detection of pedestrians and traffic accidents. It can be applied to various uses such as tracking of a vehicle that has caused

The contents described above are only related to one embodiment of the present invention, and it goes without saying that the present invention is not limited to the above contents.

[0059]

As described above, according to the present invention,
It is possible to stably extract a moving object by following changes in the image due to changes in the imaging environment.

[Brief description of drawings]

FIG. 1 is a block diagram showing the basic principle of an embodiment of a moving object extraction processing method according to the present invention.

FIG. 2 is an explanatory diagram of a method of calculating a feature change amount according to the same embodiment.

FIG. 3 is a block diagram showing a detailed configuration of the embodiment.

FIG. 4 is an explanatory diagram of a method of calculating an image change amount according to the embodiment.

FIG. 5 is an explanatory diagram of a method of calculating an image change amount when two or more features are used.

[Explanation of symbols]

 101 feature extraction unit 102 image change information extraction unit 103 feature amount correction processing unit 104 threshold value processing unit 201 image storage unit 202 image memory 203 feature extraction unit 204 feature amount memory 205 feature amount estimation unit 206 estimated value storage buffer 207 statistical amount calculation unit 208 statistical value storage buffer 209 image change amount calculation unit 210 feature change amount memory 211 image change amount memory 212 change amount correction unit 213 threshold value processing unit 214 moving object image memory 215 moving object output unit 216 feature amount correction processing unit

Claims (5)

[Claims] 1. A moving object extraction processing method for extracting a region of a moving object from a moving image including a moving object, wherein at least one type of image feature is provided for each pixel or each small region in the image of each frame of the moving image. A feature amount extraction unit that extracts the amount, and a feature of the current frame that considers a change in the imaging environment by obtaining a tendency of a time-series change of the feature amount from the feature amount of the past frame extracted by the feature amount extraction unit. By comparing the feature amount estimation unit that estimates the amount, the feature amount of the current frame extracted by the feature amount extraction unit, and the feature amount of the current frame estimated by the feature amount estimation unit. Based on the characteristic change amount calculated by the characteristic change amount calculating means, the region indicating the characteristic change amount is moved. Moving object extraction method characterized by comprising determining means for determining whether or not corresponding to the object, the. 2. The moving object extraction processing method according to claim 1, wherein the feature change amount is converted into an image change amount based on a predetermined distribution function indicating an appearance frequency of the feature change amount due to noise. The moving object extraction processing method, further comprising image change amount conversion means for performing the determination, wherein the determination means makes the determination based on the image change amount converted by the image change amount conversion means. 3. The moving object extraction processing method according to claim 1, wherein in order to extract a moving object region in a future frame, the moving object region is extracted from the current frame based on the feature change amount or the image change amount. A moving object extraction processing method, further comprising: a correction unit that corrects the characteristic amount. 4. The moving object extraction processing method according to claim 1, wherein the feature amount extraction unit extracts a plurality of types of feature amounts, and the feature amount estimation unit extracts the plurality of types of feature amounts in a current frame. Estimating, the feature change amount calculating means obtains a difference between the plurality of types of feature amounts extracted from the current frame and the estimated plurality of types of feature amounts, and integrates the differences to obtain the feature change. A method for extracting a moving object, which is characterized in that it is a quantity. 5. A moving object extraction processing method for extracting a region of a moving object from a moving image including a moving object, wherein at least one type of image feature is provided for each pixel or each small region in the image of each frame of the moving image. The feature amount extraction process for extracting the amount of features, and the feature amount of the current frame in consideration of the change of the imaging environment by obtaining the tendency of the feature amount in time series change from the feature amount of the past frame extracted by the feature amount extraction means. By comparing the feature amount estimation process for estimating the amount, the feature amount of the current frame extracted by the feature amount extracting means, and the feature amount of the current frame estimated by the feature amount estimating means, the difference between the two Based on the feature change amount calculated by the feature change amount calculating means, the characteristic change amount calculation process for obtaining the feature change amount as a change amount of the feature amount. Moving object extraction method area that is characterized in that and a determination process of determining whether an area of the moving object.