JP2012033100A - Image processing device and method and computer program - Google Patents

Abstract

An object of the present invention is to continuously detect a new change in an image and prevent inconsistency between an actual object state in the image and an analysis result of the object state in the image.
After an object that has been stationary for a long time in an image is backgrounded, shape information and elapsed time information of the backgrounded object are generated as backgrounded object information and stored in a backgrounded object storage unit 420 deep. Then, the object information of the input image is compared with the stored background object information, and if there is no object information that matches the background object information, the analysis rule is applied to the background object information. Analyze the state of the backgrounded object.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a computer program, and is particularly suitable for use in analyzing a state change of an object in an image by image processing.

In recent years, analysis of state changes of objects in an image has been performed by image processing. A change in the image is detected by an inter-frame difference method, a background difference method, or the like, and the change is analyzed as a state change of an object in the image (for example, appearance or disappearance of an object). In particular, when the background subtraction method is used, an object that remains stationary after appearing in the image can be detected, so that it is possible to know whether the object has been left or left.
In the background subtraction method, the object detection accuracy greatly depends on the background image used for comparison with the input image. Therefore, a method for using a more appropriate background image has been proposed.

Patent Document 1 discloses a method for adaptively generating a background model. Patent Document 2 discloses a method for reducing the influence of an object that should not be a background image on the background image.
With the method disclosed in Patent Document 1, after continuing to detect a stationary object, the stationary object can be used as a background image. Further, according to the method of Patent Document 2, an object that is stationary for a long time (that is, that has a high appearance frequency) can be used as a background image without using an object that has a low appearance frequency as a background image. In both methods disclosed in Patent Documents 1 and 2, a stationary object is used as a background image so that a new change occurring in the image can be detected thereafter.

JP 2007-323572 A JP 2009-64228 A

  However, since there is no difference from the input image when it becomes the background image, a stationary object is not detected as an object. In this case, the object is actually analyzed as having disappeared although the object is still (that is, continues to exist). Thereafter, when the object does not actually exist in the image due to movement or the like, the part which has been shadowed by the object so far becomes visible on the screen, and this is detected as a change. However, after analyzing that the object has disappeared, the change is analyzed as the appearance of a new object, and the state of the object and the analysis result of the state of the object become contradictory.

On the other hand, Patent Document 1 discloses a method for controlling the time for which detection is continued as a stationary object, but discloses an analysis method for when a background image is no longer detected as an object. Absent. Also, Patent Document 2 discloses a method for stopping image processing of a region where an object is detected in order to continue detection as a stationary object. An analysis method when it is no longer detected is not disclosed.
The present invention has been made in view of such problems, and analyzes the state of the actual object in the image and the state of the object in the image while continuously detecting new changes in the image. The purpose is to prevent inconsistencies with the results.

  An image processing apparatus according to the present invention includes an image input unit that inputs an image, an object detection unit that detects an object from the image input by the image input unit by a background subtraction method, and an object detected by the object detection unit. When a predetermined backgrounding condition is satisfied, a backgrounding unit that backgrounds the object, a determination unit that determines whether the object detected by the object detection unit corresponds to the backgrounded object, and An object detected by the object detecting means, and an analyzing means for analyzing the state of the backgrounded object, wherein the analyzing means determines that the object detected by the object detecting means is the object detected by the determining means. When it is determined by the backgrounding means that it does not correspond to the backgrounded object, the state of the backgrounded object is analyzed, and when it is determined that it corresponds, the state of the backgrounded object is determined. Characterized in that it does not parse.

  According to the present invention, when it is determined that the detected object does not correspond to the backgrounded object with respect to the backgrounded object, the state of the backgrounded object is analyzed. Therefore, it is possible to prevent a contradiction between the actual object state and the analysis result of the object state detected from the image while continuously detecting a change in the state of the new object on the screen.

It is a figure which shows the structure of the image processing apparatus of 1st Embodiment. It is a block diagram which shows the structure of the object detection part of 1st Embodiment. It is an operation | movement flowchart of the sampling data update part of 1st Embodiment. It is a block diagram which shows the structure of the state analysis part of 1st Embodiment. It is an operation | movement flowchart of the analysis rule application part of 1st Embodiment. It is a block diagram which shows the structure of the object detection part of 2nd Embodiment. It is a block diagram which shows the structure of the state analysis part of 2nd Embodiment.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram illustrating an example of the configuration of the image processing apparatus according to the first embodiment.
The image processing apparatus 100 includes an image input unit 110, an object detection unit 120, a state analysis unit 130, and an analysis result output unit 140. The image processing apparatus 100 receives an image and outputs the analysis result.

The image input unit 110 continuously takes in image data from an external device (not shown) connected to the image processing apparatus 100 (an image storage / playback apparatus using a video camera, a video tape, a DVD, a semiconductor memory, or the like). The image input unit 110 converts the captured image into a predetermined image format that can be handled by the object detection unit 120, and supplies the converted image data to the object detection unit 120 as image data. The image input unit 110 is realized using, for example, an input interface and an image conversion device.
The object detection unit 120 and the state analysis unit 130 will be described later.

  The analysis result output unit 140 converts the analysis information output by the state analysis unit 130 into a predetermined format that can be handled by an “apparatus that uses an image processing result” (not shown) connected to the image processing apparatus 100. Output as analysis result. The analysis result output unit 140 is realized using, for example, an output interface and a data conversion device. For example, when an image storage / playback apparatus is connected to the image processing apparatus 100 as an apparatus that uses image processing results, the analysis result output unit 140 converts the analysis information into a search index format for appearance and disappearance of stationary objects. And output. The analysis result output unit 140 can also output analysis information by sending an e-mail triggered by the input event. When an event processing device that reproduces a warning sound is connected to the image processing device 100, the analysis result output unit 140 converts the analysis information into an event format of appearance and disappearance of a stationary object and outputs the converted event information.

Next, the configuration and operation of the object detection unit 120 will be described with reference to FIGS.
FIG. 2 is a block diagram illustrating a detailed configuration of the object detection unit 120.
The object detection unit 120 includes a sampling unit 200, a sampling data storage unit 210, a difference calculation unit 220, an object data aggregation unit 230, a sampling data update unit 240, and a background object data aggregation unit 250.

  The object detection unit 120 receives the image data from the image input unit 110 and outputs the object information detected by the background difference method and the background object information to the state analysis unit 130. One of the features of the present embodiment is that background object information is used. The object detection unit 120 is realized using, for example, a data processing device that operates by program control. Since the background subtraction method is disclosed in detail in Patent Documents 1 and 2, the minimum necessary configuration and operation will be described below, and detailed description thereof will be omitted. Of course, the background subtraction method is not limited to those disclosed in these.

The sampling unit 200 extracts the feature amount of the image data from the image input unit 110 in units of pixels or in units of blocks including a plurality of pixels. As the feature amount, for example, luminance or hue values for each pixel or each block can be used. The feature quantity extracted by the sampling unit 200 is stored in the sampling data storage unit 210 as sampling data by the sampling data update unit 240 via the difference calculation unit 220.
First, sampling data recorded in the sampling data storage unit 210 will be described. The sampling data storage unit 210 stores a background feature amount (background feature amount) and a detected object feature amount (object feature amount), which are sampling units “for each pixel or block”. At this time, the object feature amount is stored together with information on elapsed time since the object having the feature amount is first detected.

  The difference calculation unit 220 compares the “feature amount for each pixel or block” from the sampling unit 200 with the “background feature amount of the corresponding pixel or block” stored in the sampling data storage unit 210. The sampling data storage unit 210 stores a feature amount for one screen. The difference calculation unit 220 receives the feature amount extracted from the image of the image data by the sampling unit 200, reads the feature amount corresponding to the position from which the sampling unit 200 has extracted the feature amount from the sampling data storage unit 210, and extracts both of them. Compare. Here, if both are within a preset threshold range, the feature quantity of the corresponding pixel or the corresponding block is the same as or similar to the background feature quantity, and therefore the difference calculation unit 220 is the background of the corresponding pixel or the corresponding block. Judge. On the contrary, if both are outside the preset threshold range, the difference calculation unit 220 determines that the corresponding pixel or the corresponding block is an object different from the background, and the “coordinate information” of the corresponding pixel or the corresponding block. And “elapsed time information” are output to the object data aggregating unit 230.

The object data aggregating unit 230 aggregates “coordinate information of“ pixel or block ”in which the object exists” from the difference calculating unit 220, and among the pixels or blocks in which the object exists, mutually adjacent pixels or blocks Are combined to generate shape information for each object. Further, the object data aggregating unit 230 aggregates elapsed time information of “pixels or blocks” in which an object exists, and generates elapsed time information since the first detection for each object. The object data aggregation unit 230 outputs the “shape information and elapsed time information” for each object to the state analysis unit 130 as object information for each object.
On the other hand, the “feature quantity for each pixel or block” from the sampling unit 200 is also passed to the sampling data update unit 240 via the difference calculation unit 220. The sampling data update unit 240 updates the “background feature amount and object feature amount for each pixel or block” stored in the sampling data storage unit 210. Hereinafter, the detailed operation of the sampling data updating unit 240 will be described with reference to FIG.

  FIG. 3 is a flowchart for explaining an example of the operation of the sampling data update unit 240. First, the sampling data update unit 240 determines whether or not the feature amount from the difference calculation unit 220 is similar to the background feature amount stored in the sampling data storage unit 210 (step S300). As a result of the determination, if the feature amount from the difference calculation unit 220 is similar to the background feature amount stored in the sampling data storage unit 210, the feature amount from the difference calculation unit 220 is that of the background. Therefore, the process is terminated without updating the feature amount. On the other hand, if the feature amount from the difference calculation unit 220 is not similar to the background feature amount stored in the sampling data storage unit 210, the feature amount from the difference calculation unit 220 is that of the object, so The process proceeds to S310.

  In step S310, the sampling data update unit 240 determines whether the feature amount from the difference calculation unit 220 is similar to the object feature amount stored in the sampling data storage unit 210. As a result of the determination, if the feature quantity from the difference calculation unit 220 is not similar to the object feature quantity stored in the sampling data storage unit 210, the feature quantity from the difference calculation unit 220 has been detected so far. It is of a new object that is different from the object that has been made. Therefore, the sampling data update unit 240 replaces the object feature amount stored in the sampling data storage unit 210 with a new feature amount from the difference calculation unit 220 (step S330). At the same time, the sampling data update unit 240 resets the elapsed time information from when the object having the feature amount is first detected.

  On the other hand, when the feature amount from the difference calculation unit 220 is similar to the object feature amount stored in the sampling data storage unit 210, the feature amount from the difference calculation unit 220 is the same as that of the same object as before. Is. Therefore, the sampling data update unit 240 increments the elapsed time information of the object feature amount in the sampling data storage unit 210 (step S320). Here, when the elapsed time information of the object feature amount continues to increase, similar feature amounts are continuously input, and the same object continues to exist at the same pixel position or the same block position. (That is, when the object is stationary on the spot). When the stationary state of the object continues for a long time, it is necessary to replace the portion of the stationary object in the image with a new background image in order to appropriately detect a new change occurring in the image thereafter.

  Therefore, in the subsequent step S340, the sampling data updating unit 240 backgrounds the object when the object detected by the difference calculation unit 220 satisfies a predetermined backgrounding condition. For example, the sampling data update unit 240 determines whether or not the elapsed time information of the object feature amount in the sampling data storage unit 210 has exceeded a preset “time threshold for background”. As a result of this determination, if the elapsed time information of the object feature amount is within the time threshold, it is not necessary to background the object of the object feature amount, and the processing is ended as it is. On the other hand, if the elapsed time information of the object feature amount exceeds the time threshold, the sampling data update unit 240 replaces the background feature amount stored in the sampling data storage unit 210 with the object feature amount information, and Information on the object feature amount is reset (step S350). For example, if the background feature amount before replacement is 500 and the object feature amount is 300, the background feature amount after replacement is 300. Further, the sampling data updating unit 240 outputs the “coordinate information and elapsed time information” of the “pixel or block” converted to the background to the backgrounded object data aggregating unit 250 and ends the processing (step S360).

  The background object data aggregating unit 250 aggregates “coordinate information of“ pixel or block ”in which a background object exists” from the sampling data updating unit 240. Then, the background object data aggregating unit 250 generates shape information for each object by linking pixels or blocks adjacent to each other among the pixels or blocks in which the background object exists. The background object data aggregating unit 250 aggregates the elapsed time information of “pixels or blocks” in which the background object exists, and generates the elapsed time information after the first detection for each background object. To do. The background object data aggregation unit 250 outputs the “shape information and elapsed time information” for each background object to the state analysis unit 130 as background object information for each background object.

Next, an example of the configuration and operation of the state analysis unit 130 will be described with reference to FIGS. 4 and 5.
FIG. 4 is a block diagram illustrating a detailed configuration of the state analysis unit 130.
The state analysis unit 130 includes an analysis rule storage unit 400, an analysis rule application unit 410, a backgrounded object storage unit 420, and a backgrounded object search unit 430. The state analysis unit 130 receives “object information and background object information” from the object detection unit 120 and outputs analysis information to the analysis result output unit 140 as a result of applying the analysis rule. The state analysis unit 130 is realized using, for example, a data processing device that operates under program control.

First, the background object storage unit 420 will be described. The backgrounded object storage unit 420 stores the backgrounded object information output from the object detection unit 120. The backgrounded object information stored in the backgrounded object storage unit 420 is deleted when the backgrounded object information having the same coordinates and shape values is input to the backgrounded object storage unit 420.
The backgrounded object search unit 430 searches the backgrounded object storage unit 420 using “values such as coordinates, shape, and elapsed time” specified from the analysis rule application unit 410 as keys, and the corresponding backgrounded object All or part of the information is returned to the analysis rule application unit 410. The analysis rule applying unit 410 uses the coordinates, shape, elapsed time, etc. of the object information being processed as a key. In the analysis rule storage unit 400, analysis rules used in an analysis rule application unit 410 described later are set and stored in advance. Here, the analysis refers to determining a change on the screen as a state change generated in the object. For example, when an object newly detected on the screen continues to exist in the same place for a certain time or longer (stills for a predetermined time or longer), the analysis rule application unit 410 leaves the object brought to that place. Analyzes such as judging. The analysis rule is a criterion for determining the state of such an object, and is expressed by a range of values of the coordinates of the object and the elapsed time, for example.

The analysis rule application unit 410 uses the analysis rule stored in the analysis rule storage unit 400 and the background object search result by the background object search unit 430 for the object information from the object detection unit 120. Apply and perform analysis processing. Hereinafter, the detailed operation of the analysis rule applying unit 410 will be described with reference to FIG.
FIG. 5 is a flowchart for explaining an example of the operation of the analysis rule application unit 410.
First, the analysis rule application unit 410 searches for backgrounded object information using the backgrounded object search unit 430 (step S500).

  Then, the analysis rule applying unit 410 determines whether there is backgrounded object information in the backgrounded object storage unit 420 based on the search result of the backgrounded object search unit 430 (step S510). If there is no background object information as a result of this determination, the analysis rule application unit 410 determines whether there is object information from the object detection unit 120 (step S530). If there is no object information as a result of this determination, the processing is terminated as it is. On the other hand, if there is object information, the analysis rule applying unit 410 applies the analysis rule to the object information and analyzes the state of the corresponding object (step S540). For example, the analysis rule application unit 410 analyzes whether an object has been left. Then, the analysis rule application unit 410 outputs analysis information that is an analysis result of the state of the object to the analysis result output unit 140 (step S550).

If it is determined in step S510 that there is background object information, the process proceeds to step S520. Then, the analysis rule application unit 410 determines whether or not the object information from the object detection unit 120 corresponds to the backgrounded object information stored in the backgrounded object storage unit 420. Whether or not the object information corresponds to the background object information is determined by, for example, whether or not the coordinates and the shape value are equal.
If the background object information does not correspond to the object information as a result of this determination, the analysis rule applying unit 410 determines that the object actually exists in the image. On the other hand, when the background object information corresponds to the object information, the analysis rule applying unit 410 determines that there is actually no object in the image. Since the difference between the feature amount of the input image and the background feature amount stored in the sampling data storage unit 210 is within the threshold when the object is turned into the background in the process of step S350 in FIG. It will not be detected. As a result, the object information of the object is not output to the object detection unit 120, so there is no object information corresponding to the background object information. After that, when the object that has become the background moves and the portion (original background) that is behind the object becomes visible, the difference between the background feature amount of the portion and the feature amount of the input image is And is output as object information. In this part, the coordinate value corresponds to the shape value. For example, when a box is left on the floor and the box becomes a background, the background floor is hidden behind the object box, but when the box is removed, the floor is visible again. It becomes like this. Here, the box corresponds to the “coordinates and shape on the screen” of the floor surface behind the box. Since this corresponding object is the original background, it is determined that the object does not exist.

  From the above, if the background object information and the object information correspond in step S520, the process is terminated as it is because no object actually exists in the image. On the other hand, if the background object information does not correspond to the object information, the process proceeds to step S540, where the analysis rule applying unit 410 applies the analysis rule to the background object information and the object information, Analyze the condition. For example, it is analyzed whether the object has been taken away. Then, the analysis rule application unit 410 outputs analysis information that is an analysis result of the backgrounded object state to the analysis result output unit 140 (step S550).

As described above, in the present embodiment, after backgrounding an object that has been stationary for a long time in an image, shape information and elapsed time information of the backgrounded object are generated as backgrounded object information, and a backgrounded object storage unit 420 is stored. Then, the object information of the input image is compared with the stored background object information, and if there is no object information corresponding to the background object information, the analysis rule is applied to the background object information. Analyze the state of the backgrounded object. Therefore, even if the stationary object is turned into the background, the state analysis process can be continued assuming that the object still exists, so that there is no contradiction between the actual object state and the object state analysis result.
Although it is determined whether there is object information corresponding to the background object information, the coordinate value, and the shape value, it is not always necessary to determine whether there is object information that exactly matches these. It may be determined whether there is object information that falls within a predetermined range.

  In the present embodiment, for example, an example of an image input unit is realized by using the image input unit 110. In addition, for example, the sampling data update unit 240 performs steps S330 and S350 in FIG. 3 to realize examples of the object detection unit and the backgrounding unit, respectively, and the determination is made by using the backgrounded object search unit 430. An example of means is realized. For example, an example of a storage unit is realized by using the background object storage unit 420.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. The present embodiment and the first embodiment are different in parts for generating and searching for background object information, but the other parts are the same. Therefore, in this embodiment, about the same part as 1st Embodiment, the detailed description is abbreviate | omitted by attaching | subjecting the code | symbol attached | subjected to FIGS. 1-5, etc., and only a different part from 1st Embodiment. Will be explained.
The block diagram illustrating the configuration of the image processing apparatus according to the present embodiment includes the object detection unit 120 illustrated in FIG. 1 as the object detection unit 125 illustrated in FIG. 6 and the state analysis unit 130 illustrated in FIG. 1 as illustrated in FIG. Each is replaced with the analysis unit 135.
The object detection unit 125 of the present embodiment has a sampling data storage unit 600 and a sampling data update unit 610 instead of the sampling data storage unit 210 and the sampling data update unit 240 in the object detection unit 120 of the first embodiment. . In addition, the object detection unit 125 according to the present embodiment further includes a reference background difference calculation unit 620 before the background object data aggregation unit 250.

  The sampling data storage unit 600 stores a reference background feature amount in addition to the sampling data (background feature amount and object feature amount) stored in the sampling data storage unit 210 in the first embodiment. Here, the reference background is a background in which no object is detected. For example, when the image processing apparatus 100 is activated, the object detection unit 120 automatically generates and stores the reference background feature amount. However, it is not always necessary to do this, and a background feature amount serving as a reference may be generated and stored according to a user instruction. The reference background feature amount is output to the reference background difference calculation unit 620 described later together with the background feature amount at that time. In the following description, “reference background feature amount” is referred to as “reference background feature amount” as necessary.

The operation of the sampling data update unit 610 is an operation obtained by deleting the process of step S360 of FIG. 3 from the operation of the sampling data update unit 240 of the first embodiment.
The reference background difference calculation unit 620 calculates a difference between the reference background feature amount from the sampling data storage unit 600 and the background feature amount at that time. The difference calculation method is the same as that of the difference calculation unit 220. When the difference falls outside the threshold range, the reference background difference calculation unit 620 outputs “coordinate information and elapsed time information” of the corresponding pixel or block to the subsequent background object data aggregation unit 250. Then, the backgrounded object data aggregation unit 250 generates backgrounded object information.

The object detection unit 125 generates background object information each time image data is input. For this reason, the state analysis unit 135 of the present embodiment does not have the background object storage unit 420. Therefore, the backgrounded object search unit 700 directly searches for the backgrounded object with respect to the backgrounded object information from the object detection unit 125. The analysis rule application unit 410 searches for backgrounded object information using the backgrounded object search unit 700 in step S500 of FIG. 5, but the other operations are the same as those in the first embodiment. .
As described above, in the present embodiment, backgrounded object information is generated from the difference between the reference background image in the absence of an object and the background image at that time. Therefore, background object information can be generated each time image data is input. Therefore, it is not necessary to store background object information, and the memory capacity can be reduced. In particular, the present embodiment is preferably applied to an input image in which the reference background is stable and does not vary.

  In the present embodiment, for example, an example of an image input unit is realized by using the image input unit 110. In addition, for example, the sampling data update unit 610 performs steps S330 and S350 in FIG. 3 to realize examples of the object detection unit and the backgrounding unit, respectively, and the determination is performed by using the backgrounded object search unit 700. An example of means is realized.

(Other examples)
The present invention is also realized by executing the following processing. That is, first, software (computer program) for realizing the functions of the above embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the computer program.

  100 image processing apparatus, 120 object detection unit, 130 state analysis unit

Claims (5)

An image input means for inputting an image;
Object detection means for detecting an object from the image input by the image input means by a background difference method;
When the object detected by the object detection means satisfies a predetermined backgrounding condition, a backgrounding means for setting the background of the object; and
Determination means for determining whether or not the object detected by the object detection means corresponds to the backgrounded object;
An analysis means for analyzing the state of the object detected by the object detection means and the backgrounded object,
The analysis unit analyzes the state of the backgrounded object when the determination unit determines that the object detected by the object detection unit does not correspond to the object backgrounded by the backgrounding unit. However, the image processing apparatus is characterized in that when it is determined that it corresponds, the state of the backgrounded object is not analyzed. When an object is backgrounded by the backgrounding means, the storage means stores information on the position of the backgrounded object,
The determination unit is configured such that, based on the information stored in the storage unit, the position and shape of the object detected by the object detection unit are within a predetermined range with respect to the object backgrounded by the backgrounding unit. The image processing apparatus according to claim 1, wherein it is determined whether or not there is any. An image input process for inputting an image;
An object detection step of detecting an object by a background subtraction method from the image input by the image input step;
When the object detected by the object detection step satisfies a predetermined backgrounding condition, a backgrounding step of setting the object as a background;
A determination step of determining whether or not the object detected by the object detection step corresponds to the backgrounded object;
When the determination step determines that the object detected by the object detection step does not correspond to the object backgrounded by the backgrounding step, an analysis step of analyzing the state of the backgrounded object;
An image processing method comprising: When the object is backgrounded by the backgrounding step, the storage step stores information on the position of the backgrounded object,
In the determination step, based on the information stored in the storage step, the position and shape of the object detected by the object detection step are within a predetermined range with respect to the object backgrounded by the backgrounding step. 4. The image processing method according to claim 3, wherein it is determined whether or not there is any. An image input process for inputting an image;
An object detection step of detecting an object by a background subtraction method from the image input by the image input step;
When the object detected by the object detection step satisfies the backgrounding condition, the backgrounding step of setting the object as a background;
A determination step of determining whether or not the object detected by the object detection step corresponds to the backgrounded object;
When the determination step determines that the object detected by the object detection step does not correspond to the object backgrounded by the backgrounding step, an analysis step of analyzing the state of the backgrounded object;
A computer program for causing a computer to execute.

Images