JP2004064438A - Monitoring system and monitoring method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a monitoring system which does not need constantly monitoring by a supervisor visual observation and to provide a method for monitoring. <P>SOLUTION: The monitoring system includes an extracting unit (102) for extracting object information for specifying an object from a video, a retrieving unit (105) for retrieving the object coincident with retrieving conditions based on the extracted object information, and a display unit (106) for reproducing and displaying the video projected in at least one or more frames of the retrieved object. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a monitoring system and a monitoring method for monitoring an object included in an image captured by a camera or the like.
[0002]
[Prior art]
Conventionally, the monitoring system is composed of dedicated equipment, so expensive dedicated lines are needed to connect the system components, and because monitoring personnel have been conducting monitoring work visually, roads and power plants have to be monitored. It was mainly used for infrastructure facilities, large-scale factories, and other specialized equipment and applications that could require the cost of a supervisor.
[0003]
However, the demand for low-cost surveillance systems is rapidly increasing due to the recent instability of social situations and the increase in crimes such as terrorism such as burglars and vandalism. In response to such demands, there have been published many monitoring systems that can use consumer devices or use a public line such as the Internet.
[0004]
[Problems to be solved by the invention]
However, even in the above-described new monitoring system, the monitoring work for detecting an abnormality is performed visually as in the past, and the burden on the monitoring personnel was large as before. In particular, in a small-scale monitoring system, it is difficult to arrange a dedicated monitoring person in terms of cost, and in many cases, visual abnormality detection cannot be performed over the entire operation time of the camera.
[0005]
An object of the present invention is to provide a monitoring system and a monitoring method that do not require visual monitoring by a monitor at all times.
[0006]
[Means for Solving the Problems]
In order to solve the above problems and achieve the object, the present invention uses the following means.
[0007]
A system for monitoring an object included in a video,
Extracting means for extracting object information for specifying the object from the video,
Search means for searching for an object that matches a search condition based on the object information extracted by the extraction means,
Display means for reproducing and displaying a video in which the object searched for by the search means is shown in at least one frame,
A monitoring system comprising:
[0008]
A method of monitoring an object included in a video,
Extracting object information identifying the object from the video;
Searching for an object that matches the search condition based on the extracted object information;
Reproducing and displaying an image in which the searched object is reflected in at least one frame;
A monitoring method comprising:
[0009]
The present invention extracts object information that specifies an object included in a video, searches for an object to be monitored based on the object information, and reproduces and displays a video in which the searched object is reflected in at least one frame. As a result, it is not necessary for the observer to always visually detect the abnormality. For this reason, even when monitoring a large number of cameras with one terminal, it is only necessary to monitor only objects that match the search conditions in the search on the terminal, and the efficiency of monitoring is greatly improved. Even when monitoring cannot be performed in real time, by storing the image of the object and the object information, the image of the object can be automatically displayed without the need to search for the image of the object.
[0010]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of a monitoring system and a monitoring method according to the present invention will be described with reference to the drawings.
[0011]
First Embodiment FIG. 1 is a diagram showing a configuration of a monitoring system according to a first embodiment of the present invention. The system includes a video input unit 101, an object extraction unit 102, an object information storage unit 103, a video storage unit 104, an object search unit 105, and a video display unit 106. These units are connected via general-purpose signal lines and communication lines. Note that the object extraction unit 102 and the object search unit 105 may be realized as software instead of hardware.
[0012]
The video input unit 101 includes, for example, a video camera, a digital still image camera that captures images at regular intervals, and the like. Although only one video input unit 101 is shown for convenience of description, a plurality of video input units may be provided.
[0013]
The object extraction unit 102 receives a video signal from the video input unit 101, extracts an object reflected in the video, creates object information (see FIG. 4) capable of specifying the extracted object, and transmits the object information to the object information storage unit 103. I do. The object information includes spatio-temporal region information describing motion features such as the shape and trajectory of the object, representative image information describing image features such as the color and texture of the object, and attribute features such as the name and classification of the object and the camera number. At least one piece of attribute information that describes. Here, the object is a group of regions in the video, such as a person, an animal, a car, a road, a river, or a part thereof (a person's face, a car tire), or a set thereof (a row of people). , Car traffic, etc., as long as it can be grasped as what should be monitored. The extraction of the object may be performed manually by the user on the screen displaying the video, or automatically performed by image processing such as background subtraction, inter-frame difference, stereo vision, template matching, color histogram extraction, and the like. You may do it. Further, it may be performed using another external device such as an infrared camera or a human sensor. Object extraction may be performed by combining these methods.
[0014]
The object information accumulation unit 103 accumulates and saves the object information created by the object extraction unit 102, and includes, for example, a hard disk, an optical disk, and a semiconductor memory.
[0015]
The video storage unit 104 receives a video signal from the video input unit 101 and stores and stores the video signal. The video storage unit 104 includes, for example, a hard disk, an optical disk, a semiconductor memory, and a video tape recorder. The video signal to be stored is not limited as long as the object reflects one or more frames, and all the video signals received from the video input unit 101 may be stored. Alternatively, the video signal may be stored only for the time interval during which the object exists.
[0016]
The video storage unit 104 may perform some thinning or compression to reduce the amount of stored data. For example, by lowering the frame rate or converting to a video compression format such as MPEG-2 or MPEG-4 and storing it, more video signals can be stored with the same capacity.
[0017]
If spatio-temporal region information or attribute information has been created as object information, the video signal may be subjected to some processing in accordance with the object information and stored. For example, using the spatio-temporal area information, create and save a video signal that is cut out only around the object, border the object outline with a specified color line, write the trajectory of the object with a specified color Alternatively, processing such as lowering the brightness of a portion other than the object or painting out the portion may be performed. In addition, the classification, the name, and the feature amount of the object may be superimposed on the video as character information using the attribute information, or the statistics may be displayed. A combination of the two may be used to switch the method of processing the object part depending on the feature amount of the attribute information.
[0018]
The object search unit 105 searches the object information stored in the object information storage unit 103 for object information corresponding to the search condition specified by the user. Search conditions may be registered in advance, or may be input by the user for each search. The above methods may be combined such that the user selects for each search from a plurality of search conditions registered in advance.
[0019]
The user designates one or more objects to be monitored based on (the object information of) the search result. When an object is designated, the reproduction information describing the designated object ID, the reproduction start time, the reproduction end time, and the like is transmitted to the video storage unit 104, and the video signal of the object specified by the reproduction information is transmitted to the video display unit 106. Let it.
[0020]
The search conditions include, for example, conditions relating to image features such as the color and texture of the object, conditions relating to movement features such as trajectory, existence time, shape and size, conditions relating to attribute features such as classification, name, camera number, and shooting date and time. Anything may be used as long as it relates to object information. Further, a plurality of conditions may be combined and used by a logical operator such as AND, OR, NOT, and the like.
[0021]
The video display unit 106 is for reproducing and displaying the video signal of the object received from the video storage unit 104, and is, for example, a CRT. At the time of reproduction and display, the object information of the object may be read from the object information storage unit 103, and some processing may be performed on the image of the object using spatio-temporal region information and attribute information in the object information. For example, using the spatio-temporal area information to cut out and save only the periphery of the object, trimming the outline of the object, performing processing such as reducing or filling the brightness of parts other than the object, and using attribute information The classification or name of the object may be superimposed on the video, or the processing method may be switched according to the feature amount.
[0022]
Also, the images of the objects may be combined so that the images of a plurality of objects can be viewed simultaneously.
[0023]
When there are a plurality of video input units 101, the object extraction units 102 and the object information storage units 103 may be prepared by the number of the video input units 101. May be processed from the video input unit 101, or a combination thereof.
[0024]
Next, the operation of the present embodiment will be described. FIG. 2 is a flowchart illustrating an example of the flow of the object extraction process. The extraction process is executed at certain time intervals such as every frame or every certain time.
[0025]
In step S21, a video signal is input from the video input unit 101 such as a camera (video input step).
[0026]
In step S22, it is determined whether a video input exists (video input determination step). If there is no video input, the object extraction processing ends. If there is a video input, an object is extracted in step S23.
[0027]
In the object extraction step S23, an object is extracted from the video signal input in the video input step S21. The object may be extracted manually by the user on the video display screen or automatically by image processing such as background subtraction, inter-frame difference, stereo vision, template matching, color histogram extraction, and the like. good. Further, it may be performed by using another external device such as an infrared camera or a human sensor. In addition, object extraction may be performed by combining user input, image processing, and external devices.
[0028]
In step S24, at least one of spatio-temporal region information, representative image information, and attribute information of the object extracted in the object extraction step S23 is created and stored as object information (see FIG. 3) (object information storage step). ).
[0029]
In step S25, the video signal acquired in the video input step S21 is stored and stored (video storage step). The image storage step S25 may be executed immediately after the image input step S21. However, the image storage step S25 is executed after the object information is created and stored, so that the object information generated in the object information storage step S24 is used to perform some processing on the image. Can be applied. For example, using the spatio-temporal area information to cut out and save only the periphery of the object, trimming the outline of the object, performing processing such as reducing or filling the brightness of parts other than the object, and using attribute information It is possible to superimpose the classification and name of the object, and to switch the processing method depending on the feature amount.
[0030]
FIG. 3 shows an example of the data structure of the object information stored in the object information storage unit 103. As shown in FIG. 3, the object information includes an object ID 401, a start time 402, an end time 403, spatiotemporal area information 404, representative image information 407, and attribute information 408. The spatio-temporal region information 404 includes a shape flag 405 and motion data 406, and the attribute information 408 includes one or more attribute names 409 and attribute values 410.
[0031]
These data may be collected as one file or may be divided into separate files. Further, the spatiotemporal region information 404, the representative image information 405, and the attribute information 406 can be omitted, respectively, or a plurality of them can be provided. The arrangement order of the spatiotemporal area information 404, the representative image information 405, and the attribute information 406 may be any order.
[0032]
The object ID 401 is an identification number uniquely assigned to distinguish the extracted object.
[0033]
The start time 402 and the end time 403 are the time when the object first appears and the time when the object last disappears, respectively.
[0034]
The spatio-temporal area information 404 holds spatio-temporal area information where an object exists, and may be any information as long as the time at which the object exists and its position and shape can be known. . For example, it is possible to use an alpha map representing a moving image in which the place where the object exists is “1” and the place where the object is not “0”, or use a data format such as MPEG-7 Spatial Temporal Locator. Is also good. The Spatial Temporal Locator approximates the object shape of each frame in the video with a rectangle, an ellipse, or a polygon with an arbitrary number of vertices. Expressed as a trajectory in the direction, a vertex trajectory is approximated by a spline function, and a figure trajectory expressed as a parameter of the approximation function, and an object shape in any one frame is approximated by a rectangle, an ellipse, or a polygon having an arbitrary number of vertices. As a reference graphic, any one of Parameter Trajectory in which the movement or shape change of the object is represented by a transformation parameter such as an affine transformation from the reference graphic, and the trajectory of the transformation parameter in the time direction is approximated by a spline function and represented as a parameter of the approximation function There is. The Spatial Temporal Locator is characterized by being able to represent an object shape and its movement with a much smaller amount of data than an alpha map.
[0035]
The shape flag 405 is a flag for indicating in what format the spatio-temporal area information is stored. A data format such as an alpha map or a Spatial Temporal Locator, a shape of an object, a function approximation method of a feature amount trajectory, and the like are described.
[0036]
The motion data 406 is data of motion in a spatiotemporal area. If it is an alpha map, it is moving image data, and if it is a Spatial Temporal Locator, it is a function parameter approximated by a function.
[0037]
The representative image information 407 is image information of one frame or a plurality of frames in which the object exists. This is information for the user to visually specify the object, such as image information such as BMP or JPEG, a pointer indicating the frame of the video signal, or the like. The image information may be the entire image of the frame or a part thereof. For example, when the position and shape in the frame are known, only the periphery of the object may be cut out from the image and used as representative image information. In addition, some processing may be performed on the image, such as lowering the luminance of the area other than the object area, or bordering the outline of the object.
[0038]
Any frame may be extracted as the representative image information as long as the object can be easily identified. For example, there is a method in which a center time of a time period in which an object exists is set as a representative frame.
[0039]
The attribute information 408 is for describing what characteristics the object has. For example, classification such as whether the object is a car, a person, or an animal, a feature amount such as a color, a shape, and a size, external device information such as a sound and a sensor acquired simultaneously with a video signal, A link to related information is described.
[0040]
The attribute information 408 includes one or more attribute names 409 and attribute values 410. The attribute name 409 and the attribute value 410 have a one-to-one correspondence. The attribute name 409 and the attribute value 410 may be described in any format. For example, MPEG-7 format which is an international standard, various database formats, text format, and the like can be used.
[0041]
FIG. 4 is a flowchart illustrating an example of the flow of an object search process. The process is executed for each object search.
[0042]
In step S31, search conditions for an object are input (search condition input step). Search conditions may be registered in advance, or may be manually input by the user for each search. The above methods may be combined such that the user selects for each search from a plurality of search conditions registered in advance.
[0043]
In step S32, each object information stored in the object information storage unit 103 is compared with the input search condition to determine whether the object information matches the search condition (object information matching step). The representative image information and the attribute information of the object determined to be matched are displayed on a CRT or the like or left in a log so that the user can browse the search result.
[0044]
In step S33, it is determined whether or not matching has been performed for all object information stored in the object information storage unit 103 (matching end determination step). Steps S32 and S33 are repeated until the determination of all object information is completed.
[0045]
In step S34, it is specified which of the objects that match the search condition is to reproduce the video signal (object specifying step). Basically, the user who browsed the search result specifies the object to play the image of the object, but if there are few search results or if you always want to see the image of the object at the latest time among the search results, the system It may be specified automatically.
[0046]
The specification may be a single object or a plurality of objects, and the reproduction start time, the reproduction end time, and the reproduction option can be simultaneously specified for the video signal of each object. These pieces of information are used as object reproduction information (see FIG. 5) when reproducing a video signal of an object. In the reproduction option, it is possible to specify a special effect method such as edging an object outline, and how to reproduce a video signal of an object when reproducing a plurality of video signals.
[0047]
In step S35, the video signal of the object is reproduced and displayed on a CRT or the like using the object reproduction information specified in the object specification step S34 (video reproduction step). At the time of reproduction, a video signal is processed and displayed based on the reproduction start time, reproduction end time, and reproduction option of the object information. FIG. 5 shows an example of a data structure of the object reproduction information designated at the time of reproducing the video signal. As shown in FIG. 5, the object reproduction information includes a designated object ID 501, a reproduction start time 502, a reproduction end time 503, and a reproduction option 504. When a plurality of objects are designated by one object reproduction information, the designated object ID 501, the reproduction start time 502, the reproduction end time 503, and the reproduction option 504 are each plural.
[0048]
The designated object ID 501 is an identification number uniquely assigned to identify an object to be reproduced and displayed. The designated object ID is the same as the object ID 401 in the object information in FIG. 4, and by specifying the object ID, the image of the object can be uniquely specified.
[0049]
The reproduction start time 502 is a start time when reproducing the video of the object. When omitted, reproduction is performed from the appearance time of the object (the head time 402 in the object information).
[0050]
The reproduction end time 503 is an end time when reproducing the video of the object. When omitted, the reproduction is performed until the disappearance time of the object (the end time 403 in the object information).
[0051]
The reproduction option 504 is an option for how to reproduce the image of the object. In the reproduction option 404, for example, a special effect such as a screen size, a reproduction rate, and an object border can be designated. When a plurality of objects are designated, a method of reproducing the plurality of objects can be designated, such as by placing the plurality of objects on one screen and reproducing them, or by designating them in order.
[0052]
As described above, according to the present embodiment, at least one of spatiotemporal region information, representative image information, and attribute information for specifying an object such as a person or a car included in a video acquired by the video input unit 101 such as a camera. The object extraction unit 102 extracts object information composed of at least one object and searches for an object to be monitored using the extracted object information. Play and display the video of a certain object.
[0053]
Thus, it is not necessary for the observer to always visually detect the abnormality. Even when monitoring a large number of cameras with one terminal, it is sufficient to check only the matching objects in the search on the terminal, and the monitoring efficiency is greatly improved. Even in a case where monitoring cannot be performed in real time, it is possible to perform monitoring without having to search for a video by storing the video of the object and the object information.
[0054]
The present invention is not limited to the embodiments described above, and can be implemented with various modifications. For example, the present invention provides a computer-readable recording medium on which a program for causing a computer to execute predetermined means, causing a computer to function as predetermined means, or causing a computer to realize predetermined functions is recorded. Can also be implemented.
[0055]
【The invention's effect】
As described above, according to the present invention, it is possible to search for an object to be monitored using object information of an object included in a video and reproduce and display the video of the object. As a result, the monitoring operation can be performed without viewing all the images, and the efficiency of the monitoring operation can be greatly increased.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an example of a configuration of a monitoring system according to an embodiment of the present invention.
FIG. 2 is an exemplary flowchart illustrating an example of an object extraction processing procedure according to the embodiment.
FIG. 3 is an exemplary view showing an example of a data structure of object information in the embodiment.
FIG. 4 is an exemplary flowchart illustrating an example of an object search processing procedure according to the embodiment.
FIG. 5 is an exemplary view showing an example of a data structure of object reproduction information in the embodiment.
[Explanation of symbols]
101 video input unit 102 object extraction unit 103 object information storage unit 104 video storage unit 105 object search unit 106 video display unit

Claims (9)

A system for monitoring an object included in a video,
Extracting means for extracting object information for specifying the object from the video,
Search means for searching for an object that matches a search condition based on the object information extracted by the extraction means,
Display means for reproducing and displaying a video in which the object searched for by the search means is shown in at least one frame,
A monitoring system comprising: The object information is at least one of spatio-temporal region information describing the shape and motion of the object, representative image information describing image features such as color and texture of the object, and attribute information describing the classification and feature of the object. With
The monitoring system according to claim 1, wherein the search unit searches for an object using at least one of the spatiotemporal area information, the representative image information, and the attribute information. The spatio-temporal area information approximates the object shape of each frame in the video with a rectangle, an ellipse or a polygon having an arbitrary number of vertices. The monitoring system according to claim 2, wherein each of the vertices is represented as a trajectory in a time direction, and a parameter of an approximation function when the vertex trajectory is approximated by a spline function is described. The spatio-temporal region information is obtained by approximating an object shape in an arbitrary frame in a video with a rectangle, an ellipse, or a polygon having an arbitrary number of vertices as a reference figure, and performing affine transformation from the reference figure on the movement or shape change of the object. 3. The monitoring system according to claim 2, wherein the monitoring system is represented by a conversion parameter, and describes a parameter of an approximation function when the trajectory of the conversion parameter in the time direction is approximated by a spline function. 5. The display device according to claim 2, wherein the display unit processes and displays an image in which the searched object is reflected in at least one frame using at least one of the spatiotemporal region information and the attribute information. 6. A monitoring system according to claim 1. 6. The image processing apparatus according to claim 1, further comprising a unit configured to process and store an image in which at least one frame of the searched object is reflected using at least one of the spatiotemporal region information and the attribute information. The monitoring system according to claim 1. A method of monitoring an object included in a video,
Extracting object information identifying the object from the video;
Searching for an object that matches the search condition based on the extracted object information;
Reproducing and displaying an image in which the searched object is reflected in at least one frame;
A monitoring method comprising: 8. The monitoring method according to claim 7, wherein the displaying step processes and displays an image in which the searched object is shown in at least one frame by using at least one of the spatiotemporal region information and the attribute information. 9. The method according to claim 7, further comprising the step of processing and storing an image in which at least one frame of the searched object is reflected using at least one of the spatiotemporal region information and the attribute information. Monitoring method.

Images