SE534706C2 - Method for video monitoring of an area of behavioral analysis and computer systems for implementing the method - Google Patents

Description

20 25 30 35 534 TÜE 2 stay close to crowds for a long time without doing what is normally done in the area, e.g. shop in a mall or travel away from a shopping mall. Pickpockets often operate in pairs or three and three and the method can detect that three people consistently stay close to each other without being so close that they seem to be in the same company. The method can also detect interesting events indirectly e.g. in that no one wants to go near a certain person in situations where people do not normally avoid others. The solution solves the current problem of video surveillance of an area and analyzing events in it and providing various forms of alarms and decision support to an operator by being carried out in the manner set out in the following independent patent claims.

The invention will be described in the following with reference to the accompanying drawing, in which fi g. 1 fi g. 2 shows a schematic diagram of a behavior analysis filter in Figure 1. shows a schematic diagram of the method and The system at the time of use uses one or your camcorders that deliver a picture sequence to a computer system. The computer system analyzes the video sequences automatically frame by frame, preferably time-synchronized if several cameras are used.

In addition to camcorders, different individuals can be located with other known sensors and the system can also use this information in its analysis. The analysis is improved if _ there is information about individuals' group affiliation and definitions of behavior classes that have been entered into the system by an operator.

The result of the analysis is a situational picture of an area. It is a continuously updated, formalized description of the individuals seen by one or more of your camcorders and their movement behaviors. The result can be presented as a descriptive text with characteristics of individuals, groups and the whole area and also include any class affiliation and any anomaly. The result can be presented to the operator or forwarded to another system for further use. 10 15, 20 25 30 35 534 706 3 A main point in the invention is that the analysis takes place in three blocks. First, individuals are analyzed individually. An important feature is that this individual analysis takes place in two parts. First, one tries to associate each observation of an individual with an individual who is known in a previous image or, if this is not the case, the individual is stated to be a new individual. For this purpose, the properties used are extracted and used to determine the state variables needed to perform the association. Depending on which state variables one is interested in in order to be able to analyze possible group affiliation and the development of the situation in the whole area, additional characteristics for the various individuals are then extracted. When a new observation has been associated with a previously known individual, the state variables that characterize the individual can be updated, which includes various forms of information about the individual's previous history stored in the previous image. The result of the analysis is a description of the individuals moving in the area.

When the individual analysis is completed, groups of individuals are analyzed based on the results of the individual analysis. The analysis of groups takes place in the same way as for individuals, with the difference that groups can be detected either automatically by some common property in the form of a certain value of one or tillstånd your state variables or specified by enumerating the individuals who are part of the group. The enumeration is done by an operator or another part of the system. The division of individuals into groups does not have to be mutually exclusive, an individual can be included in your groups. The result of the analysis is a description of the groups moving in the area.

Finally, the entire area is analyzed and provides a situation picture based on the results of individual and group analysis and on characteristics that are extracted to characterize the situation in the entire area.

The mentioned variables mentioned are usually numerical, but can be interpreted into a description in words when presented. The same applies to events that can be described as numerical changes in one or fl your state variables, but can be described in words in the presentation. 10 15 20 25 30 534 7GB 4 The value of each state variable is described by a probability distribution over the value space. In this way, one can express different degrees of certainty in the assessment of which value applies. One can also give a certain value all probability, which is a way of expressing that a specific value applies.

The person skilled in the art understands to select state variables based on the situation, in order to describe relevant properties of the moving objects. Anyone wishing to study the question is referred to A. Yilmaz, O. Javed and M. Shah, “Object tracking: a survey,” ACM Computing Surveys, Vol. 38, no. 4, Article 13, Dec. 2006.

For an individual, there are some common examples of condition variables: Position, appearance, posture, bodily movement pattern and spatial movement, speed and direction. For a group, some common examples of state variables are: Individual group, property and values that they gruppen nier the group, position and spatial motion, velocity and direction. For the whole area, some common examples of state variables are: Number of individuals, number of groups, spatial movement pattern and individual analysis. Individual analysis, see individual analysis in Figure 1. consists of finding, following and characterizing the individuals. In this analysis, no account is taken of the possible part of the individuals in any group. Detection, extraction and tracking of moving objects are well known to those skilled in the art. Anyone who wants can study the issue in e.g. A. Yilmaz et al. as above and W.

Hu, T.Tan, L. Wang and S. Maybank, “A survey on visual surveillance of object motion and behaviors”, IEEE Trans. on systems, man and cybemetics, vol. 34, no. 3, August 2004.

The various parts of the block of individual analysis in Figure 1 are presented below. The analysis is done image by image and in two main parts. The first part consists of detection, extraction and tracking and results in further observations being associated with significant traces. Traces may also have been added and disappeared. The second part of the analysis consists of extracting additional characteristics and characterizing the condition and events in order to update the characterization of the individuals. Input data, 1 in Figure 1, is a sequence of images processed one by one. Output 2 in Figure 1, is the trace and characterization of the individuals moving in the scene. 10 15 20 25 30 35 534 7GB Detection skeri block D, Detection of moving objects can be done Lex. in that they differ from the background.

Those skilled in the art are well acquainted with such background detection. Those interested can study the question in K. Kim, T. H. Chalidabhongse, D. Harwood and L. Davis, “Real-time foreground-background segmentation using codebook model”. Real-time imaging, vol. 11, no. 3, pp. 172-185, 2005 and in C. Stauffer and W. E. L. Grimson, “Learning patterns of activity using real-time tracking”. IEEE Trans. on pattern analysis and machine intelligence, voi. 22, no. 8, August 2000.

Each observation of an object (or a dei of an object or objekt your objects) is a coherent area of the image that is pointed out with a silhouette or otherwise.

The detection is done for each sensor separately and olika your different algorithms can be used for each sensor. The detection module produces a number of hypotheses for where moving objects can be found in the current image.

Extraction of properties takes place in block X., Extraction of properties needed for the addition such as. position, appearance and shape. The extraction may look different depending on the detection algorithm used. The result must be comparable with the result from the other extraction modules in the tracking module.

Tracking takes place in block T. Each observation of a moving object is associated with an individual (object track). in many cases it is a one to one correspondence between observations and known individuals, but it can happen both that an individual is described by fl your observations and that fl your individuals are described by an observation. individuals can be created and disappear. An individual who has not been observed for a long time or who is known not to return will be removed. An observation that cannot be associated with an existing individual leads to the birth of a new individual. the individual is in some sense preliminary until fl your observations have been made. Extraction of additional properties takes place in block X. Extraction of additional properties can be done when the detection is associated with a known individual. Changes over time are extracted if the individual contains data on previous observations. If the individual is classified, traits related to the class can be extracted, e.g. number of wheels if it is a vehicle or length of persons. For people, e.g. characteristics that describe bodily movement.

Individual behavior analysis takes place in the block IBF In lBF, Individual behavioral alter (individual behavioral alter), the individual's characteristics are updated with another observation. Classification and anomaly detection are done to detect unwanted and unusual events.

The result from the individual analysis is that the objects that move in the area are detected and followed so it is possible to see how the objects move in the scene. The behavior of the objects is continuously characterized so that the alarm can be given for predefined events and for anomalies.

Group analysis consists of finding, following and characterizing groups of individuals. A group is defined by enumerating individuals or by some common value of some trait. The characteristics that form the basis for group division can be determined partly by the operator and partly by analysis of the individuals.

The various parts of the group analysis block layer 1 are presented in the following. The analysis takes place in a similar way as the analysis of individuals. The input data to the block are data from the individual analysis and the result is that further observations can be associated with traces of the groups. Traces of groups may appear and disappear. Input data can also be user-defined groups which in that case must be entered into the system in addition to the analysis of images.

Detection of groups takes place in block D. Groups can be specified by an operator or some other system listing the individuals included in each group. In this case, detection of individuals included in the group is trivial. Groups can also be specified by a common property provided by an operator or other system, or the detection module can automatically find groups of individuals with similar properties. Groups do not have to be mutually exclusive, but an individual can be part of several groups.

Extraction of the group's properties takes place in block X9. Once a group has been identified, the properties of the group that are needed to associate the group with one of the known groups are extracted. 10 15 20 25 30 35 534 7GB Tracking takes place in block T, Each observation of a group is associated with a known group or a new QFUPP is created. Extraction of additional properties takes place in block Xh When there is now another observation regarding the group then the group characteristics that are interesting and needed for characterization of the group and its behavior can be extracted.

Group behavior analysis takes place in the block GBF In GBF, Group behavioral alter (group behavioral alter), the group's characteristics are updated with another observation. Unwanted and unusual behaviors of the group are detected.

The result of the group analysis is that groups in the area are followed and their behavior is characterized continuously so that alarms can be given in the event of events and anomalies. The method also analyzes the individuals who move in the scene to find any groups that may be individuals who have a common goal, individuals who have similar behavior or individuals who look alike, etc.

Based on individual and group analysis and other information about the entire area, an analysis of the entire area takes place. The various parts of the Stage Analysis block, see Figure 1, are presented below. lndata is output from the individual and group analysis. This block is analogous to the second part of the individual and group analysis blocks, first properties that describe the whole area - the scene - are extracted and then the state variables that describe the scene are updated.

Extraction of properties for the whole area X, When state variables for individuals and groups have been updated, properties for the whole area that are interesting and needed to characterize the situation in the area can be extracted.

Behavior analysis over the entire area takes place in the block SBF In SBF, Behavioral alter for scene (scene behavioral alter), properties for the entire area are updated. 10 15 20 25 30 35 534 706 The result of the analysis of the whole area is a situation that is created based on the individuals who move in the area and the characterization of their behavior and the analysis of any groups and their behavior. The situation picture would e.g. be able to tell if it is your groups that behave threateningly towards others or each other or if it is only individual individuals who behave suspiciously.

The three filters (for individual, group and the whole area) used in behavioral analysis have the same structure with four components each, see figure 2. The content of the three filters differs. Each filter has a method that calculates the current state based on new observations and current time and a method that predicts the current state at a time after the last observation. There is a method that classifies the current state in either predefined classes or in classes given by current and previous data and a method that detects anomalies of the current state by keeping statistics on how often different states occur. The first part can e.g. be done with a Kalman filter that must contain prediction of the next measurement care, update of the condition and distance calculation between two conditions.

The various parts of a behavioral filter are presented below. The filter contains two databases, one with a description of property classes, 5 in Figure 2. which can be estimated from data or given from the outside by the user. The second database, 6 in Figure 2, contains a description of normal care in order to be able to detect deviations from such normal values. The description of normal values is estimated continuously from current data.

Prediction takes place in block P The current state is a function of time and of all or part of the received sensor data. In many cases, the condition changes with time and if you want the current value some time after the last observation, the current value should be predicted based on previous data and current time.

Updating of states takes place in block T The second part of the filter consists of characterizing the current state in terms of dynamically or deterministically predetermined classes. The classes can be done in terms of directly measurable properties but can also be given an explanatory description at the definition of the class. 10 15 20 25 30 35 534 7GB Classification takes place in block C The current condition is classified in block C in Figure 2, ie. for each state vector, the class to which the state vector belongs is calculated. The definition of the classes is stored in the database 5 in Figure 2. The definition of the classes is either given by another part of the system and entered into the database from the outside (used classes) or the classes are created in block K in Figure 2. The class definition can be updated for each new state vector will then be entered into the database. The classes can be named to facilitate interpretation of the current situation. If the class division is updated dynamically, any naming must be reviewed when the classes change.

Classification of data is done to facilitate the interpretation of the current situation and support the operator's work to find connections between different individuals and events.

The method can see connections that are difficult for an operator to detect. The value of the state variables provides more detailed information but may be too extensive for an operator to interpret. Classification is done by a method known to those skilled in the art, such as e.g. K-means. An overview of this method is presented in R. O.

Duda, P. E. Hart and D. G. Stork, "Pattern Classification," 2nd Edition, Wiley, 2000.

Anomaly detection takes place in block AD. The method detects conditions that are predefined as nononal or conditions that deviate from the normal in the current sensor data. Normal behaviors can e.g. determined by statistics regarding sensor data. One can e.g. assume that a parameter is normally distributed and estimate the distribution parameters from data.

Those skilled in the art are familiar with such anomaly detection. O. Boiman and M. lrani, “Detecting irregularities in images and in video” give examples of how anomalies in movements can be detected.

The result in the form of the analysis of the invention is a situational picture of the area and can be presented in a combination of maps, sensor data and text descriptions. A map or drawing can show the area being monitored with symbols for basic conditions and current condition. Preferably, the operator is allowed to select a subset of all available information. The map can e.g. show geographical constraints and the position and field of view of sensors. Video sequences can be viewed in real time as well as data 534 706 10 from other sensors. Symbols for the current mode and analysis results can be entered in the streams.

An event log can, in chronological order or with other sorting, display a list of events. Events can e.g. get different design in the event log due to its prioritization to facilitate the operator's work. It may also be possible to let the operator freeze the image at a certain time and the possibility to go back in time to see what the situation was like at an earlier time.

Claims (2)

10 15 20 25 30 35 534 706 11 Patent claims: A method for video surveillance of an area with behavioral analysis of individuals and groups of individuals and which generates a situational picture over the entire area, comprising, for said individuals, groups and area, determining the state variables one is interested in to describe said individuals, groups and areas and their behaviors and also how said state variables depend on properties that can be extracted from an image, that each of said individuals, groups and areas is assigned such a set of state variables, that for each image in an image sequence detects the individuals present in the image and by extracting values of affected properties determines and stores the state variables needed to be able to associate an individual with the corresponding individual in any previous image in the image sequence, that when analyzing an image in the image sequence, one tries carry out the association for each individual in the image, which either succeeds, thereby obtaining states that an individual is the same as an individual in a previous image, or fails, specifying the individual as a new individual, characterized by extracting values of the properties needed to determine additional state variables that one is interested in. said individual, that based on these characteristics, and - if said each individual is associated with an individual in a previous image - previous values of state variables for this individual, said additional state variables determine and perform an individual behavioral analysis. of each individual and thereby classifies and performs anomaly detection, using the results of the behavioral analysis for each individual as input values for a group analysis, detecting for each image in an image sequence the groups present in the image and by extracting values on affected properties determines and stores the state variables needed to be able to associate a group with the corresponding group in any previous image in the image sequence, that when analyzing an image in the image sequence, one tries to carry out the association of each group in the image, which either succeeds, thereby establishing that a group 10 15 20 25 30 35 534 706 12 is the same as a group in a previous image, or fails, specifying the group as a new group, extracting values of the properties needed to determine additional state variables that one is interested in for said groups, based on these properties, and -for the groups associated with groups in a previous image -previous values of state variables for these groups, said further state variables for said groups determine and perform a group behavior analysis of said each group and thereby classify and perform anomaly detection, using the results of the behavioral analysis for each individual and the results of the behavioral analysis for said individual. each group as input values for a behavioral analysis over the whole area, to extract values of the properties needed to determine state variables which, in addition to individual and group state variables, are needed to update the situation across the area, that based on individuals and group state variables and the state variables determined specifically for updating the state image, update the state variables that describe the state image and present this state image. 2. Computer system for video surveillance of an area with behavioral analysis of individuals and groups of individuals and which generates a situational picture over the whole area, comprising that the computer system, for said individuals, groups and area, determines the state variables that are stated to be interesting to describe said individuals, groups and areas and their behaviors and also how said state variables depend on properties that can be extracted from an image, that the computer system assigns to each of said individuals, groups and areas such a set of state variables that the computer system for each image in an image sequence detects the individuals present in the image and by extracting values of affected properties determines and stores the state variables needed to be able to associate an individual with the corresponding individual in any previous image in the image sequence, that the computer system, when analyzing an image in the image sequence, try to implement the association for each e individual in the image, which either succeeds, whereby it is determined that an individual is the same as an individual in a previous image, or fails, whereby the individual is indicated as a new individual, characterized in that the computer system extracts values of the properties needed to determine additional state variables that are stated to be of interest to each individual, that the computer system is based on these properties, and - if said each individual is associated with an individual in a previous image - previous values of state variables for this individual determines and stores said further state variables, that the computer system performs an individual behavioral analysis of each individual and thereby classifies and performs anomaly detection, that the computer system uses the results of the behavioral analysis of each individual as input values for a group analysis, that the computer system for each image in an image sequence detects the groups that occur in b fire and by extracting values of affected properties determines and stores the state variables needed to associate a group with the corresponding group in any previous image in the image sequence, that the computer system, when analyzing an image in the image sequence, tries to implement the association for each group in the image, which either succeeds, determining that a group is the same as a group in a previous image, or failing, specifying the group as a new group, that the computer system extracts values of the properties needed for to determine additional state variables which are stated to be of interest to said groups, that the computer system based on these properties, and -for the groups associated with groups in a previous image -previous values of state variables for these groups, determine and store said further to stand variables for said groups, that the computer system performs a group behavior analysis of said habit group and thereby ssi fi certifies it and performs anomaly detection, that the computer system uses the results of the behavioral analysis for each individual and the results of the behavioral analysis for the said each group as input values for a behavioral analysis over the whole area, 534 706 14 that the computer system extracts values of the properties needed to determine condition variables , in addition to the state variables of individuals and groups, it is necessary to update the state image over the area, that the computer system, based on the state variables of individuals and groups and the state variables determined specifically for updating the state image, updates the state variables describing the state image and presents this state.