CN1200397C - Method for object action set-up mold - Google Patents

Abstract

An object behavior modeling method capable of efficiently modeling complex objects such as human bodies is provided. The object behavior modeling method comprises steps: (a) obtaining an optical flow vector from a video sequence; (b) using the optical flow vector to obtain the probability distribution of feature vectors of multiple video frames; (c) using the feature vector's probability distributions, for state modeling; and (d) based on state transitions, for expressing the behavior of objects in a video sequence. According to this modeling method, complex behaviors such as human activities can be efficiently modeled and recognized without segmenting objects in the fields of video indexing and recognition.

Description

Object Behavior Modeling Method

technical field

The present invention relates to object behavior modeling methods, and in particular, to an object behavior method for effectively analyzing the behavior of complex objects such as human behavior. The invention also relates to an object behavior recognition method for recognizing behavior or events of objects in a video sequence using an object behavior model modeled by the object behavior modeling method.

Background technique

Human behavior including sitting, walking, standing up, or turning in circles can be captured using a video camera and stored as digital video. After the digital video is acquired, the content of the digital video can be analyzed. For example, using stochastic models based on training data can characterize the temporal and spatial characteristics of the behavior of digital videos. These models can be used to match provided video sequences and database videos for pattern recognition. After pattern analysis, the videos can be semantically indexed using these patterns. Also in this process, semantic induction of video content can be obtained.

Traditional object behavior analysis methods can be divided into two categories. In the first method of analysis, a device designed to analyze behavior is attached to the human body and used for behavioral analysis. In the second analysis method, geometric features or images of objects are used for behavioral analysis. However, in the first method, since the device must be attached to the human body, the device attached to the human body restricts the movement of the person. Also, in the second method it is necessary to segment each individual object from the video. However, in many cases individual objects cannot be accurately segmented from the video. In particular, it is very difficult to apply the second analysis method to complex objects such as human bodies that cannot be easily segmented.

Contents of the invention

In order to solve the above problems, an object of the present invention is to provide an object behavior modeling method capable of modeling complex objects such as human bodies.

Another object of the present invention is to provide an object behavior recognition method using a behavior model modeled by an object behavior modeling method.

In order to achieve the above object of the present invention, a method for modeling object behavior is provided, comprising the following steps: (a) obtaining an optical flow vector from a video sequence; (b) using the optical flow vector to obtain feature vectors of multiple video frames Probability distribution of , wherein the feature vector is a d×L dimensional vector, d is the dimension and L is the number of pixels in a video frame or region of interest; (c) using the probability distribution of the feature vector, the state modeling; and (d) expressing the behavior of objects in a video sequence based on state transitions.

Preferably step (a) is based on affine motion estimation.

Preferably step (a) also includes sub-steps: (a-1) grouping the input video frame into a plurality of video frame groups, and dividing each video frame group into independent states; (a-2) being the video of each independent state Each video in the frame group obtains an affine motion parameter; and (a-3) obtains an optical flow vector from the affine motion parameter.

Preferably step (a-2) comprises the step of, when I represents intensity, t represents time, x represents pixel position (x, y), V represents motion vector, so that in a given video, based on the expression I _t ( x) = It _-1 (xV(x)) sum of squares of differences in pixel intensities on the object Σ( _It (x)-It _-1 (xV(x))) ² The smallest parameter is determined for motion parameter.

Preferably, step (b) comprises the step of calculating the probability distribution P(Z|Ω) according to the following formula:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}$

where P=(p ₁ , p ₂ ,...p _d ) represents the motion vector calculated at each pixel position (x, y), L represents the number of pixels in a video frame or region of interest, and d represents Dimensions, the eigenvector Z of the d×L-dimensional vector is Z=(p ¹ ₁ , p ² ₁ ,...p ^L ₁ , p ¹ ₂ , p ² ₂ ,...,p ^L ₂ , p ¹ _d , p ² _d ,..., p ^L _d ) ^T , m is the mean vector of eigenvector Z, Q is the covariance matrix of eigenvector Z, and it is assumed that eigenvector Z is provided from the observation class (observation class) Ω .

Preferably step (b) also includes the step of decomposing the covariance matrix Q according to the following formula:

Q＝ ^φΛφT

The columns of φ are the orthogonal eigenvectors of the covariance matrix Q, Λ corresponds to the diagonal eigenvalues of the covariance matrix Q; and the probability distribution P(Z|Ω) is calculated according to the following formula:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; [</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="2"\; label="y\; i"\; filenames="C0112263100121.png,C0112263100122.png"\; state="\{\{state\}\}">y</figure-callout></span><figure-callout\; id="2"\; label="y\; i"\; filenames="C0112263100121.png,C0112263100122.png"\; state="\{\{state\}\}">\; </figure-callout>}}_{\mathrm{<span\; class="notranslate">\; </span>}}^{\mathrm{<span\; class="notranslate">i</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}{<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="2"\; label="y\; i"\; filenames="C0112263100121.png,C0112263100122.png"\; state="\{\{state\}\}">y</figure-callout></span><figure-callout\; id="2"\; label="y\; i"\; filenames="C0112263100121.png,C0112263100122.png"\; state="\{\{state\}\}">\; </figure-callout>}}_{\mathrm{<span\; class="notranslate">\; </span>}}^{\mathrm{<span\; class="notranslate">i</span>}}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; )</span>}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;\&rho;</span>}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ]</span>$

Where M is the number of principal elements, y _i is the i-th element of Y, α _i is the i-th eigenvalue of Q, and p is passed $\mathrm{\&rho;}=\frac{1}{N-m}\underset{m+1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&alpha;}}_i$ The optimal value is obtained and assumes that the feature vector Z is provided from the observation class Ω.

Preferably in step (c), the behavior of objects in the video sequence is expressed using a Hidden Markov Model (HMM) based on state transitions.

It is best to express the Hidden Markov Model (HMM) as λ={Ξ, A, B, ∏}, where N is the number of possible states, and Ξ satisfies Ξ={q ₁ , q ₂ ,...,q _N }, A is the transformation {a _ij } between hidden states i and j, B is the observed symbol probability {b _j (.)} corresponding to state j, ∏ is the initial state distribution, and state Ξ={q ₁ , q ₂ , ..., q _N } and the initial state distribution Π are determined in advance based on the video data.

In order to achieve another object of the present invention, a method for object behavior recognition is provided, comprising the steps of: (a) obtaining a feature vector of a video frame by motion estimation, wherein the feature vector is a d×L dimension vector, and d is a dimension and L is the number of pixels in a video frame or region of interest; (b) using the acquired feature vectors to determine the state to which each frame belongs; and (c) using the transformation matrix for the determined states, determine the behavior model and the behavior model with the highest probability between the provided video frames from the given behavior model dictionary, as the identified behavior.

Description of drawings

The above objects and advantages of the present invention will become clearer through the following detailed description of preferred embodiments with the aid of the accompanying drawings, wherein:

Fig. 1 is a flow chart representing the main steps of the object behavior modeling method according to an embodiment of the present invention;

Figure 2A is an example diagram representing a hidden Markov model (HMM) of a behavior before training, wherein a person starts to stand up, but returns to a sitting position;

Figure 2B is an example graph representing a Hidden Markov Model (HMM) of a behavior after training, wherein a person starts to stand up, but returns to a sitting position; and

FIG. 3 is a flow chart showing the main steps of the object behavior recognition method according to the embodiment of the present invention.

Detailed ways

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The present invention is not limited to the following examples, and various changes can be made within the concept and scope of the present invention. The embodiments of the present invention are provided only to more fully explain the present invention to those skilled in the art.

FIG. 1 is a flowchart representing main steps of a method for modeling object behavior according to an embodiment of the present invention. Since all types of object behavior can be interpreted as varying objects undergoing different types of motion, it is best to think of behavior as a distribution of motion about the object. Therefore, in the present invention, human behavior is modeled based on the motion distribution. In the present invention, model-based motion estimation is used instead of precise motion estimation.

Referring to Fig. 1, in the object behavior modeling method according to the embodiment of the present invention, first, in step 102, the video frame that selects input by the state model of manual manipulation is grouped into a plurality of video frame groups, and each group Groups of video frames are divided into independent states.

In step 104, affine motion parameters are obtained for each video in each video frame group in each independent state by affine motion estimation. Here, when I denotes intensity, t denotes time, x denotes pixel position (x, y), and V denotes motion vector, the motion estimation is based on the pixel intensity on the object, which is expressed as Equation 1 below:

I _t (x)=I _t-1 (xV(x)) ......(1)

That is, in a given area, parameters that minimize the sum of squares of differences Σ(I _t (x)-I _t-1 (xV(x))) ² are estimated as motion parameters.

In model-based motion estimation, if the size of the object is much smaller than the distance between the camera and the object, an affine model can be used to approximate the motion of the object. If a local window is used to represent each point in the video, say, with a 5×5 pixel-sized window, motion can be approximated using affine model parameters. The affine model is expressed by Equation 2:

V (x, y) = ψ (x, y) K ... (2)

同时，应注意k₁和k₄相应于正交移动，而k₂、k₃、k₅和k₆相应于表面的变形。此时，如果忽略k₂、k₃、k₅和k₆，则运动向量V可以表达为V＝(k₁，k₄)^T。该运动向量V＝(k₁，k₄)^T是典型的光流向量。从而，在步骤106中，可以从仿射运动参数中获得光流向量。Here, (x, y) represents the coordinates of any point on the object, w(x, y)=(u(x, y), w(x, y)) ^T is the motion vector, K=(k ₁ , k ₂ , k ₃ , k ₄ , k ₅ , k ₆ ) are affine model parameters,

At the same time, it should be noted that k ₁ and k ₄ correspond to orthogonal movement, while k ₂ , k ₃ , k ₅ and k ₆ correspond to deformation of the surface. At this time, if k ₂ , k ₃ , k ₅ , and k ₆ are ignored, the motion vector V can be expressed as V=(k ₁ , k ₄ ) ^T . This motion vector V=(k ₁ , k ₄ ) ^T is a typical optical flow vector. Therefore, in step 106, the optical flow vector can be obtained from the affine motion parameters.

Next consider the motion vector P = (p ₁ , p ₂ , . . . p _d ) calculated at each pixel position (x, y). For example, P can be a 6-dimensional (6-D) affine motion parameter or a 2-D optical flow vector. When L represents the number of pixels in a video frame or region of interest, and d represents the dimensionality, the affine motion parameters can be expressed by Equation 3 below:

Z=(p ¹ ₁ , ^{p 2} ₁ , ..., p ^L ₁ ^, p 1 2 , p ² ₂ , ..., p ^L ₂ , p _{1 d} , p ² _d , ..., p ^L _d ) ^T .....(3)

That is, the feature vector Z composed of an affine motion vector or an optical flow vector can be represented as a d×L-dimensional vector. Using this method, at step 108, a feature vector Z is obtained from the optical flow vector.

At the same time, the eigenvector Z can be simulated using a Gaussian function. In this case, the mean value of the Gaussian function is referred to as m, and the covariance matrix of the eigenvector Z expressed in a matrix is referred to as Q. If the feature vector Z is provided from the observation class Ω, the probability distribution P(Z|Ω) is calculated according to Equation 4 below:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Here, Z represents the eigenvector, m represents the mean vector of the eigenvector Z, and Q is the covariance matrix of the eigenvector Z.

等于Z-m。接下来，如果φ的列为Q的正交特征向量，Λ相应于对角特征值，则协方差矩阵可以按照下式5分解：However, if the probability for classification of observations is calculated according to Equation 4, and the number and dimensionality of video pixels are taken into consideration, the computation required to obtain the probability is very large. Therefore, in this embodiment, the calculation of this formula is simplified by using the Karhunen-Loeve transform (KLT). First, define

equal to Zm. Next, if the columns of φ are the orthogonal eigenvectors of Q, and Λ corresponds to the diagonal eigenvalues, the covariance matrix can be decomposed according to Equation 5 below:

Q＝ ^φΛφT ......(5)

Based on this, if M is the number of principal elements, y _i is the ith element of Y, α _i is the first eigenvalue of Q, and ρ is passed $\mathrm{\&rho;}=\frac{1}{N-m}\underset{m+1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&alpha;}}_i$ The optimal value obtained, then Equation 4 can be approximated as Equation 6 below:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; [</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}{<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; )</span>}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;\&rho;</span>}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

Thus, in this embodiment, if the feature vector Z is provided from the observation class Ω, then in step 110 formula 6 is used to calculate the probability distribution P(Z|Ω). Next, in step 112, each state is modeled using the probability distribution calculated as described above.

Then, in step 114, the behavior of the objects in the video sequence is expressed based on the state transitions. In this embodiment, a Hidden Markov Model (HMM) is used to express object behavior in a video sequence. HMM is a very well-known stochastic model in training and recognizing data that changes over time. In particular, HMMs are widely used in speech recognition for online features or continuous input. In speech recognition using an HMM, a reference Markov model is generated by acquiring probability parameters of the Markov model at the time of training processing on the assumption that speech can be modeled as a Markov model. Likewise, in speech recognition processing, speech is recognized by estimating a benchmark Markov model that is most similar to the input expression. Generally, a Hidden Markov Model is used as a model for recognizing speech, and the reason is to adapt to different variations of speech patterns. Here, the word "hidden" means that the state is hidden in the model regardless of the mode of speech. When N is the number of possible states, Ξ satisfies Ξ={q ₁ , q ₂ ,...,q _N }, A is the transformation {a _ij } between hidden states i and j, and B is the transformation corresponding to state j When the observed symbol probability {b _j (.)} of , and ∏ is the initial state distribution, the general HMM can be expressed as the following formula 7:

λ＝{Ξ, A, B, ∏}                                                                                                                                                                                                                                                                          ... (7)

State Ξ={q ₁ , q ₂ , . . . , q _N } and initial state distribution Π are determined in advance based on video data. Hidden Markov model parameters A and B can be repeatedly trained using the well-known Baum-Welsh reestimation formula.

The state model or the number of states can be determined empirically, and in this embodiment, an example of selecting 4 states will be explained. Also, in this embodiment, behavior is modeled using 4 states, and an example of setting the same value for the transition probability of each state will be explained.

Figures 2A and 2B show an example of a Hidden Markov Model of behavior in which a person starts to stand up, but then returns to a sitting position (hereinafter referred to as "bd"). FIG. 2A shows an example of a hidden Markov model of bd before training; FIG. 2B shows an example of a hidden Markov model of bd after training. Referring to FIG. 2A , the probability that transition from one state to another state will occur and the probability that transition from one state to the previous state will occur are uniformly set to 0.333. For the convenience of model development, it is assumed that the return from a state called 4 to 4 occurs with probability 1. However, referring to FIG. 2B , the probability of occurrence of transition from one state to another state and the probability of occurrence of transition from one state to the previous state are set to different values. Then, using this to set different transformation probabilities, a transformation matrix is obtained. Next, a plurality of different states defined by respective different probability distributions and the obtained transformation matrix are determined as a behavior model. In this way, behavioral modeling is completed.

According to the object behavior modeling method described above, complex behaviors such as human activities can be effectively modeled in the field of video indexing and recognition. In particular, according to the present object behavior modeling method, object behavior required for behavior recognition can be modeled without segmenting the object.

The object behavior modeling method according to the above-mentioned embodiments can be applied to systems such as still camera. However, if the system to which it is desired to apply the method is a motion camera, the person's motion must first be recovered. Subsequent steps are the same as those in the above-mentioned embodiment.

The processing of the recognition action will now be described. FIG. 3 is a flow chart showing the main steps of the object behavior recognition method according to the embodiment of the present invention. Referring to FIG. 3 , first in step 302 , a video frame containing a desired behavior to be recognized is input. Next, in step 304, feature vectors are obtained through motion estimation of input video frames. It can be considered that step 304 is substantially the same as step 106 described with reference to FIG. 1 .

Next, in step 306, the state to which each video frame belongs is determined using the obtained feature vector. If T is a positive integer representing the number of frames forming a video sequence, Z ₁ , Z ₂ , ..., Z _T are respectively the feature vectors of the first frame, the second frame, ..., the T frame, and if Given video frame O={Z ₁ , Z ₂ ,..., Z _T }, E is the number of state models, then in step 308, the provided video frame and the given behavior model dictionary {λ ₁ , λ ₂ ,..., λ _E } among the behavior models, the behavior model with the largest probability P(O|λ) is determined as the recognized behavior. The transformation matrix is obtained in the training process by using an expectation-maximization (EM) algorithm based on the observed symbol probabilities {b _j (.)} corresponding to scene j. In order to improve the search speed, it is best to track the motion trajectory in a window with the same size as that used in training, based on a prediction algorithm using Kalman filtering consisting of three steps: initialization, state prediction, estimation, and update.

In this way, complex object behaviors such as human activities in video sequences can be efficiently recognized. According to this object behavior recognition method, complex behavior such as human behavior can be efficiently recognized. In particular, according to the object behavior recognition method, behavior can be recognized without segmenting the object.

In addition, the object behavior modeling method and the object behavior recognition method according to the present invention can be written as programs executed on a personal computer or a server computer. Computer programmers skilled in the art can very easily derive the program codes and code segments for constructing the program. Also, the program can be stored in a computer-readable recording medium. The recording media may include magnetic recording media, optical recording media, and radio media.

As described above, according to the present invention, in the field of video indexing and recognition, it is possible to effectively model and recognize complex behaviors such as human behaviors without segmenting objects.

Claims (11)

1. A method for object behavior modeling, comprising the following steps: (a) Obtain the optical flow vector from the video sequence; (b) Using the optical flow vector, obtain the probability distribution of a feature vector for multiple video frames, where the feature vector is a d×L dimensional vector, where d is the dimension and L is the the number of pixels in the region; (c) performing state modeling using the probability distribution of the feature vector; and (d) Express the behavior of objects in video sequences based on state transitions. 2. The object behavior modeling method of claim 1, wherein step (a) is based on affine motion estimation. 3. The object behavior modeling method as claimed in claim 2, wherein step (a) also comprises sub-steps: (a-1) grouping the input video frame into a plurality of video frame groups, and dividing each video frame group into independent states; (a-2) obtaining affine motion parameters for each video in the group of video frames in each independent state; and (a-3) Obtain the optical flow vector from the affine motion parameters. 4. The object behavior modeling method as claimed in claim 3, wherein step (a-2) comprises the step: when I represents intensity, t represents time, x represents pixel position (x, y), V represents motion vector, Let Σ(I _t ( _x )−I _{t−1 (} xV(x))) ² The smallest parameter is determined as the motion parameter. 5. The object behavior modeling method as claimed in claim 1, wherein step (b) comprises the step: calculate the probability distribution P (Z|Ω) according to the following formula: $\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span>}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}$ where P=(p ₁ , p ₂ ,...p _d ) represents the motion vector calculated at each pixel position (x, y), L represents the number of pixels in a video frame or region of interest, and d represents dimension, the eigenvector Z of the d×L-dimensional vector is Z=(p ¹ ₁ , p ² ₁ ,..., p ^L ₁ , p ¹ ₂ , p ² ₂ ,..., p ^L ₂ , p ¹ _d , p ² _d ,..., p ^L _d ) ^T , m is the mean vector of eigenvector Z, Q is the covariance matrix of eigenvector Z, and it is assumed that eigenvector Z is provided from the observation class Ω. 6. The object behavior modeling method as claimed in claim 1, wherein step (b) also comprises the step: Decompose the covariance matrix Q according to the following formula: Q＝ ^φΛφT where the columns of φ are the orthogonal eigenvectors of the covariance matrix Q, and Λ corresponds to the diagonal eigenvalues of the covariance matrix Q; and Calculate the probability distribution P(Z|Ω) according to the following formula: $\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; [</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}{<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; )</span>}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;\&rho;</span>}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ]</span>$ Where M is the number of principal elements, y _i is the i-th element of Y, α _i is the i-th eigenvalue of Q, and ρ is passed $\mathrm{\&rho;}=\frac{1}{N-m}\underset{m+1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&alpha;}}_i$ Obtain the optimal value and assume that the feature vector Z is provided from the observation class Ω. 7. The object behavior modeling method as claimed in claim 1, wherein in step (c), the object behavior in the video sequence is expressed using a Hidden Markov Model (HMM) based on state transitions. 8. The object behavior modeling method as claimed in claim 7, wherein the Hidden Markov Model (HMM) is expressed as λ={Ξ, A, B, Π}, wherein N is the number of possible states, and Ξ satisfies Ξ ={q ₁ ,q ₂ ,...,q _N ), A is the transformation {a _ij ) between hidden states i and j, B is the observed symbol probability corresponding to state j {b _j (.)} , Π is the initial state distribution, and the state Ξ={q ₁ , q ₂ , . . . , q _N } and the initial state distribution Π are determined in advance based on video data. 9. A method for object behavior recognition, comprising the steps of: (a) obtaining the feature vector of the video frame by motion estimation, wherein said feature vector is a d×L dimensional vector, d is the dimension and L is the number of pixels in a video frame or region of interest; (b) using the acquired feature vectors to determine the state to which each frame belongs; and (c) Using the transformation matrix for the determined state, determine a behavior model that maximizes the probability between the behavior model and the video frame provided from the given behavior model dictionary, as the recognized behavior. 10. The object behavior recognition method as claimed in claim 9, wherein step (c) comprises the step: when T is a positive integer representing the number of frames forming a video sequence, Z ₁ , Z ₂ , ..., Z _T respectively When is the feature vector of the first frame, the second frame, ..., the T frame, and a given video frame O={Z ₁ , Z ₂ , ..., Z _T }, E is the number of state models, Find the behavior model that maximizes the probability P(O|λ) from the given behavior model dictionary {λ ₁ ,λ ₂ ,...,λ _E ). 11. The object behavior recognition method according to claim 10, wherein the transformation matrix is obtained in the training process by using an Expectation Maximum (EM) algorithm based on the observed symbol probability {b _j (.)} corresponding to the scene j.

Images