CN100486336C - Real time method for segmenting motion object based on H.264 compression domain - Google Patents

Abstract

The invention proposes a method for real-time segmentation of moving objects in H.264 compressed domain based on matching matrix. The only information the segmentation depends on is the motion vector field based on uniform sampling of 4*4 blocks extracted from H.264 video. Firstly, the motion vector field of consecutive multiple frames is normalized and iteratively back-projected to obtain the accumulated motion vector field to enhance salient motion information. Then the global motion compensation is performed on the accumulated motion vector field, and a fast statistical region growing algorithm is used to divide it into multiple regions according to the motion similarity. Using the results of the above two aspects, a moving object segmentation method based on matching matrix is proposed, so that it can effectively perform object tracking and updating, object merging and splitting, new object appearance and object disappearance in video sequences. The experimental results of the MPEG-4 test sequence show that on a computer with a CPU of 3.0GHz and a memory of 512M to process a video sequence in CIF format, the average processing time of each frame is 38ms, which can meet the requirements of most real-time applications at 25fps. requirements, and has good segmentation quality. Since the method proposed by the present invention only uses motion vector field information, it is also applicable to moving object segmentation in MPEG compression domain.

Description

Real-time Segmentation Method of Moving Objects Based on H.264 Compressed Domain

technical field

The present invention relates to a real-time segmentation method of moving objects based on H.264 compressed domain, especially different from the existing method, it avoids the complete decoding of compressed video, and only the motion vector extracted by entropy decoding is used for segmentation required motion features, so the amount of computation is greatly reduced. Moreover, it is not limited to video sequences with static backgrounds. For video sequences with moving backgrounds or static backgrounds, it can quickly and reliably segment moving objects. Since this method only uses the information of the motion vector field, it is also applicable to the segmentation of moving objects in the MPEG compression domain.

Background technique

Moving object segmentation is a prerequisite for many content-based multimedia applications such as video indexing and retrieval, intelligent video surveillance, video editing, and face recognition. Since content-based video coding was proposed by MPEG-4, most researches on moving object segmentation have focused on the pixel domain, while the moving object segmentation based on compressed domain has only attracted attention in recent years. Carrying out moving object segmentation in the compressed domain is more suitable for the needs of practical applications than the segmentation method in the pixel domain. In particular, most of the video sequences in practical applications have been compressed into a certain format, and the moving object segmentation is directly performed in this compressed domain, which can avoid completely decoding the compressed video; moreover, the amount of data to be processed in the compressed domain is larger than that of pixels. The domain is much less, so the amount of calculation is greatly reduced; in addition, the motion vector and DCT coefficient extracted from the compressed video only through entropy decoding can be directly used as the motion feature and texture feature required for segmentation. Therefore, the segmentation of moving objects from the compressed domain has the characteristics of rapidity, which can solve the problem that the traditional pixel domain segmentation method is difficult to meet the requirements of real-time segmentation, and is suitable for many applications with real-time requirements.

At present, although the moving object segmentation method in the compressed domain has been proposed, it is basically aimed at the MPEG-2 compressed domain. H.264 is the latest video coding standard, which doubles the coding efficiency compared to MPEG-2. More and more applications are turning to H.264 to replace MPEG-2, but so far there is little research on moving object segmentation in H.264 compression domain. In contrast to the MPEG compressed domain, the DCT coefficients of I-frames in the H.264 compressed domain cannot be directly used as texture features for segmentation, since they are transformed on the spatial prediction residual of the block instead of the original block . Therefore, the only feature that can be directly used for moving object segmentation in the H.264 domain is the motion vector information. Currently in the H.264 domain, only Zeng et al. proposed a block-based MRF model to segment moving objects from the sparse motion vector field, assigning different types of marks to each block according to the magnitude of the motion vector of each block, and maximizing The posterior probabilities of the MRF flag blocks that belong to moving objects. However, this method is only suitable for video sequences with static backgrounds, and the accuracy of segmentation is not high, and the amount of calculation is relatively large.

Contents of the invention

The object of the present invention is to provide a real-time segmentation method for moving objects based on H.264 compressed domain. The only information used for segmentation is the motion vector field based on 4×4 block uniform sampling extracted from H.264 compressed video. This method can avoid the complete decoding of compressed video, and uses the motion vector extracted by entropy decoding as the motion feature required for segmentation, so the calculation amount is greatly reduced, so as to achieve the purpose of real-time moving object segmentation.

For achieving above-mentioned purpose, design of the present invention is:

As shown in Figure 1, the motion vectors are extracted and normalized from the input H.264 compressed video stream, and then iterative back projection is performed to obtain the accumulated motion vector field which can significantly enhance the motion information. Then perform global motion compensation and divide the accumulated motion vector field into multiple areas according to the motion similarity, and then use a matching matrix to represent the correlation between the segmented area of the current frame and the projection of the moving object in the previous frame in the current frame, based on this matching matrix Segment moving objects.

According to above-mentioned design, technical scheme of the present invention is:

A method for real-time segmentation of moving objects based on H.264 compressed domain, characterized in that the motion vector field of continuous multi-frames is normalized and iteratively back-projected to obtain the cumulative motion vector field; then the cumulative motion vector field is globally motion compensation, adopt fast statistical region growing algorithm to divide the accumulative motion vector field into a plurality of regions according to motion similarity; utilize above-mentioned two aspects result, adopt the moving object segmentation method based on matching matrix proposed by the present invention to segment out moving object, Among them, various situations such as tracking and updating of objects, merging and splitting of objects, appearance and disappearance of objects, etc. can be effectively carried out in the video sequence; the steps are:

a. Normalization of the motion vector field: extract the motion vector field from the H.264 video and perform normalization in the time and space domains;

b. Cumulative motion vector field: use the motion vector field of multiple consecutive frames to perform iterative back projection to obtain a more reliable cumulative motion vector field;

c. Global motion compensation: Compensate after performing global motion estimation on the accumulated motion vector field to obtain the residual of each 4×4 block;

d. Region Segmentation: The cumulative motion vector field is segmented into multiple regions with similar motion using the statistical region growing method;

e. Object Segmentation: Segment moving objects using a matching matrix-based segmentation method.

The above-mentioned steps of normalizing the motion vector field are:

(1) Temporal normalization: divide the motion vector of the current frame by the number of frames between the current frame and the reference frame, which is the distance in the instant domain;

(2) Spatial normalization: directly assign the motion vector of each macroblock whose size is larger than 4*4 to all 4*4 blocks covered by the macroblock.

The above-mentioned steps of accumulating the motion vector field are:

(1) Use the motion vector field of several frames after the current frame to back-project the motion vector field of the adjacent frame, that is, the projection of the current block is obtained by multiplying the motion vector of each projected block by different scale factors and adding them together The selection method of the motion vector and the scale factor is: if the total area of the overlapping area is greater than half of the current block area, the scale factor of each projected block is taken as the overlapping area of the projected block and the current block divided by all projected blocks and The total area of the overlapping area of the current block; otherwise, the scale factor of each projected block is taken as the ratio of its overlapping area to the area of the current block;

(2) Start iterative accumulation from the subsequent frame to obtain the accumulated motion vector field of the current frame;

The above steps of global motion compensation are:

(1) Estimate the global motion vector field using an affine motion model with 6 parameters;

①Model parameter initialization: let m=(m ₁ , m ₂ , m ₃ , m ₄ , m ₅ , m ₆ ) be the parameter vector of the global motion model, and the model parameter m ⁽⁰⁾ is initialized as: $m^{\left(0\right)}={\left[\begin{array}{cccccc}1& 0& \frac{1}{N}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}{\mathrm{mvx}}_i& 0& 1& \frac{1}{N}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}{\mathrm{mvy}}_i\end{array}\right]}^T,$

的直方图，然后剔除直方图中那些偏差幅度平方和大于25％的运动矢量；② Eliminate outliers: first calculate the estimated center coordinates of the ith block in the previous frame with the center coordinates of the current frame (xi _, y _i ) $\left[\begin{array}{c}{x}_i^{\′}\\ {\mathrm{the\; y}}_i^{\′}\end{array}\right]=\left[\begin{array}{cc}{m}_1& m_2\\ m_4& m_5\end{array}\right]\left[\begin{array}{c}{x}_i\\ {\mathrm{the\; y}}_i\end{array}\right]+\left[\begin{array}{c}{m}_3\\ m_6\end{array}\right],$ Then predict the motion vector The deviation (ex _i , ey _i ) from the original cumulative motion vector (mvx _i , mvy _i ) is calculated as: $\begin{array}{c}{\mathrm{ex}}_i=x_i^{\′}-x_i-{\mathrm{mvx}}_i\\ e{\mathrm{the\; y}}_i={\mathrm{the\; y}}_i^{\′}-{\mathrm{the\; y}}_i-{\mathrm{mvy}}_i\end{array},$ Use this formula to calculate the prediction deviation (ex _i , ey _i ) of each 4×4 block, and finally calculate the sum of the squares of the deviation magnitude

The histogram of the histogram, and then remove those motion vectors in the histogram whose sum of the squares of the deviation magnitude is greater than 25%;

③Model parameter update: Use the motion vector and Newton-Raphson method retained in the previous steps to update the model parameters. The new model parameter vector m ^(l) in the first iteration is defined as follows: m ^(l) = m ^{(l- 1)} -H ^-1 b, where the Hessian matrix H and the gradient vector b are calculated as follows:

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccccc}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]$

$\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; =</span>{\left[\begin{array}{cccccc}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ex</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ex</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; ex</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ey</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ey</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; ey</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}\right]}^{\mathrm{<span\; class="notranslate">\; T</span>}}$

Here R represents the collection of retained blocks;

④End condition: Repeat steps ② and ③ up to 5 times, and if one of the following two conditions is met, the iteration will also end early: (i) calculate m ^(l) -m _static , get a difference vector, if the difference Each parameter component in the value vector is all less than 0.01, just judges to belong to the situation of still camera, finishes iteration; Wherein m _static =[100010] ^T is the global motion vector under the still situation of camera; (ii) calculate m ^{(l )} and m ^(l-1) , if the parameter components m ₃ and m ₆ of this difference are less than 0.01, and other parameter components are less than 0.0001, then the iteration ends;

最后得到全局运动矢量场

⑤ Substitute the obtained global motion model parameter vector m into $\left[\begin{array}{c}{x}_i^{\&prime;}\\ {\mathrm{the\; y}}_i^{\&prime;}\end{array}\right]=\left[\begin{array}{cc}{m}_1& m_2\\ m_4& m_5\end{array}\right]\left[\begin{array}{c}{x}_i\\ {\mathrm{the\; y}}_i\end{array}\right]+\left[\begin{array}{c}{m}_3\\ m_6\end{array}\right],$ Find the estimated coordinates of the previous frame

Finally, the global motion vector field is obtained

(2) Calculate the residuals of each 4×4 block in the global motion vector field and the cumulative motion vector field.

The above region segmentation is to use the statistical region growing algorithm to divide the accumulated motion vector field into multiple regions with similar motion, the steps are as follows:

(1) Calculate the motion difference measure of any adjacent block group in the four neighborhoods;

(2) All adjacent block groups are sorted according to the order of motion difference measure from small to large;

(3) Merge the adjacent blocks with the smallest motion difference measure, and start the region growing process here. When each region grows, the current two block groups belong to two adjacent regions respectively, then judge the two The condition for whether regions are merged is whether the difference between the average motion vectors of these two regions is less than a threshold condition:

$\mathrm{\&Delta;}\left(R\right)=\frac{{\mathrm{SR}}^2}{2Q\left|R\right|}(\min (\mathrm{SR},\left|R\right|)\log (1+\left|R\right|)+2\log 6\mathrm{wh}),$ Among them, SR represents the dynamic range of the motion vector, |R| represents the number of motion vectors contained in the region, wh represents the size of the motion vector field, and the parameter Q is used to control the division degree of the motion vector field, so that the motion vector field can be moderately Segmentation into several regions with similar motion;

(4) Calculate the average residual error of each segmented region after global motion compensation;

剩下的区域作为运动对象可能存在的区域

最后对当前帧所分割的M个对象区域和1个背景区域分别标记，分割结果记为 (5) Distinguish the most reliable background area and the area where other objects are located, and select the area with the smallest average residual error among several segmented areas whose area is larger than 10% of the entire motion vector field as the reliable background area, marked as

The remaining area is the area where the moving object may exist

Finally, the M object regions and 1 background region segmented by the current frame are respectively marked, and the segmentation result is denoted as

The above object segmentation is to use the segmentation result of the moving object obtained in the previous frame at time t-1 to determine whether each segmented area matches an object in the previous frame at time t in the current frame, so as to construct a matching matrix ;Based on the matching matrix, judge the tracking and updating of objects, the merging of objects, the splitting of objects, the appearance of new objects, the disappearance of old objects, etc., and finally obtain some moving objects in the current frame; the steps are:

和1个背景对象

标记出来，然后采用后向投影的方法获得前一帧各个对象在当前帧的投影区域。就是利用当前帧累积运动矢量场中任意块的坐标和其对应的累积运动矢量的差求出这个块在前一帧中的匹配位置，然后将前一帧匹配位置上的块对象投影到当前帧并逐个标记出来，记为

(1) Use the backward projection method to obtain the projection area of each object at the time t of the current frame at the time t-1 of the previous frame, and first move the N moving objects of the previous frame

and 1 background object

Mark it out, and then use the method of back projection to obtain the projection area of each object in the previous frame in the current frame. It is to use the difference between the coordinates of any block in the cumulative motion vector field of the current frame and its corresponding cumulative motion vector to find the matching position of this block in the previous frame, and then project the block object at the matching position of the previous frame to the current frame And marked out one by one, denoted as

和

构造3个M+1行N+1列的矩阵CM^t，CMR^t，CMC^t。其中矩阵CM^t中的任意元素CM^t(i，j)取值为在中标记为i且在标记为j的象素数目，即分割区域

与对象投影

相互重叠的面积。而矩阵CMR^t(i，j)定义为CMR^t第i行的各个元素是分割区域

落在各个对象投影内的比例；矩阵CMC^t(i，j)定义为CMC^t第j列的各个元素是对象的投影落在各个分割区域内的比例；(2) Construct a matrix CM ^t , which represents the overlapping area of the segmentation region and the object projection; construct a matrix CMR ^t , which represents the proportion of each segmentation region within each object projection; construct a matrix CMC ^t , which represents each object The proportion of the projection falling within each segmented area; according to the marker image

and

Construct three matrices CM ^t , CMR ^t , and CMC ^t with M+1 rows and N+1 columns. Among them, any element CM ^t (i, j) in the matrix CM ^t takes a value in marked i in and in The number of pixels marked as j, that is, the segmented area

with object projection

overlapping area. The matrix CMR ^t (i, j) is defined as each element of the i-th row of CMR ^t is the segmentation region

The proportion that falls within the projection of each object; the matrix CMC ^t (i, j) is defined as each element of the jth column of CMC ^t is the object The proportion of the projection of the falls in each segmented area;

和

之间的相关信息；CMM^t首先置为M+1行、N+1列的零矩阵；接着对CMR^t进行行扫描找到每一行最大值所在的位置，对CMM^t中相应位置处的元素值加1；然后对CMC^t进行列扫描找到每一列最大值所在的位置，对CMM^t中相应位置处的元素值加2；生成的矩阵CMM^t的纵坐标依次表示为当前帧背景区域

和运动区域

横坐标依次表示为前一帧背景对象

和运动对象

矩阵中各元素的可能取值为0，1，2，3；CMM^t中任意不为0的元素CMM^t(i，j)表明了分割区域

与对象存在一定的相关性，具体而言：(3) Construct a matrix CMM ^t , which represents the degree of correlation between the current frame segmentation area and the object projection, and the matrix CMM ^t records the values reflected by CMR ^t and CMC ^t

and

The relevant information between; CMM ^t is first set as a zero matrix with M+1 rows and N+1 columns; then row-scan CMR ^t to find the position of the maximum value of each row, and the element value at the corresponding position in CMM ^t Add 1; then scan the columns of CMC ^t to find the position of the maximum value of each column, and add 2 to the element value at the corresponding position in CMM ^t ; the ordinate of the generated matrix CMM ^t is represented as the background area of the current frame in turn

and sports area

The abscissa represents the background object of the previous frame in sequence

and moving objects

The possible values of each element in the matrix are 0, 1, 2, 3; any element CMM ^t (i, j) that is not 0 in CMM ^t indicates the segmentation area

with the object There are certain correlations, specifically:

在很大程度上属于前一帧对象

①CMM ^t (i, j)=1, indicating the segmented area

Objects that belong largely to the previous frame

中；②CMM ^t (i, j)=2, indicating that the object in the previous frame largely contained in the segmented area

middle;

和

具有极强的相关性；需要进一步比较，如果CMR^t(i，j)>CMC^t(i，j)，则CMM^t(i，j)＝1；否则，CMM^t(i，j)＝2；最后生成的CMM^t取值范围为0，1，2；③ CMM ^t (i, j) = 3, which includes the above two situations at the same time, indicating that

and

There is a strong correlation; further comparison is needed, if CMR ^t (i, j) > CMC ^t (i, j), then CMM ^t (i, j) = 1; otherwise, CMM ^t (i, j) = 2 ;The last generated CMM ^t ranges from 0, 1, 2;

(4) Carry out object segmentation based on the matching matrix CMM ^t for tracking and updating of a single object, new object appearance, object merging, object splitting and object disappearance; through the matrix CMM ^t , the segmentation area and The association relationship of moving objects, which can effectively handle the following five situations in a unified manner:

①Single object tracking and updating (1→1);

②A new object appears (0→1);

③ Merge of objects (m→1);

④ Splitting of objects (1→m);

⑤ Disappearance of the object (1→0).

Compared with the prior art, the present invention has the following outstanding features and advantages: The real-time moving object segmentation method based on the compressed domain provided by the present invention is based on the H.264 video stream, which is completely different from the existing method. The compressed domain video object segmentation method is mainly applicable to the MPEG domain, while the present invention is not only applicable to the H.264 compressed domain, but also applicable to the MPEG compressed domain. Moreover, the present invention is not limited to video sequences with static backgrounds, and can quickly and reliably segment moving objects no matter for video sequences with moving backgrounds or static backgrounds. In addition, the method for segmenting moving objects by matching matrix proposed by the present invention can almost perform real-time segmentation for various situations of video object motion, so the effect of segmenting objects is very good and has strong applicability.

Description of drawings

Fig. 1 is a program block diagram of the H.264 compressed domain moving object real-time segmentation method based on the matching matrix of the present invention.

FIG. 2 is a block diagram of the normalized motion vector field and the accumulated motion vector field in FIG. 1 .

Fig. 3 is a structural block diagram of global motion compensation and region segmentation in Fig. 1 .

Fig. 4 is a structural block diagram of object segmentation in Fig. 1 .

Fig. 5 is an illustration of the moving object segmentation results for each typical frame (frame 4, 37, 61, 208) in the sequence Coastguard.

Fig. 6 is an illustration of the moving object segmentation results of each typical frame (4th, 43rd, 109th, 160th frame) in the sequence Mobile.

Detailed ways

An implementation example of the present invention is described in detail as follows in conjunction with accompanying drawing:

The H.264 compressed domain moving object real-time segmentation method based on the matching matrix of the present invention is according to the program block diagram shown in Figure 1, and is programmed on a PC test platform with a CPU of 3.0GHz and a memory of 512M. Figure 5 and Figure 6 show the simulation test result.

Referring to Fig. 1, the H.264 compressed domain moving object real-time segmentation method based on the matching matrix of the present invention enhances significant motion information through the normalization and accumulation of the motion vector field, and then performs global motion compensation on the accumulated motion vector field, using Statistical region growing algorithm and moving object segmentation method based on matching matrix are used to segment regions and moving objects, which have the characteristics of simple algorithm, fast object segmentation speed and good segmentation effect.

The steps are:

(1) Normalization of the motion vector field: extract the motion vector field from the H.264 video and perform normalization in the time and space domains;

(2) Cumulative motion vector field: use the motion vector field of multiple consecutive frames to perform iterative back projection to obtain a more reliable cumulative motion vector field;

(3) Global motion compensation: Compensate after performing global motion estimation on the accumulated motion vector field to obtain the residual error of each 4×4 block;

(4) Region segmentation: The accumulated motion vector field is segmented into multiple regions with similar motions using the statistical region growing method;

(5) Object Segmentation: The moving object is segmented by a segmentation algorithm based on matching matrix.

The process of the motion vector field normalization of above-mentioned step (1) is as follows:

①Temporal normalization: Divide the motion vector of the current frame by the number of frames between the current frame and the reference frame, which is the distance in the instant domain;

② Spatial domain normalization: directly assign the motion vector of each macroblock whose size is larger than 4*4 to all 4*4 blocks covered by the macroblock.

The process of accumulating motion vector field of above-mentioned steps (2) is as follows:

① Use the motion vector field of several frames after the current frame to back-project the motion vector field of the adjacent frame;

② Iteratively accumulate from the next frame to obtain the accumulated motion vector field of the current frame.

The process of the global motion compensation of above-mentioned steps (3) is as follows:

① Estimate the global motion vector field using a 6-parameter affine motion model;

② Calculate the residual error of each 4×4 block after global motion compensation.

The process of the region segmentation of the above-mentioned step (4) is as follows:

① Calculate the motion difference measure of any adjacent block group in the four neighborhoods;

② All adjacent block groups are sorted according to the order of motion difference measure from small to large;

③Merge the adjacent blocks with the smallest motion difference measure, and start the region growing process here;

④ Calculate the average residual error of each segmented area after global motion compensation;

⑤ Distinguish the most reliable background area from the area where other objects are located.

The process of object segmentation in the above step (5) is as follows:

① Use the back projection method to obtain the projection area of each object in the previous frame in the current frame;

② Construct a matrix CM ^t , which represents the overlapping area of the segmented area and the projected object; construct a matrix CMR ^t , which represents the proportion of each segmented area in the projection of each object; construct a matrix CMC ^t , which represents the area of each object projected. The proportion in each segmented area;

③ Construct matrix CMM ^t , which represents the degree of association between the current frame segmentation area and the object projection;

④ Based on the matching matrix CMM ^t, object segmentation is performed for five types of situations: single object tracking and updating, new object appearance, object merging, object splitting, and object disappearance.

Below the five steps of this implementation example in conjunction with the general block diagram (Fig. 1) are given in further detail:

a. Motion vector field normalization:

As shown in Figure 2, the motion vector of the current frame is divided by the number of frames between the current frame and the reference frame to obtain normalization in the time domain, and the motion vector of the block larger than 4×4 in the current frame is directly assigned to the All 4x4 blocks covered by the block get normalized over the spatial domain.

b. Cumulative motion vector field:

As shown in FIG. 2 , the motion vector fields of several frames after the current frame are used to back-project the motion vector fields of adjacent frames. The projected motion vector of the current block is obtained by multiplying the motion vectors of each projected block by different scale factors and adding them together. The selection method of the scale factor is: if the total area of the overlapping area is greater than half of the area of the current block, each projection The scale factor of a block is taken as the overlapping area of the projected block and the current block divided by the total area of the overlapping area of all projected blocks and the current block; otherwise, the scale factor of each projected block is taken as the ratio of its overlapping area to the area of the current block . Then iteratively accumulate from the next frame to obtain the accumulated motion vector field of the current frame.

c. Global motion compensation:

As shown in Figure 3, a 6-parameter affine motion model is used to estimate the global motion vector field, and the difference between it and the cumulative motion vector field can be used to obtain the residual error of any block in the cumulative motion vector field after global motion compensation. The steps are:

(1) Estimate the global motion vector field using an affine motion model with 6 parameters:

①Model parameter initialization:

Let m=(m ₁ , m ₂ , m ₃ , m ₄ , m ₅ , m ₆ ) be the parameter vector of the global motion model, and the model parameter m ⁽⁰⁾ is initialized as: $m^{\left(0\right)}={\left[\begin{array}{cccccc}1& 0& \frac{1}{N}\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{mvx}}_i& 0& 1& \frac{1}{N}\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{mvy}}_i\end{array}\right]}^T;$

② Eliminate outliers:

$\left[\begin{array}{c}{x}_i^{\&prime;}\\ y_i^{\&prime;}\end{array}\right]=\left[\begin{array}{cc}{m}_1& m_2\\ m_4& m_5\end{array}\right]\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]+\left[\begin{array}{c}{m}_3\\ m_6\end{array}\right],$ 则预测运动矢量

和原始累积运动矢量(mvx_i，mvy_i)的偏差(ex_i，ey_i)计算为： $\begin{array}{c}{\mathrm{ex}}_i=x_i^{\&prime;}-x_i-{\mathrm{mvx}}_i\\ e{y}_i=y_i^{\&prime;}-y_i-{\mathrm{mvy}}_i\end{array}$ 。使用这个式子计算出每个4×4块的预测偏差(ex_i，ey_i)，最后计算出偏差幅度的平方和

的直方图，然后剔除直方图中那些偏差幅度平方和大于25％的运动矢量。First calculate the estimated center coordinates of the ith block in the previous frame with the center coordinates of the current frame ( _xi , y _i )

$\left[\begin{array}{c}{x}_i^{\&prime;}\\ {\mathrm{the\; y}}_i^{\&prime;}\end{array}\right]=\left[\begin{array}{cc}{m}_1& m_2\\ m_4& m_5\end{array}\right]\left[\begin{array}{c}{x}_i\\ {\mathrm{the\; y}}_i\end{array}\right]+\left[\begin{array}{c}{m}_3\\ m_6\end{array}\right],$ Then predict the motion vector

The deviation (ex _i , ey _i ) from the original cumulative motion vector (mvx _i , mvy _i ) is calculated as: $\begin{array}{c}{\mathrm{ex}}_i=x_i^{\&prime;}-x_i-{\mathrm{mvx}}_i\\ e{\mathrm{the\; y}}_i={\mathrm{the\; y}}_i^{\&prime;}-{\mathrm{the\; y}}_i-{\mathrm{mvy}}_i\end{array}$ . Use this formula to calculate the prediction deviation (ex _i , ey _i ) of each 4×4 block, and finally calculate the sum of the squares of the deviation magnitude

The histogram of the histogram, and then remove those motion vectors whose sum of squared deviation magnitudes is greater than 25% in the histogram.

③Model parameter update:

Use the motion vectors preserved from the previous steps and the Newton-Raphson method to update the model parameters. The new model parameter vector m ^(l) in the first iteration is defined as follows: m ^(l) = m ^(l-1) -H ^-1 b, where the Hessian matrix H and the gradient vector b are calculated as follows:

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccccc}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]$

$\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; =</span>{\left[\begin{array}{cccccc}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ex</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ex</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; ex</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ey</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; ey</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; ey</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}\right]}^{\mathrm{<span\; class="notranslate">\; T</span>}}$

Here R represents the set of preserved blocks.

④ End condition: Repeat steps ② and ③ up to 5 times, and if one of the following two conditions is met, the iteration will also end early:

(i) Calculate m ^(l) -m _static to obtain a difference vector, if each parameter component in the difference vector is less than 0.01, it is judged to belong to the situation where the camera is still, and the iteration ends; where m _static =[ 100010] ^T is the global motion vector when the camera is still;

(ii) Calculate the difference between m ^(l) and m ^(l-1) , if the parameter components m ₃ and m ₆ of this difference are less than 0.01, and other parameter components are less than 0.0001, then the iteration ends.

最后得到全局运动矢量场

⑤ Substitute the obtained global motion model parameter vector m into $\left[\begin{array}{c}{x}_i^{\&prime;}\\ {\mathrm{the\; y}}_i^{\&prime;}\end{array}\right]=\left[\begin{array}{cc}{m}_1& m_2\\ m_4& m_5\end{array}\right]\left[\begin{array}{c}{x}_i\\ {\mathrm{the\; y}}_i\end{array}\right]+\left[\begin{array}{c}{m}_3\\ m_6\end{array}\right],$ Find the estimated coordinates of the previous frame

Finally, the global motion vector field is obtained

(2) Calculate the residuals of each 4×4 block in the global motion vector field and the cumulative motion vector field.

d. Region segmentation:

As shown in FIG. 3 , the present invention uses a statistical region growing algorithm to realize region segmentation of the accumulated motion vector field. The steps are detailed below:

(1) Calculate the motion difference measure of any adjacent block group in the four neighborhoods;

(2) All adjacent block groups are sorted according to the order of motion difference measure from small to large;

(3) Merge the adjacent blocks with the smallest motion difference measure, and start the region growing process here. When each area grows, the current two block groups belong to two adjacent areas respectively, and the condition for judging whether the two areas are merged is whether the difference between the average motion vectors of the two areas is less than the threshold condition: $\mathrm{\&Delta;}\left(R\right)=\frac{{\mathrm{SR}}^2}{2Q\left|R\right|}(\min (\mathrm{SR},\left|R\right|)\log (1+\left|R\right|)+2\log 6\mathrm{wh}),$ Among them, SR represents the dynamic range of the motion vector, |R| represents the number of motion vectors contained in the region, wh represents the size of the motion vector field, and the parameter Q is used to control the division degree of the motion vector field, so that the motion vector field can be moderately Segmentation into several regions with similar motion;

(4) Calculate the average residual error of each segmented region after global motion compensation;

剩下的区域作为运动对象可能存在的区域

最后对当前帧所分割的M个对象区域和1个背景区域分别标记，分割结果记为

(5) Distinguish the most reliable background region from the region where other objects are located. Among several segmented regions whose area is larger than 10% of the entire motion vector field, the region with the smallest average residual error is selected as a reliable background region, which is marked as

The remaining area is the area where the moving object may exist

Finally, the M object regions and 1 background region segmented by the current frame are respectively marked, and the segmentation result is denoted as

e. Object Segmentation

As shown in Figure 4, first find the matching blocks in two adjacent frames by calculation, then project the moving object of the previous frame to the current frame and mark it as object projection, and then use the correlation between the current frame object projection and the segmented area Construct three matrices CM ^t , CMR ^t , and CMC ^t with M+1 rows and N+1 columns. Then the matching matrix CMM ^t is generated from the matrices CMR ^t and CMC ^t , and based on this matching matrix, five different types of moving objects are segmented. The steps are detailed below:

和1个背景对象

标记出来，然后采用后向投影的方法获得前一帧各个对象在当前帧的投影区域。就是利用当前帧累积运动矢量场中任意块的坐标和其对应的累积运动矢量的差求出这个块在前一帧中的匹配位置，然后将前一帧匹配位置上的块对象投影到当前帧并逐个标记出来，记为

(1) Use the backward projection method to obtain the projection area of each object at the time t of the current frame at time t-1 in the previous frame. First move the N moving objects of the previous frame

and 1 background object

Mark it out, and then use the method of back projection to obtain the projection area of each object in the previous frame in the current frame. It is to use the difference between the coordinates of any block in the cumulative motion vector field of the current frame and its corresponding cumulative motion vector to find the matching position of this block in the previous frame, and then project the block object at the matching position of the previous frame to the current frame And marked out one by one, denoted as

构造3个M+1行N+1列的矩阵CM^t，CMR^t，CMC^t。其中矩阵CM^t中的任意元素CM^t(i，j)取值为在

中标记为i且在

标记为j的象素数目，即分割区域

与对象投影

相互重叠的面积。而矩阵CMR^t(i，j)定义为CMR^t第i行的各个元素是分割区域

落在各个对象投影内的比例；矩阵CMC^t(i，j)定义为CMC^t第j列的各个元素是对象的投影落在各个分割区域内的比例。(2) Construct a matrix CM ^t , which represents the overlapping area of the segmentation region and the object projection; construct a matrix CMR ^t , which represents the proportion of each segmentation region within each object projection; construct a matrix CMC ^t , which represents each object The proportion of projections that fall within each segmented region. according to tagged image and

Construct three matrices CM ^t , CMR ^t , and CMC ^t with M+1 rows and N+1 columns. Among them, any element CM ^t (i, j) in the matrix CM ^t takes a value in

marked i in and in

The number of pixels marked as j, that is, the segmented area

with object projection

overlapping area. The matrix CMR ^t (i, j) is defined as each element of the i-th row of CMR ^t is the segmentation region

The proportion that falls within the projection of each object; the matrix CMC ^t (i, j) is defined as each element of the jth column of CMC ^t is the object The proportion of the projection of which falls within each segmented area.

和

之间的相关信息。CMM^t首先置为M+1行N+1列的零矩阵；接着对CMR^t进行行扫描找到每一行最大值所在的位置，对CMM^t中相应位置处的元素值加1；然后对CMC^t进行列扫描找到每一列最大值所在的位置，对CMM^t中相应位置处的元素值加2。生成的矩阵CMM^t的纵坐标依次表示为当前帧背景区域和运动区域

横坐标依次表示为前一帧背景对象和运动对象

矩阵中各元素的可能取值为0，1，2，3。CMM^t中任意不为0的元素CMM^t(i，j)表明了分割区域

与对象

存在一定的相关性，具体而言：(3) Construct a matrix CMM ^t , which represents the degree of association between the segmented area of the current frame and the object projection. The matrix CMM ^t records the CMR ^t and CMC ^t reflected

and

related information between. CMM ^t is first set as a zero matrix of M+1 rows and N+1 columns; then row scan CMR ^t to find the position of the maximum value in each row, and add 1 to the element value at the corresponding position in CMM ^t ; then CMC ^t Perform column scanning to find the position of the maximum value of each column, and add 2 to the element value at the corresponding position in CMM ^t . The vertical coordinates of the generated matrix CMM ^t are sequentially represented as the background area of the current frame and sports area

The abscissa represents the background object of the previous frame in sequence and moving objects

The possible values of each element in the matrix are 0, 1, 2, 3. Any non-zero element CMM ^t (i, j) in CMM ^t indicates the segmentation area

with the object

There are certain correlations, specifically:

①CMM ^t (i, j)=1, indicating the segmented area Objects that belong largely to the previous frame

中；②CMM ^t (i, j)=2, indicating that the object in the previous frame largely contained in the segmented area

middle;

和

具有极强的相关性。需要进一步比较，如果CMR^t(i，j)>CMC^t(i，j)，则CMM^t(i，j)＝1；否则，CMM^t(i，j)＝2。最后生成的CMM^t取值范围为0，1，2。③ CMM ^t (i, j) = 3, which includes the above two situations at the same time, indicating that

and

have a strong correlation. For further comparison, if CMR ^t (i, j)>CMC ^t (i, j), then CMM ^t (i, j)=1; otherwise, CMM ^t (i, j)=2. The finally generated CMM ^t ranges from 0, 1, 2.

(4) Carry out object segmentation based on the matching matrix CMM ^t for tracking and updating of a single object, appearance of new objects, merging of objects, splitting of objects and disappearance of objects. The relationship between the segmented area and the moving object can be effectively established through the matrix CMM ^t , which can effectively handle the following five situations in a unified manner:

只与对象

存在相关性，根据CMM^t(i，j)的取值采取不同的策略：如果CMM^t(i，j)＝2，采取更新策略，用当前帧的分割区域来表示更新后的对象，即 $O_j^t=R_i^t$ 。如果CMM^t(i，j)＝1，一般采取对象跟踪的策略，即用前一帧对象的投影来表示当前帧的对象，即 $O_j^t=\mathrm{Proj}\left(O_j^{t-1}\right);$ 另外，如果分割区域

同时还满足阈值条件： $\left|R_i^t\right|>T_s,$ 且 ${\mathrm{ER}}_i^t>\mathrm{\&alpha;}{\mathrm{ER}}_0^t,$ 其中T_s＝64，α＝1.5，

表示区域

所包含的运动矢量数目，

表示区域的平均残差，

表示背景的平均残差；则认为

是一个可靠的运动区域，可用来表示当前帧的运动对象，即 $O_j^t=R_i^t.$ ①Single object tracking and updating (1→1): If the i-th row of CMM ^t has only one non-zero element CMM ^t (i, j), and the j-th column also has only this non-zero element CMM ^t (i, j) , then the segmented region

only with object

There is a correlation, different strategies are adopted according to the value of CMM ^t (i, j): if CMM ^t (i, j) = 2, an update strategy is adopted, and the updated object is represented by the segmented area of the current frame, namely $o_j^t=R_i^t$ . If CMM ^t (i, j) = 1, the strategy of object tracking is generally adopted, that is, the projection of the object in the previous frame is used to represent the object in the current frame, that is $o_j^t=\mathrm{Proj}\left(o_j^{t-1}\right);$ Also, if splitting the region

Also meet the threshold condition: $\left|R_i^t\right|>T_{\mathrm{the\; s}},$ and ${\mathrm{ER}}_i^t>\mathrm{\&alpha;}{\mathrm{ER}}_0^t,$ where T _s =64, α=1.5,

Indicates the area

the number of motion vectors included,

Indicates the area the average residual error of

Represents the average residual of the background; then it is considered

Is a reliable motion area, which can be used to represent the moving object of the current frame, namely $o_j^t=R_i^t.$

在前一帧还是背景对象并不属于已有的任何运动对象。如果

同时满足上面①中的阈值条件，则可认为

是一个新出现的运动对象，记为 $O_{M+1}^t=R_i^t.$ ②A new object appears (0→1): If the i-th row of CMM ^t has only one non-zero element and is located in the first column, the value is 1, indicating that the segmented area

was the background object in the previous frame Does not belong to any existing motion object. if

At the same time meet the threshold conditions in ① above, it can be considered

is a new moving object, denoted as $o_{m+1}^t=R_i^t.$

可能与多个对象存在相关性。在这种情况下，只需要将前一帧的对象投影到当前帧作为当前帧的运动对象，实现对象的跟踪，即 $O_j^t=\mathrm{Proj}\left(O_j^{t-1}\right).$ In the above two cases of ① and ②, the number of non-zero elements in a row of CMM ^t is 1. If there are multiple non-zero elements in the i-th row of CMM ^t , it indicates a split region

There may be dependencies on multiple objects. In this case, it is only necessary to project the object of the previous frame to the current frame as the moving object of the current frame to realize the tracking of the object, namely $o_j^t=\mathrm{Proj}\left(o_j^{t-1}\right).$

中，则

表示了这些对象合并后的新对象，记为 $O_{M+1}^t=R_i^t$ 。在这种情况下，分割区域

往往包含了2个或2个以上具有十分相似的运动且在空间相互邻接的对象，则它们在当前帧作为一个新的合并对象而被分割出来。③ Object merging (m→1): If there are more than 2 elements with a value of 2 on the i-th row of CMM ^t except column 1, it indicates that more than 2 objects in the previous frame largely contain in the new partition

in, then

Represents the new object after these objects are merged, denoted as $o_{m+1}^t=R_i^t$ . In this case, the split region

Often contains two or more objects with very similar motion and adjacent to each other in space, they are segmented as a new merged object in the current frame.

在当前帧分裂成多个分割区域

即使这些区域在空间上并不邻接，在当前帧的分割中，仍然认为这些分割区域属于同一个对象，直到这些具有相同对象标记却在空间相互不邻接的多个分割区域在随后的若干帧中表现出不同的运动，则对这些分割区域赋予不同的对象标记，记为 $O_{s_i}^t=R_{s_i}^t,$ 以实现真正的对象分裂。④Splitting of objects (1→m): If there are more than 2 elements in the jth column of CMM ^t with a value of 1, it indicates that the previous frame object

split into multiple split regions at the current frame

Even if these regions are not adjacent in space, in the segmentation of the current frame, these segmentation regions are still considered to belong to the same object, until these multiple segmentation regions with the same object label but not adjacent to each other in space are in the following frames show different motions, then assign different object labels to these segmented regions, denoted as $o_{{\mathrm{the\; s}}_i}^t=R_{{\mathrm{the\; s}}_i}^t,$ to achieve true object splitting.

的投影落在当前帧的背景区域则认为

在当前帧消失。⑤ Disappearance of the object (1→0): If the jth column of CMM ^t has only 1 non-zero element and is located in the first row, the value is 2, indicating that the previous frame object

The shadow of falls on the background area of the current frame then think

Disappears at the current frame.

As mentioned above, five situations that may occur during the segmentation of moving objects in video sequences can be effectively dealt with. However, when the scene changes greatly, a tracking strategy is adopted for all objects in multiple consecutive frames, that is, all objects are projections of the objects in the previous frame, indicating that the segmentation regions of the current frame are closely related to the moving objects in the previous frame. Weak, so it will judge whether there is a new object according to the situation ②, and the moving object needs to be detected again.

The following is an example when the input video format is 352×288 CIF, and the H.264 encoder of JM8.6 version is used to encode the MPEG-4 standard test sequence as the H.264 compressed video for testing. The configuration of the H.264 encoder is as follows: Baseline Profile, IPPP, inserting 1 I frame and 3 reference frames every 30 frames, the search range of motion estimation is [-32, 32], the quantization parameter is 30, and the number of encoded frames is 300 frames. In the experiment, we calculated the cumulative motion vector field every 3 frames (the number of frames used in the motion vector accumulation process), and obtained a total of 100 frames of cumulative motion vector field to test the performance of the moving object segmentation algorithm proposed in this paper . Firstly, the region segmentation result is obtained from the current frame by accumulating the motion vector field, and then the moving object in the previous frame is projected to the current frame. Based on these two results, the segmentation method based on matching matrix is used to segment the moving object.

The typical standard test sequences Coastguard and Mobile are used as the input video for testing, and the experimental results are shown in Figure 5 and Figure 6, respectively. The first column in the two figures is the original image of the current frame, the second column is the region segmentation result of the current frame by the cumulative motion vector field segmentation, the third column is the projection area of the previous frame moving object in the current frame, and the second column is Column 4 is the moving object segmented from the current frame. The average processing time of each frame is 38ms, which can meet the requirement of 25fps for most real-time applications. Considering that the segmentation method in this paper is actually segmenting moving objects every 3 frames, for the given original video sequence, it is completely possible to segment corresponding moving objects while decoding in real time, even if each frame is required to be segmented To generate the corresponding moving objects, it only needs to perform object projection on the remaining frames, and the calculation amount is very small, and the moving objects can still be segmented in real time.

Experiment 1: The sequence Coastguard has obvious global motion. The camera first pans from right to left to track the small boat in the middle of the picture, and then moves from left to right to track the big boat that appears from the left of the picture. Line 1 in Figure 5 (frame 4 in the sequence) is the camera tracking the movement of the boat from right to left, line 2 in Figure 5 (frame 37 in the sequence) is the movement of the new object ship from left to right, line 3 in Figure 5 (frame 3 in the sequence Frame 61) shows that two moving objects, a large ship and a small ship, completely appear in the scene of the camera. Line 4 in Figure 5 (frame 208 of the sequence) shows that the camera starts to track the movement of the large ship from left to right. From the images in the second column of Figure 5, it can be seen that most of the segmentation of the accumulated motion vector field can accurately segment the areas where the two moving objects are located, and most of the background areas that conform to the global motion model are also included in a large In the segmentation area of , the white area represents the most reliable background area after motion compensation, so the accumulation and segmentation method of the motion vector field adopted in this paper is effective, and a moderate segmentation result can be obtained by using the motion vector information. Combining the projection area of each object in the previous frame shown in the third column in the current frame, using the moving object segmentation method based on the matching matrix, the moving object shown in the fourth column can be stably and reliably segmented in the entire sequence.

Experiment 2: Sequence Mobile has a more complex global motion. In addition to the pan and tilt motion of the camera, there is also an obvious zoom motion in the first half of the sequence. The scene in row 1 of Figure 6 (frame 4 of the sequence) includes a total of 3 moving objects. The small train pushes the ball to move on the track, while the wall calendar moves up and down intermittently, so the segmentation of moving objects is more difficult. It can be seen from the segmentation result in Fig. 6 that the moving object segmentation algorithm proposed by the present invention can segment the moving object through object projection when the moving object stops moving, as shown in the second line of Fig. 6 (the 43rd frame of the sequence) The ball and the wall calendar in line 3 of Figure 6 (frame 109 of the sequence). In addition, the experimental results in Figure 6 also show that the moving object segmentation algorithm in this paper can handle the merging and splitting of moving objects well. In line 3 of Figure 6 (frame 109 of the sequence), since the train is already pushing the ball seamlessly, two moving objects that are closely adjacent in space and have exactly the same motion are considered to have merged objects; Line 4 in Figure 6 (frame 160 of the sequence) pairs two objects with a gap, and when the motion levels are no longer the same, the two objects are divided into two regions, and the split of the two objects is truly realized.

Claims (6)

1. the H.264 compression domain motion object real time method for segmenting based on the coupling matrix is characterized in that the motion vector field normalization of the continuous multiple frames row iteration rear orientation projection that goes forward side by side is obtained the cumulative motion vector field; Then the cumulative motion vector field is carried out global motion compensation, adopting fast simultaneously, the statistical regions growth algorithm is divided into a plurality of zones according to kinematic similarity with the cumulative motion vector field; Utilize above-mentioned two aspect results, the motion Object Segmentation method based on the coupling matrix that adopts the present invention to propose is partitioned into the motion object, wherein can carry out multiple situations such as the appearance of the merging of the tracking of object and renewal, object and division, object and disappearance effectively in video sequence; Its step is as follows:

A. motion vector field normalization: from video H.264, extract motion vector field and carry out normalization on time domain and the spatial domain;

B. cumulative motion vector field: utilize the motion vector field of continuous multiple frames to carry out the iterative backward projection and obtain cumulative motion vector field more reliably;

C. global motion compensation: after carrying out overall motion estimation on the cumulative motion vector field, compensate to obtain each residual error of 4 * 4;

D. Region Segmentation: adopt the statistical regions growing method that the cumulative motion vector field is divided into a plurality of zones with similar movement;

E. Object Segmentation: adopt dividing method that the motion Object Segmentation is come out based on the coupling matrix.

2. the H.264 compression domain motion object real time method for segmenting based on the coupling matrix according to claim 1, it is characterized in that the normalized step of described motion vector field is: (1) time domain normalization: with the motion vector of present frame interval frame number, i.e. time domain distance divided by present frame and reference frame; (2) spatial domain normalization: give all 4 * 4 that this macro block covered greater than direct tax of each macroblock motion vector of 4 * 4 with all sizes.

3. the H.264 compression domain motion object real time method for segmenting based on the coupling matrix according to claim 1, the step that it is characterized in that described cumulative motion vector field is: (1) utilizes the present frame motion vector field of some frames afterwards, motion vector field to consecutive frame carries out rear orientation projection, multiply by the projection motion vector that addition behind the different scale factors obtains current block by motion vector exactly to each projecting block, the method for selecting of scale factor is: if the gross area of overlapping region greater than half of current block area, then the scale factor of each projecting block is taken as the gross area of the equitant area of this projecting block and current block divided by all projecting blocks and current block overlapping region; Otherwise the scale factor of each projecting block is taken as the ratio of its overlapping area and current block area; (2) begin the iteration accumulation to obtain the cumulative motion vector field of present frame from the back frame then.

4. the H.264 compression domain motion object real time method for segmenting based on the coupling matrix according to claim 1, it is characterized in that global motion compensation, be to adopt affine motion model estimation global motion vector field earlier, calculate each 4 * 4 residual error behind global motion compensation then.Step is as follows:

(1) adopt the affine motion model of 6 parameters to estimate the global motion vector field:

1. model parameter initialization: establish m=(m ₁, m ₂, m ₃, m ₄, m ₅, m ₆) be the parameter vector of global motion model, model parameter m ⁽⁰⁾Be initialized as: $m^{\left(0\right)}={\left[\begin{array}{cccccc}1& 0& \frac{1}{N}\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{mvx}}_i& 0& 1& \frac{1}{N}\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{mvy}}_i\end{array}\right]}^T;$

2. reject point not in the know:

At first calculating the present frame centre coordinate is (x _i, y _i) i piece at the estimation centre coordinate of former frame $(x_i^{\&prime;},y_i^{\&prime;}):\left[\begin{array}{c}{x}_i^{\&prime;}\\ y_i^{\&prime;}\end{array}\right]=\left[\begin{array}{cc}{m}_1& m_2\\ m_4& m_5\end{array}\right]\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]+\left[\begin{array}{c}{m}_3\\ m_6\end{array}\right],$ Motion vectors then

With original cumulative motion vector (mvx _i, mvy _i) deviation (ex _i, ey _i) be calculated as: $\begin{array}{c}{\mathrm{ex}}_i=x_i^{\&prime;}-x_i-{\mathrm{mvx}}_i\\ {\mathrm{ey}}_i=y_i^{\&prime;}-y_i-\mathrm{mv}{y}_i\end{array},$ Use this formula to calculate each prediction deviation (ex of 4 * 4 _i, ey _i), calculate the deviation amplitude quadratic sum at last

Histogram, reject those deviation amplitude quadratic sums in the histogram then greater than 25% motion vector;

3. model parameter is upgraded

Use the motion vector and the Newton-Raphson method that remain in the preceding step to upgrade model parameter, new model parameter vectors m in the l step iteration ^(l)Be defined as follows: m ^(l)=m ^(l-1)-H ^-1B, Hessian matrix H and gradient vector b are calculated as follows here:

$H=\left[\begin{array}{cccccc}\underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i^2& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i{y}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i& 0& 0& 0\\ \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i{y}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i^2& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i& 0& 0& 0\\ \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}1& 0& 0& 0\\ 0& 0& 0& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i^2& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i{y}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i\\ 0& 0& 0& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i{y}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i^2& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i\\ 0& 0& 0& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}1\end{array}\right]$

$b={\left[\begin{array}{cccccc}\underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i{\mathrm{ex}}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i{\mathrm{ex}}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{\mathrm{ex}}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{x}_i{\mathrm{ey}}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{y}_i{\mathrm{ey}}_i& \underset{i\&Element;R}{\mathrm{\&Sigma;}}{\mathrm{ey}}_i\end{array}\right]}^T$

The set of the piece that R representative here remains;

4. termination condition: repeating step 2. and 3. maximum 5 times, and if one of following two conditions be satisfied also finishing iteration in advance: (i) calculate m ^(l)-m _Static, obtain a difference value vector, if all less than 0.01, just being judged as, each the parameter component in this difference value vector belongs to the static situation of video camera, finishing iteration; M wherein _Static=[1 0001 0] ^TBe the global motion vector under the static situation of video camera; (ii) calculate m ^(l)And m ^(l-1)Difference, if the parameter component m of this difference ₃And m ₆Less than 0.01, and other parameter component is less than 0.0001, and then iteration finishes;

5. with the global motion model parameters vector m substitution that obtains $\left[\begin{array}{c}{x}_i^{\&prime;}\\ y_i^{\&prime;}\end{array}\right]=\left[\begin{array}{cc}{m}_1& m_2\\ m_4& m_5\end{array}\right]\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]+\left[\begin{array}{c}{m}_3\\ m_6\end{array}\right],$ Obtain the estimated coordinates of former frame

Obtain the global motion vector field at last

(2) calculate each residual error of 4 * 4 in global motion vector field and the cumulative motion vector field.

5. the H.264 compression domain motion object real time method for segmenting based on the coupling matrix according to claim 1, it is characterized in that described Region Segmentation, is to adopt the statistical regions growth algorithm that the cumulative motion vector field is divided into a plurality of zones with similar movement; Step is as follows:

(1) the differences in motion opposite sex of calculating any adjacent block group in the neighbours territory is measured;

(2) all adjacent block groups sort according to differences in motion opposite sex tolerance order from small to large;

(3) the minimum adjacent block of differences in motion opposite sex tolerance is made up also, to begin area growth process herein, when each region growing, current two piece groups belong to two adjacent zones respectively, then judge condition that whether these two zones merge be the difference of average motion vector in these two zones whether less than threshold condition:

$\mathrm{\&Delta;}\left(R\right)=\frac{{\mathrm{SR}}^2}{2Q\left|R\right|}(\min (\mathrm{SR},\left|R\right|)\log (1+\left|R\right|)+2\log 6\mathrm{wh}),$ Wherein SR represents the dynamic range of motion vector, | R| represents the motion vector number that the zone comprises, wh represents the size of motion vector field, and parameter Q is used for the dividing degree of controlled motion vector field, just motion vector field moderately can be divided into some zones with similar movement with regard to sample;

(4) calculate the mean residual of each cut zone behind global motion compensation;

(5) distinguish the zone at the most reliable background area and other object place, as background area reliably, be labeled as in zone that area is selected the mean residual minimum in greater than some cut zone of whole motion vector field 10%

The zone that remaining zone may exist as the motion object

Mark is distinguished in M subject area and 1 background area that present frame is cut apart at last, and segmentation result is designated as

6. the H.264 compression domain motion object real time method for segmenting based on the coupling matrix according to claim 1, it is characterized in that described Object Segmentation, be to utilize former frame, t-1 constantly, the motion Object Segmentation result who has obtained judges present frame, and t constantly, each cut zone whether with certain object coupling of former frame, construct the coupling matrix with this; Judge the division of merging, the object of the tracking of object and renewal, object, the appearance, the situations such as disappearance of old object of object newly based on the coupling matrix, finally obtain some motion objects of present frame; Step is as follows:

(1) adopt rear orientation projection's method to obtain former frame, in the t-1 moment, each object is in present frame t view field constantly, and elder generation is with N motion object of former frame

With 1 background object

Mark comes out, and adopts the method for rear orientation projection to obtain the view field of each object of former frame at present frame then; Utilize the coordinate of any piece in the present frame cumulative motion vector field and the difference of its corresponding cumulative motion vector to obtain the matched position of this piece in former frame exactly, then the block object on the former frame matched position is projected to present frame and one by one mark come out, be designated as

(2) structural matrix CM ^t, its expression cut zone and overlapped area of object projection; Structural matrix CMR ^t, it represents that each cut zone drops on the ratio in each object projection; Structural matrix CMC ^t, it represents that each object projection drops on the ratio in each cut zone; According to the mark image With

Construct the Matrix C M of 3 capable N+1 row of M+1 ^t, CMR ^t, CMC ^tMatrix C M wherein ^tIn arbitrary element CM ^t(i, j) value be

In be labeled as i and Be labeled as the number of pixels of j, i.e. cut zone

With the object projection

Overlapped area; And Matrix C MR ^t(i j) is defined as CMR ^tEach element that i is capable is a cut zone Drop on the ratio in each object projection; Matrix C MC ^t(i j) is defined as CMC ^tEach element of j row is an object Projection drop on ratio in each cut zone;

(3) structural matrix CMM ^t, the correlation degree between its expression present frame cut zone and the object projection, Matrix C MM ^tWrite down CMR ^tAnd CMC ^tReflected

With

Between relevant information; CMM ^tAt first be changed to the null matrix that M+1 is capable, N+1 is listed as; Then to CMR ^tCarry out line scanning and find the position at each row maximum place, to CMM ^tThe element value of middle corresponding position adds 1; Then to CMC ^tCarry out column scan and find the position at each row maximum place, to CMM ^tThe element value of middle corresponding position adds 2; The Matrix C MM that generates ^tOrdinate be expressed as the present frame background area successively

And moving region

(i=1,2 ... M) abscissa is expressed as the former frame background object successively

With the motion object

(i=1,2 ... M) the possible value of each element is 0,1,2,3 in the matrix; CMM ^tIn arbitrarily be not 0 Elements C MM ^t(i j) has shown cut zone

With object

There is certain correlation, particularly:

1. CMM ^t(i j)=1, shows cut zone

Belong to the former frame object to a great extent

2. CMM ^t(i j)=2, shows the former frame object

Be included in cut zone to a great extent In;

3. CMM ^t(i j)=3, has comprised above-mentioned two kinds of situations simultaneously, shows

With

Has extremely strong correlation; Need further relatively, if CMR ^t(i, j)〉CMC ^t(i, j), CMM then ^t(i, j)=1;

Otherwise, CMM ^t(i, j)=2; The CMM of Sheng Chenging at last ^tSpan is 0,1,2;

(4) based on coupling Matrix C MM ^tTracking and renewal, new object appearance, the merging of object, the division of object and the disappearance five class situations of object to single object are carried out Object Segmentation; By Matrix C MM ^tCan set up the incidence relation of cut zone and motion object effectively, it can handle following five kinds of situations effectively with a kind of uniform way:

1. single to image tracing and renewal (1 → 1);

(0 → 1) appears in 2. new object;

3. the merging of object (m → 1);

4. the division (1 → m) of object;

5. the disappearance of object (1 → 0).

Images