JPH10112863A - Moving subject information extraction method and apparatus - Google Patents

Abstract

(57) [Problem] To provide a moving subject information extraction method capable of extracting correct moving subject information from video compression encoded data including a motion vector irrelevant to a moving subject. SOLUTION: In the moving subject information extracting method for calculating moving subject information from a motion vector included in video compression encoded data encoded by a compression encoding method using a motion prediction method, the video compression encoded data includes A first step of removing, as noise, a motion vector irrelevant to the moving object from the included motion vectors, and a moving object corresponding to the moving object from the motion vector from which the irrelevant motion vector has been removed in the first step. The method includes a second step of extracting a region, and a third step of extracting moving subject information from the moving subject region extracted in the second step.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for extracting moving subject information, and more particularly to moving subject information for extracting correct moving subject information from video compression encoded data including a motion vector unrelated to a moving subject. The present invention relates to an extraction method and apparatus.

[0002]

2. Description of the Related Art The amount of video data is enormous, and the only way to know its contents is to look at all the videos in chronological order.

[0003] By dividing an image based on a certain criterion and using it as an index, it is useful for skipping the image and understanding the contents roughly. The following gazette (a) describes a method of dividing an image at a transition between scenes called a cut point in such image division.

Further, in order to further divide an image in the same scene, a method of dividing an image by using camerawork information as an index by using the fact that the content of the image changes due to camerawork or the like is disclosed in the following document. It is described in (b).

(A) Japanese Patent Laid-Open No. 5-37853, "Automatic Video Cut Division Method" (b) "Proposal of Video Unit Using Time-Series Optical Flow" (Image Coding Symposium PCSJ91) Many video compression encoding methods have been proposed to have data strings and to make them easier to handle. Among them, the MPEG method, which is an encoding method including motion prediction, is a typical method.

[0006] In each prediction coded image data in the image data sequence coded by the MPEG system, each image is divided into partial areas called macroblocks (MB), and coding is performed for each area.

At this time, three types of coding systems are selected.
One is an intra-coding method called Intra,
One is a predictive coding method using a motion vector called MC Code, and the other is No MC Code.
This is a predictive coding method called d which does not use a motion vector.

[0008] The selection of these coding methods is performed according to the following rules.

If (var <vmc) MBType = "
Intra “else if (v0 <α * vmc) MBType =”
No MC Coded “else MBType =“ MC Coded ”where var is the variance of the pixel value of the macroblock (MB), vmc is the variance of the pixel value difference between frames when motion prediction is performed, and v0 is the motion prediction Is the variance of pixel values between frames when is not used, and α is a bias value of 1.25
A degree value is used. These selections are made because the smaller the variance, the higher the coding efficiency.

When the image content changes due to the presence of camera work or a moving subject, predictive coding using a motion vector reflecting motion information of camera work, a moving subject, or the like can be performed by selecting the above-described coding method. Done.

When extracting camera work information from an original image data sequence in which camera work exists, conventionally,
After decoding once, camerawork information is extracted, which requires costly calculation processing.

For this reason, a method for extracting motion information such as camera work information using motion vector information included in encoded information is described in the following literature (c) or (d).

(C) "Extraction of moving area using MPEG data" (IEICE Technical Report IE96-25, p. 69-) (d) "Video Parsing Using Compressed Data" (SPI
E Vol.2182 Image and Video Processing II 1994)

[0014]

In the method described in the above document (c) or (d), image data is encoded under an ideal environment, and motion information is extracted from the extracted motion vectors. Has been extracted.

However, noise or screen flicker occurs in an actual image. For this reason, when video data obtained by encoding an analog video signal actually captured by a camera is targeted, a motion vector irrelevant to motion information (hereinafter referred to as a noise motion vector) due to flickering of screen or noise. ) Appear in large numbers.

In particular, a large number of noise motion vectors that hinder the extraction of such motion information appear when there is a wide area extending over several blocks with uniform brightness and luminance.

That is, when various lightness and color components are included in a macroblock (MB), each macroblock (M
In B), various pixel values are distributed. Due to the restriction of the prediction based on the position information, the screen is not greatly influenced by the flickering or the noise of the screen, and the motion prediction is performed in an almost ideal environment. Can be obtained.

On the other hand, if a uniform area having a small brightness and little color change exists in an image over several macroblocks (MB), ideally, the motion vector 0,
When the region is relatively moving, a motion vector 0 inside the region and a motion vector reflecting the motion in the outline portion should be observed.

However, since it is originally a uniform area in which both brightness and color do not change, if coding is actually performed,
Even a small amount of screen flicker or noise will increase the variance of the pixel values of the inter-frame difference.

Therefore, the motion vector does not become 0 but has another value. Furthermore, since flickering or noise on the screen appears at random, a motion vector appears at random under this influence. That is, a noise motion vector appears.

As described above, when the content of the image data is stationary, only the zero vector should ideally appear. However, if a uniform area exists, in the actual image, not the zero vector but the random vector is generated. Noise motion vector appears.

FIG. 14 shows an example of the distribution of motion vectors when there is a moving subject under an ideal environment, when there is a camera work, and when there is noise.
Shown in

FIG. 3A shows a case where a moving subject exists in an original image under an ideal environment, and FIG. 3B shows a case where a moving subject exists in an original image under an ideal environment. In addition, when camera work (pan) is present, FIG. 3C shows an environment where noise exists, and when a moving subject exists in the original image, FIG. Shows a case where a moving subject exists in the original image and camera work (pan) exists.

According to the dividing method described in the above-mentioned document (b), it is possible to further divide an image in the same scene. However, if the moving subject information can be extracted from the motion vector as information that goes into the video content rather than the camera work information, it is effective as an index for finely dividing the video.

When extracting moving subject information from a motion vector, a process for extracting a motion vector reflecting camera work (hereinafter, referred to as a camera work motion vector) is required. However, since the above-mentioned noise motion vector is a random motion vector, when extracting the moving subject information from the motion vector, in some cases, the camera work motion vector cannot be extracted, or the noise motion vector May be extracted as a single moving subject area.

Therefore, in order to extract a moving subject area from a motion vector, a noise motion vector is removed,
A process for separating a moving subject area from the corrected motion vector from which the noise component has been removed and extracting moving subject information is required.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems of the prior art, and an object of the present invention is to provide a method and apparatus for extracting moving subject information, the method comprising the steps of: It is an object of the present invention to provide a technique capable of extracting correct moving subject information from encoded data.

The above and other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.

[0029]

SUMMARY OF THE INVENTION Among the inventions disclosed in the present application, the outline of a representative one will be briefly described.
It is as follows.

In a moving subject information extracting method for calculating moving subject information from a motion vector included in video compression encoded data encoded by a compression encoding method using a motion prediction method, the video compression encoding data includes Removing a motion vector irrelevant to the moving subject from the motion vectors to be removed as noise, and a moving subject area corresponding to the moving subject from the motion vector from which the irrelevant motion vector has been removed in the first step. And a third step of extracting moving subject information from the moving subject region extracted in the second step.

In the first step, camera work information irrelevant to the moving subject area is extracted in advance, and a motion vector reflecting camera work is removed.

In the second step, a set of motion vectors having characteristics similar to those of temporally or spatially adjacent motion vectors is separated and extracted as a moving subject area.

In a moving subject information extracting apparatus for calculating moving subject information from a motion vector included in video compression encoded data encoded by a compression encoding method using a motion prediction method, the moving object information extracting device includes: A preprocessing unit that removes a motion vector unrelated to the moving subject from the motion vector to be removed as noise, and a moving subject area corresponding to the moving subject from the motion vector from which the motion vector unrelated to the moving subject has been removed by the preprocessing unit. And a moving subject information extracting unit that extracts moving subject information from the moving subject region extracted by the moving subject region separating unit.

The preprocessing unit includes a camerawork removing unit that extracts camerawork information irrelevant to the moving subject area in advance and removes a motion vector reflecting camerawork.

The moving subject region separating section separates and extracts a set of motion vectors having characteristics similar to those of temporally or spatially adjacent motion vectors as a moving subject region.

When a set of motion vectors exhibiting characteristics similar to the temporally or spatially adjacent motion vectors is separated and extracted as a moving subject region, at least one feature amount of the motion vectors is calculated, and The feature values are compared between the two motion vectors, and if the comparison result satisfies a predetermined value, it is determined that the two adjacent motion vectors are similar.

According to the above means, a motion vector irrelevant to the moving subject is removed from the motion vector distribution including a motion vector irrelevant to the moving subject, and a correct moving subject is removed from the motion vector from which the irrelevant motion vector has been removed. Is selected, so that the correct moving subject information can be extracted even in an ideal environment, so that the content of the original image can be extracted without decoding the original image. It is possible to guess.

According to the above means, the motion vector reflecting the camerawork is removed in advance, and the motion vector reflecting the correct moving subject is selected from the motion vectors from which the motion vector reflecting the camerawork has been removed. Even in an ideal environment, correct moving subject information can be extracted.

According to the above-mentioned means, a set of motion vectors having characteristics similar to those of temporally or spatially adjacent motion vectors is separated and extracted as a moving subject area. It is possible to do.

[0040]

Embodiments of the present invention will be described below in detail with reference to the drawings.

In all the drawings for explaining the embodiments, parts having the same functions are denoted by the same reference numerals, and their repeated description will be omitted.

FIG. 1 is a diagram for explaining a moving subject information extracting method according to an embodiment of the present invention.

The moving subject information extracting method of the present embodiment targets only the inter-frame predictive coded image data sequence 1 having a motion vector from the input MPEG data sequence 10. In order to facilitate extraction of a moving subject vector, a noise motion vector and a camerawork motion vector are removed from these inter-frame prediction coded image data strings 1 in a pre-processing stage 2, and a corrected motion vector 3 is extracted. .

Next, in a moving subject area separating step 4, a moving subject area vector 5 which is a motion vector relating to the moving subject is separated from the corrected motion vector 3.

Next, in the moving subject information extracting step 6, moving subject information (the size of the moving subject region, the average and maximum and minimum of the size of the motion vector, etc.) is extracted from the moving subject region vector 5.

FIG. 2 is a block diagram showing a schematic configuration of a video reproducing apparatus having a moving subject information extracting apparatus according to an embodiment of the present invention.

As is well known, in the video reproducing apparatus shown in FIG. 2, an input MPEG data string 20 is composed of a variable length decoder 60, an inverse scanning section 61, an inverse quantization section 62, an inverse DC
The T unit 63, the adder 64, and the motion compensating unit 65 form a reproduced (decoded) image data sequence 66 through variable-length decoding, inverse quantization, inverse DCT, and motion compensation. The reproduced (decoded) image data sequence 66 is displayed on the display by the reproduction display section 67.

In the video reproducing apparatus shown in FIG.
The input coded data may be compression coded information data including a motion prediction coding method, and may be MPEG1 or MP1.
Either of EG2 and H261 or other encoding may be used. The data format in MPEG may be either inter-frame prediction coding or inter-field prediction coding. Furthermore, the motion prediction method may be any one of the forward direction, the bidirectional direction, and the reverse direction, as long as it has a motion prediction vector.

In the video reproducing apparatus shown in FIG.
One unit of the PEG data 20 is an I-picture composed of only Intra MBs (macroblocks), and fifteen frames composed of 14 frames of a P-picture employing an inter-frame forward motion prediction coding system. And these are combined into one GOP (Gro
"IPP" configured as "up of Picture"
PPPPPPPPPPPPPIPPPPPPPPPPPPP
The data format is “PP”.

In the moving subject information extracting apparatus according to the present embodiment, first, the shot unit classifying section 21
From the DCT coefficients output from the second unit, the shot unit (S
t, St + 1, St + 2,...) 22.

The shot unit classification unit 21 classifies shots into shot units 22, for example, as described in "MPEG
A known method described in "Method for Detecting Cut Points from Two Images" (Aikura, Taniguchi, Akutsu, Hamada, IEICE Autumn Information and Communication Society Conference, 1996) can be used.

Next, the encoding information analysis unit 23 extracts an inter-frame prediction encoded image data sequence (Pt, Pt + 1, Pt + 2,...) 24 included in each shot for each shot unit 22. I do. That is, in the moving subject information extraction device of the present embodiment, only the inter-frame prediction coded image data sequence 24 included in each shot is to be processed.

Next, in the noise vector removing section 25, the inter-frame prediction coded image data sequence 2 included in each shot
4 to obtain a corrected motion vector (FPt, FPt + 1, FPt + 2,...) 26. The noise vector removing unit 25 will be described later.

Next, in the camera work removing section 27, the corrected motion vectors (FPt, FPt + 1, FPt + 2,...) 26
, The camera work motion vector is removed, and the re-corrected motion vector (F2 Pt, F2 Pt + 1,
F2 Pt + 2,...) 28 are obtained. The camera work removing unit 27 will be described later.

In the present embodiment, the camera work is intended only for camera work such as pan, tilt and track, and is not intended for camera work such as zoom and dolly whose focal length changes.

Next, in the moving subject area separating section 29, the re-corrected motion vector (F2P) obtained by removing the camera work motion vector and the noise motion vector from the motion vector.
t, F2Pt + 1, F2Pt + 2,...) 28, a region where similar motion vectors are concentrated is obtained, and a moving subject region (Pt ｛V1, V2,...｝) 30 is separated. The moving subject area separating section 29 will be described later.

Next, the moving subject information extracting section 31 obtains moving subject information (Pt ｛IV1, IV2,...) 3 based on the obtained moving subject area (Pt ｛V1, V2,...｝) 30.
2 is extracted. Also, the moving subject information (Pt @ IV
1, IV2,...｝) 32 to calculate a feature amount to facilitate use as an index.

This moving subject information (Pt @ IV1, IV
2,...) 32 include, for example, the number of moving subject areas, the size of the moving subject areas,
Information such as the density of the motion vector, the average and maximum and minimum of the magnitude of the motion vector, the direction and magnitude of the average motion vector, and how the moving subject area has changed with time are calculated.

The shot unit (St, St + 1, St + 2,...) 22 from the shot unit classifying unit 21 and the moving subject information from the moving subject information extracting unit 31 are input to the reproduction display unit 67. As a result, the reproduction (decoding) image data sequence 66 can be displayed in the reproduction display unit 67 in units of shots or moving objects.

Hereinafter, the noise vector removing section 25 of the present embodiment will be described in detail.

FIG. 3 is a schematic diagram showing one image in an image sequence taken by a camera performing panning from right to left, and FIG. 4 shows a prediction of the original image shown in FIG. FIG. 6 is a diagram illustrating a distribution of motion vectors when encoding.

In FIG. 4, a motion vector generated in a uniform area such as an empty area surrounded by a circle is a vector irrelevant to panning or a moving subject. In the present embodiment, such a motion vector that does not reflect camera work or a moving subject is treated as noise.

As is clear from FIG. 4, a motion vector (noise motion vector) that does not reflect camera work or a moving subject has a solid area in which brightness and color are uniform and there is no change in a plurality of macroblocks (MB). In the case where it exists over the entire screen, it is caused by flickering in the image, disturbance of the signal, and the like.

That is, when a uniform area having a small brightness and little color change over several macroblocks (MB) is present in an image, ideally only 0 vector should be extracted. When calculating the motion vector at the time, the value that minimizes the variance of the pixel difference of the motion prediction is selected, so the area with uniform brightness and color is easily affected by noise and screen flicker. , A noise motion vector is generated.

Among the noise motion vectors, a noise motion vector which becomes an obstacle when calculating the moving subject information is a case where the size of the motion vector is large and no vector similar in time and space exists.

It should be noted that motion vectors are similar when, for example, the size ratio is less than or equal to 2 and the angle difference is within 10 degrees, this is treated as similar.

FIG. 5 is a flowchart showing an example of the processing procedure of the noise vector removing unit 25 of the present embodiment.

Hereinafter, an example of the processing procedure of the noise vector removing unit 25 of the present embodiment will be described in detail with reference to FIG.

In the processing procedure shown in FIG. 5, when there is no similar motion vector among spatially adjacent motion vectors for each motion vector, the processing procedure for determining this motion vector as a noise motion vector is performed. It is.

First, the variable (i) is set to 0 (step 1).
00), it is determined whether there is a motion vector spatially adjacent to the motion vector of interest (step 101).

That is, in step 101, the size of the motion vector of interest is large to some extent, and among the motion vectors spatially adjacent to the motion vector of interest, If there is no similar vector, the motion vector of interest is determined to be a noise motion vector.

Next, in step 101, if there is a motion vector spatially adjacent to the current motion vector, the value of the motion vector is changed to an irrelevant value (NULL). (Step 102).

That is, in step 102, if the current motion vector is determined to be a noise motion vector, this motion vector is removed as a noise motion vector.

Next, it is determined whether or not the series of processing has been completed for all motion vectors (step 103). If the series of processing has not been completed for all motion vectors, the variable (i) is incremented by +1. Then, step 101 to step 102 are repeated.

Since the feature of these noise motion vectors is coding of an area having a large size and originally having a small brightness or color change, the motion prediction residual has only a very small value due to flicker. Most DCT coefficients are also 0
It has the characteristic of becoming.

Then, the magnitude L of the motion vector {vi}
{Vi} and direction D {vi}, DC component DC of DCT coefficient
{Vi}, the number NAC of non-zero DCT coefficients among the AC components of the DCT coefficients NAC {vi}, the number NS of motion vectors included in adjacent macroblocks (MB) having a high degree of similarity to the target motion vector Let vi｝ be a feature quantity of a motion vector, and determine whether the motion vector of interest is a noise motion vector based on whether each feature quantity satisfies a threshold given to each. Good.

As a result, as shown in FIG. 6, most of the noise motion vectors unrelated to the moving subject can be removed from the motion vectors shown in FIG.

FIG. 6 shows the motion vectors shown in FIG.
It is a figure which shows the motion vector distribution after removing the motion vector determined as the noise motion vector.

Hereinafter, the camera work removing section 27 of the present embodiment will be described in detail.

The camera work motion vector has a feature that it appears many times on the entire screen. In the case of a characteristic motion such as panning or tilting, in an ideal environment, the value of the cumulative sum of motion vectors by angle shows a peak at a specific angle corresponding to the panning or tilting direction.

As is clear from FIGS. 4 (b) and 6 (b), the cumulative sums of the motion vectors before and after the noise motion vector elimination are compared by angle. However, it can be seen that a peak coinciding with the panning direction was observed.

The definition of the direction and angle of the motion vector shown in FIG. 4 or FIG. 6 is as shown in FIG. The method of extracting the cumulative sum for each angle is as follows.

Angle = 0 to 360 deg for the total number of macroblocks (MB) Angle = direction V Hist (Angle) + = Length (V) where V is a motion vector on the macroblock (MB) and Angle (V) is Motion vector direction, Leng
th (V) is the length of the motion vector, Hist (Angl
e) shows the components of the cumulative sum for each angle.

Therefore, the camerawork removing section 27 of the present embodiment creates an angular frequency distribution of motion vectors, and determines the motion vector having the highest frequency as a camerawork motion vector.

FIG. 8 is a flowchart showing an example of the processing procedure of the camera work removing section 27 of the present embodiment.

Hereinafter, an example of the processing procedure of the camera work removing section 27 of the present embodiment will be described in detail with reference to FIG.

First, the variable (i) is set to 0 (step 1).
10), an angular frequency distribution of the motion vector of interest is created (step 111).

Next, it is determined whether or not the processing of step 111 has been completed for all motion vectors (step 11).
2) If the processing of step 111 has not been completed for all motion vectors, the variable (i) is incremented by 1 (step 113), and step 110 is repeated.

In step 112, if the processing in step 111 has been completed for all motion vectors, the motion vector having the maximum angular frequency distribution is set as the camera work motion vector (CV) (step 114).

Next, the variable (i) is set to 0 (step 11).
5) It is determined whether or not the motion vector of interest is a camera work motion vector (CV) (step 116).
If the motion vector of interest is a camera work motion vector (CV), the value of the motion vector is changed to an irrelevant value (NULL) (step 117).

Next, it is determined whether or not the processing in step 117 has been completed for all the motion vectors (step 11).
8) If the processing in step 117 has not been completed for all the motion vectors, the variable (i) is incremented by 1 (step 119), and the steps 116 to 11 are performed.
Repeat step 7.

When determining the camera work motion vector, the motion vector in an appropriate range may be determined as the camera work motion vector by looking at the dispersion of the angular frequency distribution.

Hereinafter, the moving subject area separating section 29 of the present embodiment will be described in detail.

The moving subject region has a characteristic of being concentrated as a region. Therefore, in the present embodiment, a region where similar motion vectors are spatially concentrated is defined as a moving subject region.

Since the noise motion vector and the camera work motion vector have already been removed, it is easy to separate the moving subject area. All the motion vectors are compared with spatially adjacent motion vectors, and when the similarity condition is satisfied, the same labeling is applied to the motion vector of interest and the adjacent similar vector.

If an adjacent similar motion vector has already been labeled with another vector, the same labeling is performed together with the third motion vector. In this way, the connection between adjacent objects is strengthened, and a huge area is gradually created.

FIG. 9 is a flowchart showing an example of the processing procedure of the moving subject area separating section 29 according to the present embodiment.

Hereinafter, the subject area separating unit 2 of the present embodiment will be described.
An example of the processing procedure of No. 9 will be described in detail with reference to FIG.

First, the variable (i) is set to 0 (step 1).
20) It is determined whether there is a motion vector similar to the motion vector of interest (step 12).
1).

In step 121, if there is a motion vector similar to the current motion vector,
It is determined whether the similar motion vector has already been labeled with another motion vector (step 12).
4).

In step 124, if the similar motion vector is already labeled with another motion vector, the target motion vector is changed to an area in which the similar motion vector is already labeled. (Step 125).

If it is determined in step 124 that the similar motion vector has not been labeled with another motion vector, the current motion vector is labeled with the similar motion vector (step 124). 126).

Next, it is determined whether or not the above series of processing has been completed for all the motion vectors (step 122). If the series of processing has not been completed for all the motion vectors, the variable (i) is changed. +1 (step 123),
Step 121, Step 122, Step 12
4. Steps 125 and 126 are repeated.

In the processing procedure shown in FIG. 9, a primitive example in which adjacent ones are compared has been described. (For example, m × m macroblocks (MB), etc.), and the feature amount of the motion vector between adjacent regions (the size, direction, etc. of the average vector)
May be calculated, and the moving subject areas may be separated by comparing the respective areas and merging the areas.

Although the processing procedures shown in FIGS. 5, 6 and 9 have been described mainly with respect to spatial processing, since they are moving subjects, they are subjected to temporal processing to achieve higher accuracy. A moving subject area is also possible.

Next, the processing procedure of the noise vector removing unit 25 taking into account the temporal processing procedure, the camera work removing unit 2
7 and the processing procedure of the moving subject area separation unit 29 will be described.

FIG. 10 is a flowchart showing another example of the processing procedure of the noise vector removing unit 25 of the present embodiment.

Hereinafter, another example of the processing procedure of the noise vector removing unit 25 of the present embodiment will be described in detail with reference to FIG.

First, the variable (i) is set to 0 (step 1).
100) Also, the counter (Counter) is reset (step 1101).

Next, for the motion vector of interest, in steps 1102 to 1105, it is determined whether there is a motion vector spatially or temporally adjacent to the motion vector of interest. Is determined, and if there is a motion vector temporally or spatially adjacent to the motion vector of interest, the counter (Counter) is incremented by one.

Next, it is determined whether or not the value of the counter (Counter) is larger than the threshold value (1) (step 1).
106), and in step 1106, a counter (Count
If the value of (er) is smaller than the threshold value (1), the value of the motion vector is changed to an irrelevant value (NULL) (step 1107).

Next, it is determined whether or not the above series of processing has been completed for all motion vectors (step 1108).
If the series of processing has not been completed for all the motion vectors, the variable (i) is incremented by 1 (step 1109),
Steps 1101 to 1108 are repeated.

That is, in the processing procedure shown in FIG. 10, not only a motion vector that is spatially similar to a motion vector of interest but also a motion vector that exists at a position adjacent to a temporally past frame is compared. And calculate how many similar motion vectors exist. If the number of similar motion vectors is small, it is determined to be a noise motion vector.

FIG. 11 is a flowchart showing another example of the processing procedure of the camera work removing unit 27 of the present embodiment.

If camera work exists, a continuous motion vector appears over several frames. That is,
Motion vectors in frames that are temporally adjacent to each other are similar. Therefore, in the processing procedure of the camera work removing unit 27 shown in FIG. 11, steps 1110, 1115, and 116 are added to the processing procedure of the camera work removing unit 27 shown in FIG. 8, and the angles of the motion vectors over three frames are added. The distribution is calculated by the processing of steps 1110 to 1116, and the peak value is used as camera work.

In the processing procedure shown in FIG. 11, since the camera work is calculated from the angular frequency distribution over a plurality of frames, a sudden change of only one frame due to a flash or the like, a sudden noise motion vector of only one frame, or the like. And the camera work vector can be accurately determined.

FIG. 12 is a flowchart showing another example of the processing procedure of the moving subject area separating section 29 according to the present embodiment.

Hereinafter, the subject area separating section 2 according to the present embodiment will be described.
Another example of the processing procedure of No. 9 will be described in detail with reference to FIG.

In the processing procedure of the moving subject area separation unit 29 shown in FIG. 9, labeling is performed spatially one by one. However, in the processing procedure shown in FIG. A motion vector at a corresponding position is labeled on each area with reference to the area.

Next, the variable (i) is set to 0 (step 11).
21) It is determined whether or not the motion vector of interest is labeled (step 1122).

In step 1122, if the current motion vector is labeled, it is determined whether the current motion vector is similar to an adjacent motion vector having the same label (step 112).
3).

If it is determined in step 1123 that the motion vector of interest is not similar to an adjacent motion vector of the same label, the label is initialized (step 1).
124), if the motion vector of interest is similar to an adjacent motion vector of the same label, its labeling is not changed.

Next, if the motion vector of interest is not labeled in step 1122, or if the label is initialized in step 1124, a motion vector similar to the motion vector of interest exists. It is determined whether or not to perform (step 1125).

In step 1125, if there is a motion vector similar to the current motion vector, it is determined whether the similar motion vector has already been labeled with another motion vector (step 1125). 1
126).

If it is determined in step 1126 that the similar motion vector has already been labeled with another motion vector, the target motion vector is changed to an area in which the similar motion vector is already labeled. (Step 1128).

If it is determined in step 1126 that the similar motion vector has not been labeled with another motion vector, the target motion vector is labeled with the similar motion vector (step 1126). 1127).

Next, it is determined whether or not the above series of processing has been completed for all motion vectors (step 1129).
If the series of processing has not been completed for all motion vectors, the variable (i) is incremented by 1 (step 1130),
Steps 1122 to 1129 are repeated.

According to the processing procedure shown in FIG. 12, labeling is performed in advance by using past area information, so that the speed of labeling can be reduced.

FIG. 13 is a flowchart showing another example of the processing procedure of the moving subject area separating section 29 according to the present embodiment.

Hereinafter, the subject area separating section 2 of the present embodiment will be described.
Another example of the processing procedure of No. 9 will be described in detail with reference to FIG.

In the processing procedure shown in FIG. 13, a moving subject region candidate 1147 is calculated for each frame in steps 1140 to 1146. Note that the processing of steps 1140 to 1146 is the same as that of FIG.
And the detailed description is omitted.

Next, the obtained moving subject area candidate 114
7 as the next area candidate 1148,
It is determined how much the overlap between the obtained moving subject region candidate 1147 and the past moving subject region candidate 1149 overlaps (step 1150).

In general, when a moving subject area actually exists, a moving subject area should exist in a similar size and position in a past frame.

For this reason, in the processing procedure shown in FIG. 13, in step 1150, the obtained moving subject region candidate 1147 on the frame of interest is compared with the previously calculated past moving subject region candidate 1149, and Only when a large number of past moving subject region candidates 1149 exist at positions corresponding to the moving subject region candidates 1147 on the frame, the moving subject region candidate 1147 on the target frame is determined to be a correct moving subject region.

It is to be noted that whether or not the moving subject area is correct is determined by the moving subject area candidate 1 on the target frame of interest.
The determination is made based on how much the motion vector overlaps between 147 and the past moving subject region candidate 1149. In this comparison, not only the number of overlapping motion vectors but also the direction and magnitude of the motion vectors included in each area may be determined as to whether or not they overlap.

In the present embodiment, only the pan and the tilt are targeted, and the zoom and the dolly are not targeted when the camera motion vector is determined. However, it is also possible to detect this by incorporating a new operation when determining the camera work motion vector. Since some conventional methods have already been proposed, such a method may be applied.

In the present embodiment, the camera work motion vector is removed in advance to facilitate the separation of the moving subject area. However, the processing for removing the camera work motion vector is performed in the preprocessing unit. Instead, the moving subject area may be separated.

In this case, the camera work motion vector group is also determined as one moving subject area. However, in general, the camerawork motion vector spreads over the entire screen and is the largest area. Therefore, what has the largest area may be separated as the camera work area by using these features.

In the present embodiment, the threshold value used to determine the similarity vector is given in advance. However, the threshold value may be changed as appropriate according to the contents of the video or image. The tuning may be performed by hand and information may be given as appropriate.

As described above, according to the moving subject information extracting apparatus of the present embodiment, moving subject information is extracted from a compressed image data sequence, and is combined with cut point information extracted by a conventional method. Thereby, the contents of the compressed image data sequence can be browsed at high speed and easily on the basis of special information in the video contents such as cut points and moving subject information. Further, the moving subject information extracting apparatus according to the present embodiment can be easily realized only by describing its specifications by software and executing it on a general-purpose computer.

As described above, the invention made by the present inventors is described below.
Although the present invention has been described in detail with reference to the embodiment, the present invention is not limited to the embodiment, and it is needless to say that various changes can be made without departing from the scope of the invention.

[0142]

The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.

(1) According to the present invention, in the moving subject information extracting method and apparatus, even if the environment is not ideal,
It is possible to extract a motion vector reflecting a correct moving subject without being affected by disturbance of an analog signal or noise at the time of A / D conversion, and thereby, it is possible to extract a motion vector without decoding an original image. It is possible to infer the contents of the image.

(2) According to the present invention, in the method and apparatus for extracting moving subject information, it is possible to accurately determine the moving subject region and to accurately separate the moving subject region.

(3) According to the present invention, in the method and apparatus for extracting moving subject information, it is possible to accurately determine and remove a motion vector irrelevant to motion information.

(4) According to the present invention, in the method and apparatus for extracting moving subject information, it is possible to accurately determine camera work, and to accurately determine and remove a motion vector reflecting camera work.

(5) According to the present invention, the contents of a compressed image data sequence can be browsed at high speed and easily.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining a moving subject information extraction method according to an embodiment of the present invention;

FIG. 2 is a block diagram illustrating a schematic configuration of a video reproduction device including a moving subject information extraction device according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating one image in an image sequence captured by a camera performing panning from right to left.

FIG. 4 is a diagram showing a distribution of motion vectors when the original image shown in FIG. 3 is predictively encoded.

FIG. 5 is a flowchart illustrating a processing procedure of a noise vector removing unit 25 according to the present embodiment.

6 is a diagram showing a motion vector distribution after removing a motion vector determined as a noise motion vector from the motion vectors shown in FIG. 4;

FIG. 7 is a diagram for explaining the definition of the direction and angle of the motion vector shown in FIG. 4 or FIG.

FIG. 8 is a flowchart illustrating an example of a processing procedure of a camera work removing unit 27 according to the present embodiment.

FIG. 9 is a flowchart illustrating an example of a processing procedure of a moving subject area separating unit 29 according to the present embodiment.

FIG. 10 is a flowchart illustrating another example of the processing procedure of the noise vector removing unit 25 according to the present embodiment.

FIG. 11 is a flowchart illustrating another example of the processing procedure of the camera work removing unit 27 according to the present embodiment.

FIG. 12 is a flowchart illustrating another example of the processing procedure of the moving subject area separation unit 29 according to the present embodiment.

FIG. 13 is a flowchart illustrating another example of the processing procedure of the moving subject area separation unit 29 according to the present embodiment.

FIG. 14 is a diagram illustrating an example of a distribution of motion vectors in a case where a moving subject exists in an ideal environment, a case where camera work exists, and a case where noise exists.

[Explanation of symbols]

1... Inter-frame predictive coded image data sequence, 10, 20,.
Input MPEG data string, 21... Shot unit classification unit, 2
3: encoded information analyzer, 25: noise vector remover,
27: camera work removing unit, 29: moving subject region separating unit, 31: moving subject information extracting unit, 60: variable length decoder, 61: inverse scanning unit, 62: inverse quantization unit, 63: inverse DC
T section, 64: adder, 65: motion compensation section, 66: reproduced (decoded) image data string, 67: reproduction display section.

Claims (8)

[Claims] 1. A moving subject information extracting method for calculating moving subject information from a motion vector included in video compression encoded data encoded by a compression encoding method using a motion prediction method, comprising: A first step of removing, as noise, a motion vector irrelevant to the moving object from among the motion vectors included in the motion vector; A moving subject information extraction method, comprising: a second step of extracting a subject region; and a third step of extracting moving subject information from the moving subject region extracted in the second step. 2. The moving subject according to claim 1, wherein in the first step, camera work information irrelevant to the moving subject region is extracted in advance, and a motion vector reflecting the camera work is removed. Information extraction method. 3. The method according to claim 1, wherein, in the second step, a set of motion vectors having characteristics similar to those of temporally or spatially adjacent motion vectors is separated and extracted as a moving subject region. The moving subject information extraction method according to claim 2. 4. When separating and extracting a set of motion vectors having characteristics similar to the temporally or spatially adjacent motion vectors as a moving subject area, at least one feature amount of the motion vector is calculated. A feature value is compared between two adjacent motion vectors, and when the comparison result satisfies a predetermined value, it is determined that the two adjacent motion vectors are determined to be similar. 4. The moving subject information extraction method according to claim 3, wherein: 5. A moving subject information extraction apparatus for calculating moving subject information from a motion vector included in video compression encoded data encoded by a compression encoding method using a motion prediction method, comprising: A preprocessing unit that removes, as noise, a motion vector unrelated to the moving subject from the motion vectors included in the motion vector; and a motion corresponding to the moving subject from the motion vector from which the motion vector unrelated to the moving subject has been removed by the preprocessing unit. A moving subject information extraction unit comprising: a moving subject region separating unit that extracts a subject region; and a moving subject information extracting unit that extracts moving subject information from the moving subject region extracted by the moving subject region separating unit. apparatus. 6. The camera according to claim 5, wherein the pre-processing unit includes a camera work removing unit that extracts camera work information irrelevant to the moving subject area in advance and removes a motion vector reflecting the camera work. The described moving subject information extraction device. 7. The moving subject area separating section separates and extracts a set of motion vectors having characteristics similar to temporally or spatially adjacent motion vectors as a moving subject area. A moving subject information extracting device according to claim 6. 8. When separating and extracting a set of motion vectors having characteristics similar to the temporally or spatially adjacent motion vectors as a moving subject region, at least one feature amount of the motion vector is calculated, A feature amount is compared between two adjacent motion vectors, and when the comparison result satisfies a predetermined value, it is determined that the two adjacent motion vectors are similar. The moving subject information extracting apparatus according to claim 7, wherein