JP2013206262A - Method and program for separating two or more subject area - Google Patents

Abstract

In subject separation processing, a plurality of subjects with occlusions are separated using video captured by one camera without the need for information from other cameras.
A method for separating a plurality of subjects in which occlusion occurs in a target video is obtained by dividing each frame image into a foreground region and a background region, tracking each subject region between the frame images, and From the tracking result of the subject area, a foreground area where occlusion occurs in the frame image is detected as an occlusion area, and boundaries between the plurality of subject areas tracked by the subject tracking process in the occlusion area are determined.
[Selection] Figure 1

Description

  The present invention relates to a method and a program for separating a plurality of overlapping subjects in a video. More specifically, in order to generate textures for each subject in the process of generating an image at a virtual viewpoint from an image of an event taking place in a large space, a plurality of time-series subject tracking results are used. It is related with the method and program which isolate | separate the object which overlapped.

  As a method of generating an image at a virtual viewpoint from an image obtained by capturing an event performed in a large space, Patent Document 1 generates a texture from each subject and maps the subject to a plane arranged in a three-dimensional space. Proposed method.

  When generating the texture of the subject, if there is an occlusion where the subjects overlap each other, there is a problem that a plurality of subjects are included in one texture and an inaccurate virtual viewpoint image is generated.

  With respect to the above problem, in Patent Document 1, it is determined whether or not occlusion has occurred in the adjacent viewpoint for the subject. If no occlusion has occurred in the adjacent viewpoint, the background area of the adjacent viewpoint is occluded. By extracting the texture of the subject area by projecting onto the area where the image is generated, and when occlusion occurs from the adjacent viewpoint, the texture that contains multiple subjects is used as it is Has proposed.

  Further, in order to solve the above problem, Non-Patent Document 1 uses the information of images taken from a plurality of other viewpoints to restore rough three-dimensional information, thereby separating a subject existing in the foreground and creating a texture. We have proposed a method of acquiring and using textures from other viewpoints where no occlusion has occurred for the back subject. Specifically, a plane parallel to the image in which occlusion occurs is set near the subject position in the three-dimensional space, and the area of the subject existing in the foreground extracted from another viewpoint is projected onto this plane. The texture area of the subject in front can be acquired.

  Although the technique to be solved is different from the present invention, as a conventional technique that is technically similar, Non-Patent Document 2 discloses a method of separating a subject from a background using a time-series subject tracking result. Proposed.

JP 2011-170487 A

Takayoshi Furuyama, Satoshi Kitahara, Yuichi Ota, "Live distribution and presentation of large-scale space free viewpoint video using virtual reality technology", IEICE technical report, PRMUVol.103, No.96, pp.61 -66, (2003) Lili Ma, Jing Liu, Jinqiao Wang, Jian Cheng, Hanqing Lu, “A improved silhouette tracking approach integrating particle filter with graph cuts”, in Proc. ICASSP2010, pp.1142-1145

  In Patent Document 1 and Non-Patent Document 1, information of images taken from a plurality of viewpoints is used to separate a plurality of subjects that have caused occlusion. Therefore, when occlusion has occurred in the subject from other viewpoints, Patent Document 1 cannot separate a plurality of subjects, and Non-Patent Document 1 has a problem that the separation accuracy of a plurality of subjects decreases. It was.

  In a large-scale space, it is difficult to perform camera calibration and foreground extraction processing with high accuracy. In Patent Literature 1 and Non-Patent Literature 1, by using the results of foreground extraction processing of other viewpoints and the results of camera calibration, a plurality of subjects are separated by projecting to the viewpoint where the occlusion occurs. Therefore, subject separation accuracy is reduced due to camera calibration errors and foreground extraction processing errors. Since the camera calibration error and the foreground extraction process error are accumulated according to the number of cameras, it is difficult to improve accuracy by increasing the number of cameras used. The two problems described above are derived from the fact that Patent Document 1 and Non-Patent Document 1 perform subject separation using information on images taken from a plurality of cameras.

  Accordingly, an object of the present invention is to perform separation of a plurality of subjects that have caused occlusion using video captured by one camera without requiring information of other cameras in subject separation processing. And

  Non-Patent Document 2 proposes a method of separating a subject from the background using time-series subject tracking results, but is not a technique for handling a plurality of subjects, and performs tracking and separation of a single subject. Due to the technology, it is impossible to separate a plurality of subjects that have caused occlusion.

  In order to solve the above problems, the present invention detects an occlusion region where subjects overlap each other based on a time-series tracking result between frames of each subject, and detects each subject included in the occlusion region. Based on the texture information, we devise a method for properly separating each subject. By using the result of tracking between frames, it is possible to separate a plurality of subjects that have caused occlusion from an image captured by one camera without requiring information from other cameras. The specific processing procedure is as follows.

  In order to solve the above-described problem, the subject separation method of the present invention is a method for separating a plurality of subjects in which occlusion occurs in a target video, and includes foreground region extraction processing for dividing each frame image into a foreground region and a background region. A foreground region in which occlusion occurs in the current frame image as an occlusion region based on a step, a subject tracking process step for tracking each subject region between the frame images, and a tracking result of the foreground region and the subject region Detecting, and determining a boundary between a plurality of subject areas tracked by the subject tracking process in the occlusion area.

  The subject tracking process step is preferably performed using a particle filter.

  In the step of detecting as an occlusion area, when the tracked subject is included in a connected foreground area including a plurality of subjects, the area is preferably detected as an occlusion area.

  Further, the step of determining the boundary between the plurality of subject areas includes a sub-step of extracting the subject area for each subject existing in the occlusion area and a region extraction result for each subject to integrate the subject area regions. It is also preferable to include a sub-step for determining the boundary.

  The step of determining boundaries between the plurality of subject areas includes a sub-step of extracting a rectangular area circumscribing the occlusion area, and a seed for area division for each subject included in the occlusion area in the rectangular area. A sub-step for generating, a sub-step for extracting the subject region by binary region division based on the generated seed, and a sub-step for determining a boundary between subject regions by integrating the region extraction results for each subject It is also preferable to include a step.

  Further, the sub-step of generating the seed for dividing the region uses a two-dimensional map representing the subject likeness at each position in the rectangular region as a subject map, and uses the tracking result of the subject to positively add the subject map. A sub-step of adding a value, a sub-step of adding a negative value to the subject map using a tracking result of subjects other than the subject, a sub-step of normalizing the subject map, and the normalized subject map It is also preferable to include a sub-step of generating a seed for region division using information on the background region obtained by the foreground extraction process.

  The region dividing method is preferably based on a graph cut algorithm.

  In order to solve the above problems, a program according to the present invention provides a computer for separating a plurality of subjects in which occlusion occurs in a target video, means for foreground region extraction processing for dividing each frame image into a foreground region and a background region, Means for subject tracking processing for tracking each subject area between the frame images; and means for detecting a foreground area where occlusion occurs in the frame image as an occlusion area from the tracking results of the foreground area and the subject area. The occlusion area functions as means for determining boundaries between a plurality of object areas tracked by the object tracking process, and separates the plurality of objects that have caused occlusion.

  In the present invention, since the subject can be separated using one camera image without using a plurality of camera images, the subject separation accuracy does not deteriorate due to the occlusion generated by another camera.

  Furthermore, in the present invention, since the subject can be separated using one camera image without using a plurality of camera images, the object separation accuracy due to the camera calibration error and the foreground extraction processing error in other cameras can be improved. There is no reduction.

  Further, in the conventional method, since subject separation is performed using a plurality of camera images, there are cases where the sparse camera arrangement cannot be applied depending on the geometric relationship between the cameras. In the present invention, since the subject can be separated using one camera image, it can be applied to a sparse camera arrangement regardless of the geometric relationship between the cameras.

2 shows a flowchart illustrating a method for separating a plurality of subjects according to the present invention. The object tracking process by a particle filter is shown. The example of an image of an occlusion area | region is shown. The graph cut seed generation processing using the particle information of the particle filter is shown. The example of the production | generation process of the seed for graph cuts is shown. An example of a region division result before determining a boundary between subject regions is shown. An example of the result of determining the boundary between subject areas is shown. The example of the change of the virtual viewpoint image by the presence or absence of application of this invention is shown.

  The best mode for carrying out the present invention will be described in detail below with reference to the drawings. FIG. 1 is a flowchart illustrating a method for separating a plurality of subjects according to the present invention. Hereinafter, description will be given based on this flowchart.

  The present invention uses the result of subject tracking, does not use information from multiple cameras, and is subject to the occlusion of other cameras by performing subject separation on an image taken from one camera. Instead, the areas of a plurality of subjects are separated with high accuracy in the occlusion area. According to the present invention, an image captured for a certain period of time by a single camera is input, and a plurality of subjects having occlusion are separated from the current frame image.

  Step 1: Each frame image is divided into a foreground area and a background area. The foreground region is a region whose position and shape change with time, and corresponds to a subject region such as a person when the target space is a soccer field. The background area is an area other than the foreground area. A background image in which the foreground area is not captured is acquired in advance, and a mask image is generated by dividing the foreground area and the background area by performing a background difference from the current frame image and the background image.

  Step 2: Subject tracking processing is performed by a particle filter for each subject in the current frame. Subject tracking by the particle filter is performed by time-series processing between frames. FIG. 2 is a flowchart showing time-series subject tracking processing by the particle filter. The subject tracking process will be described below based on this flowchart.

  Step 21: The particle filter circumscribing frame for each subject in the initial frame is set as the circumscribing region of the mask image. Input an initial frame in which no occlusion occurs in the subject. It is also possible to visually check the mask image in the initial frame in which occlusion occurs, and manually set a circumscribed area for each subject for the subject in which occlusion has occurred.

Step 22: A state quantity for each particle existing in each subject particle filter at time t is defined as follows.
Here, x (t) and y (t) are the two-dimensional coordinates of the particle, and Δx (t) and Δy (t) respectively correspond to the horizontal speed and the vertical speed in the subject image. To do. The state transition function is set as follows using Gaussian noise ω (t), assuming a constant velocity linear motion between frames.

Step 23: The process of calculating the likelihood of each particle is divided into two stages. In the first stage, it is determined whether the position of each particle is the foreground region or the background region in the mask image, and the particles determined to be the background region have a likelihood of zero. The particle determined to be the foreground region determines the likelihood based on the prior information given for each subject. When the subject is a soccer player, it is determined whether or not the pixel value I at the particle position satisfies the following equation using the threshold value th _ref based on the uniform color information I _ref. If not, the likelihood is 1.
| I-I _ref |> th _ref

  Step 24: Particles with a likelihood of 0 are extinguished and rearranged in the vicinity of a particle with a likelihood of 1.

  Step 25: The circumscribed area of the rearranged particle group is calculated for each subject, and the circumscribed frame of the subject particle filter is updated. Steps 22 to 25 are repeated until the final frame.

  Step 3: A foreground area where occlusion has occurred is detected using the result of dividing the foreground area and the background area and the result of the subject tracking process using the particle filter in the current frame. For all the subjects tracked by the subject tracking process, the connected foreground region including a plurality of subjects is determined based on the position of the circumscribed frame of the particle filter to determine in which connected foreground region the subject is included. Detect as an occlusion area.

  Step 4: For each occlusion area detected in step 3, a rectangular area circumscribing the occlusion area is extracted from the current frame image. FIG. 3 is an image example of the occlusion area.

  FIG. 3a is an example of a rectangular area circumscribing the occlusion area. FIG. 3b is a mask image in the rectangular area. FIG. 3c is a foreground region image in which the mask image (FIG. 3b) is superimposed on the original image (FIG. 3a) of the rectangular region.

  Steps 5, 6, and 7 after step 4 are performed independently for each occlusion area for the rectangular area.

  Step 5: Generate a seed for graph cut using the particle information of the particle filter in the subject tracking process. In steps 5 and 6, attention is paid to one of the plurality of subjects included in the occlusion area, and both the steps are performed independently for each subject in the occlusion area. Step 5 is a preparation step for performing area division by graph cut in Step 6. In order to perform region division by graph cut, it is necessary to provide pixels with labels called seeds. In this embodiment, all pixels are given a label of any one of “target subject”, “target subject candidate”, “background candidate”, and “background” as a seed. In steps 5 and 6, the target subject and the background are divided using all pixels other than the target subject in the rectangle as the background. In step 5, seeds based on the four labels are generated so as to be prior information for dividing the region by graph cut in step 6.

  FIG. 4 is a flowchart showing a process for generating a graph cut seed using particle information of a particle filter. The graph cut seed generation process will be described below based on this flowchart.

  Step 31: Generate a subject map based on the particle information of the particle filter. The subject map is a two-dimensional map representing the likelihood of the subject of interest at each position in the rectangular area. Specifically, a subject map is generated by superimposing a two-dimensional Gaussian distribution around each sample position, with particles in the subject of interest as positive samples and particles in subjects other than the subject of interest as negative samples.

Particle number n _m of the target _object, the number of particles of the object other than the target object and n _e. Further, for each i-th particle of the subject of interest and the other subjects, assuming that the positions in the rectangular area are p _mi and p _ei and the likelihoods are c _mi and c _ei , the value v at the position p on the subject map (P) is expressed by the following equation using a two-dimensional Gaussian function G (p | p ₀ ) with the position p ₀ as the center.

Step 32: Normalize the subject map. Assuming that the maximum value and the minimum value in the map are v _max and v _min , respectively, the value v _norm (p) obtained by normalizing the value v (p) at the position p on the map is expressed by the following equation.
The range of the normalized value v _norm (p) is −1 ≦ v _norm (p) ≦ 1.

Step 33: Generate a seed based on the normalized subject map and the result of the foreground extraction process. The value on the normalized subject map at the position p is v _norm (p), the threshold is th (where 0 <th <1), and the result r (p) of the foreground extraction process in step 1 is the position p If r (p) = 0 in the foreground extraction process and r (p) = 1 in the foreground, the seed label l (p) at the position p is expressed by the following equation.

  FIG. 5 shows an example of the process of generating the graph cut seed. FIG. 5A is an example of particle filter particles of a subject of interest and other subjects. White dots are particles of the subject of interest, and dark gray points are particles of subjects other than the subject of interest. FIG. 5b is an example of an image in which a mask image (FIG. 3b) is superimposed on a normalized subject map. The larger the value on the map, the higher the brightness. FIG. 5c is an example of a seed for graph cut. A black region is provided with a background label, a dark gray region is provided with a background candidate label, a light gray region is provided with a subject subject candidate label, and a white region is provided with a subject subject label.

  Step 6: Extract the subject area of interest by dividing the rectangular area into the subject area of interest and other areas by the binary graph cut area division process based on the seed generated in step 5.

  Step 7: The subject areas extracted for each subject are integrated to determine a boundary between the subject areas. Through steps 5 and 6, the subject area has already been acquired for each of the plurality of subjects included in the occlusion area. However, since Steps 5 and 6 are processed independently for each subject included in the occlusion region, there may be regions included in a plurality of subject regions, and it is necessary to determine the boundary between the subject regions. There is. FIG. 6 is an example of the region division result before the processing of steps 5 and 6 is performed on all subjects in the occlusion region and the boundary between the subject regions is determined. The background region is black, one subject region is white, the other subject region is dark gray, and the regions included in both subject regions are light gray.

  The integration process for determining the boundary between the subject areas can be performed, for example, by a process for determining the subject areas in order from the subject whose lowermost part of the subject area is higher. FIG. 7 shows an example of the result of determining the boundary between subject areas in the occlusion area by the integration process.

  FIG. 8 is an example of a change in the virtual viewpoint image depending on whether or not the present invention is applied. FIG. 8a is a partial image near the goal post at a certain viewpoint. FIG. 8b shows a region where occlusion occurs in FIG. 8a. FIG. 8c is an example of a virtual viewpoint image using a texture that does not separate a plurality of subjects in the occlusion of FIG. 8b. FIG. 8d is an example of a virtual viewpoint image generated using an image obtained by separating a plurality of subjects as a texture according to the present invention. By separating a plurality of subjects according to the present invention, it can be seen that the position, shadow, etc. of the subject on the back side are synthesized with higher accuracy.

  Moreover, all the embodiments described above are illustrative of the present invention and are not intended to limit the present invention, and the present invention can be implemented in other various modifications and changes. Therefore, the scope of the present invention is defined only by the claims and their equivalents.

Claims (8)

A method for separating a plurality of subjects having occlusion in a target video,
Foreground region extraction processing steps for dividing each frame image into a foreground region and a background region;
Subject tracking processing steps for tracking each subject region between the frame images;
From the tracking results of the foreground area and the subject area, detecting a foreground area where occlusion occurs in the current frame image as an occlusion area;
Determining boundaries between a plurality of subject areas tracked by the subject tracking process in the occlusion area;
A method for separating a plurality of subjects in which occlusion has occurred.   The method for separating a plurality of subjects with occlusions according to claim 1, wherein the subject tracking processing step is performed using a particle filter.   3. The step of detecting as an occlusion area detects the area as an occlusion area when the tracked object is included in a connected foreground area including a plurality of objects. A method of separating multiple subjects that have caused occlusion. Determining a boundary between the plurality of subject areas,
A sub-step of extracting a region of the subject for each subject existing in the occlusion region;
A sub-step for determining a boundary between subject areas by integrating region extraction results for each subject;
4. The method for separating a plurality of subjects in which occlusion has occurred according to claim 1. Determining a boundary between the plurality of subject areas,
A sub-step of extracting a rectangular area circumscribing the occlusion area;
A sub-step of generating a seed for region division for each subject included in the occlusion region in the rectangular region;
A sub-step of extracting the subject region by binary region division based on the generated seed;
A sub-step for determining a boundary between subject areas by integrating region extraction results for each subject;
The method for separating a plurality of subjects that have caused occlusion according to claim 1, The sub-step of generating the seed for region division is as follows:
A sub-step of adding a positive value to the subject map using a tracking result of the subject as a two-dimensional map representing the subjectness at each position in the rectangular region as a subject map;
A sub-step of adding a negative value to the subject map using a tracking result of subjects other than the subject;
A sub-step of normalizing the subject map;
Generating a seed for dividing the region using the normalized subject map and information on the background region obtained by the foreground extraction process;
6. The method for separating a plurality of subjects having caused occlusion according to claim 5.   7. The method for separating a plurality of subjects with occlusion according to claim 5, wherein the region dividing method is based on a graph cut algorithm. A computer for separating multiple subjects that have occluded in the target video,
Means for foreground region extraction processing for dividing each frame image into a foreground region and a background region;
Means for subject tracking processing for tracking each subject region between the frame images;
Means for detecting, as an occlusion area, a foreground area where occlusion occurs in the frame image from the tracking results of the foreground area and the subject area;
Means for determining boundaries between a plurality of subject areas tracked by the subject tracking process in the occlusion area;
A program characterized in that it separates a plurality of subjects that caused occlusion.