JP2001014474A - People extraction method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To transmit an image of a person, it is necessary to clear objects that the user does not want to see the background. When a person is extracted by the background subtraction method, background noise is output or the extracted person becomes wormy. To solve these issues. SOLUTION: A person is extracted using a background difference method, and an image is binarized. Then, background noise is removed by applying a majority filter. Next, the binary image subjected to the majority filter is divided into blocks, and the worms of the person are removed. Further, edge detection is performed using the blocked binary image. Then, the color image created by the majority filter processing is replaced with the block which is an edge. In addition, a non-edge block is replaced with a color image created by the blocking process to obtain a more beautiful human extracted image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a person extracting method used for a video conference system or a video telephone, and more particularly to a person extracting method using a background subtraction method from a photographed screen.

[0002]

2. Description of the Related Art In a conventional video conference system or video telephone, a method of directly encoding a photographed image and transmitting the image, or a method of extracting a person based on a background difference or an inter-frame difference and transmitting only a person image. Is used.

[0003]

However, in the case of the technique of transmitting the photographed image as it is, if the background other than the person is not desired to be seen earlier, it is necessary to clear unnecessary objects included in the background. Further, even when a person is extracted using a background difference or an inter-frame difference, the background may be output as noise, or the extracted image of the person may be wormy. As described above, it is impossible with the conventional method to extract only a person clearly.

The present invention has been made in view of such a conventional problem. When a person is extracted using the background subtraction method, background noise is output or the extracted person becomes wormy. It is an object of the present invention to realize a person extraction method that can prevent such a situation.

[0005]

According to the first aspect of the present invention, two images, a background image that has been shot and stored in advance and a mixed image in which the background and a person have been shot, are obtained using a background difference method. The first part obtains a background difference processed image by using the same part as a background image and the different part as a person image.
And setting the pixels of the human image to a first value and the pixels of the background image to a second value in order to binarize the background difference processing image obtained in the first step. A second step of generating a set background difference binary image, and, when a specific pixel of the background difference binary image obtained in the second step is set as a target pixel, a plurality of pixels surrounding the target pixel , A total value of the first value and a total value of the second value are calculated, and the value of the pixel of interest is set to the first value or the second value based on the counting result. By performing the majority filtering process, a third step of obtaining a majority filtering binary image, and multiplying the pixels of the mixed image by the majority filtering binary image obtained in the third step , Majority filtering A fourth step of obtaining a color image,
A fifth step of dividing the one-frame majority filter-processed binary image obtained in the third step into blocks composed of a plurality of pixels to obtain a blocked binary image, and In the block processing binary image obtained in the step, edge detection is performed in block units to obtain an edge detection processing binary image indicating an edge or non-edge, and the sixth step is obtained in the sixth step. In the edge detection processing binary image, the majority decision filter processing color image obtained in the fourth step is replaced with a block determined as an edge, and a block composed of the first value determined as a non-edge is replaced. The seventh step of obtaining the output image by replacing the mixed image and replacing the desired background color with the block consisting of the second value determined as a non-edge. When, it is characterized in that it has a.

According to a second aspect of the present invention, in the person extracting method according to the first aspect, the third step includes performing a majority process of a first value and a second value in pixels around the pixel of interest. In this case, the value of the pixel of interest is a majority filtering process that converts the value of the pixel of interest into a larger value as a result of the majority process.

According to a third aspect of the present invention, in the person extracting method according to the first aspect, the fifth step is performed when at least one of the pixels included in the predetermined block has the first value. This is a process of converting all the pixels of the predetermined block to the first value to obtain a blocked binary image.

According to the invention of claim 4 of the present application, in the person extraction method of claim 1, the sixth step is a step of calculating the first value from the second value in the vertical direction or the horizontal direction of each block.
Or a pixel that changes from the first value to the second value is determined to be an edge block, and the other is determined to be a non-edge block to obtain an edge detection binary image. It is characterized by the following.

According to a fifth aspect of the present invention, in the person extraction method of the first aspect, the sixth step is a step of calculating the first value from the second value in the vertical and horizontal directions of each block.
Or a pixel that changes from the first value to the second value is determined to be an edge block, and the other is determined to be a non-edge block to obtain an edge detection binary image. It is characterized by the following.

In the invention according to claim 6 of the present application, in the person extracting method according to claim 1, the fourth step is performed by using the first filtering processing in the majority filtering binary image obtained in the third step. This is a process of obtaining a majority-filtered color image by replacing only pixels having a value with pixels of the mixed image.

As described above, the image from which the person has been extracted by using the background subtraction method is binarized and subjected to a majority filter to remove background noise, and in the next step, the binary image subjected to the majority filter is removed. Blocking and processing are performed to remove human worming, and in the next step, edge detection is performed using the obtained blocked binary image. , And replacing the blocks other than the edges with the color image created by performing the blocking process, it is possible to create a more beautiful human extracted image.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A method for extracting a person using a background subtraction method according to the present invention will be described below with reference to flowcharts of FIGS. The signal processing in steps S101 to S313 in FIG. 1 shows each process of the person extraction method using the background difference method.

FIG. 1 and FIG. 2 are flow charts relating to the photographing of a background image and the capturing of a mixed image including a person, and the background difference and majority filter processing for these image data. First, in step S101, when a person is extracted, a background image is photographed to specify a portion to be processed as a background, and data of the background image is obtained.
Note that the background image to be acquired may be captured over a plurality of frames, but in this example, it is assumed that only one frame is captured.

When a communication such as a videophone call or a video conference starts, the process proceeds to step S102, where a communication image is captured, and data of a mixed image including a background and an extracted person is obtained. In the next step S103, the background image and the mixed image are compared. The background subtraction method is performed between these two images, but when capturing a background image and a mixed image, it is necessary to acquire the same positional information. The background difference is
The processing is performed on all pixels or all blocks in the image. Here, the processing is performed on each pixel, and the specific method is as follows.

In step S104, one of the differences in luminance and color difference between pixels in the background image and the mixed image is checked. If the difference value is equal to or higher than the arbitrary level, the process branches to step S105, in which the pixel is regarded as a part of a person, and one value is input as data of the background difference binary image of the pixel. If it is determined in step S104 that both the difference values of the luminance and the color difference are less than the arbitrary level, the process branches to step S106. In this case, the pixel is regarded as the background, and 0 value is entered as the data of the background difference binary image of the pixel. In this example, the comparison difference is performed on both the luminance and color difference data. However, the comparison may be performed on the RGB data or the like as only the luminance, only the color difference, or pixel data in a format different from the luminance and the color difference.

In step S107, it is determined whether or not the background subtraction processing has been completed for one frame. If the processing has been completed for one frame, the process proceeds to the next majority filter processing. If one frame has not been completed in step S107, the process returns to step S104 to compare the difference of the next pixel. The order of the pixel comparison processing does not matter. An image processed using the background subtraction method may not be able to cut the background cleanly, may include background noise, or may have an extracted person in a worm-like state. For this reason, the image is processed using the method described below. For this reason, subsequently, majority filtering processing is performed on the background difference binary image created in the processing of steps S103 to S107 in steps S108 to S112 of FIG.

First, in step S108, the pixel value (0 value or 1 value) in an arbitrary range around one arbitrary pixel to be processed is counted for the background difference binary image created by the above processing. I do. Next step S10
In step 9, it is determined whether or not the 0 value occupies a majority of the number of pixels in an arbitrary range in order to perform a majority decision. As a result of the majority decision, when the 0 value occupies the majority of the number of pixels in the arbitrary range, the process branches to step S110, and the pixel value of the background difference binary image is set to 0 as the data of the majority filtered binary image of the pixel to be processed. Replace with a value. In step S109,
Conversely, if the 0 value does not occupy a majority of the number of pixels in the arbitrary range, that is, if the 1 value is a majority, step S1
The process branches to 11 and replaces the pixel value of the background difference binary image with one value as data of the majority filter-processed binary image of the pixel to be processed.

FIGS. 6 and 7 show examples of the majority filter processing. 6 and 7, x, y is an address (coordinate value) indicating a pixel position, and the pixel (x, y) is a pixel to be processed (pixel of interest) on which majority filtering is performed. The pixels that take the majority decision are, for example, 9 pixels including the pixel to be processed [(x-1, y-1), (x, y-1), (x + 1, y-1), (x-1 , y),
(X, y), (x + 1, y), (x-1, y + 1), (x, y + 1), (x + 1, y +
1)] is used. FIG. 6A shows data before the majority filter processing, and the 0-valued pixel has six pixels [(x-1, y-1), (x,
y-1), (x + 1, y-1), (x-1, y), (x + 1, y), (x-1, y + 1)]
Existing. In addition, one-valued pixel has three pixels [(x, y), (x,
y + 1), (x + 1, y + 1)]. After the majority filter processing, the number of 0-valued pixels is large, so that the pixel to be processed (x, y) is 0
The values are converted into values as shown in FIG.

FIG. 7A shows another example of the data before the majority filter processing. Here, there are two 0-valued pixels [(x + 1, y-1), (x + 1, y)], and one-valued pixels are seven pixels [(x-1, y-1), ( x, y-1), (x-1, y), (x, y), (x-1,
y + 1), (x, y + 1), (x + 1, y + 1)]. Since the number of one-valued pixels is large, the pixel to be processed (x, y) remains at the one-valued value after the majority filter processing, as in FIG.
become that way.

In step S112 of FIG. 2, it is checked whether or not the majority filtering process has been completed for one frame, and if one frame has been completed, the process proceeds to the next image forming process. If one frame has not been completed in step S112, the process returns to step S108, and the majority pixel filtering process for the next pixel is performed.

The flowcharts shown in FIGS. 2 and 3 form a majority filtering filtered color image using the majority filtering binary image and the mixed image, block the majority filtering binary image, and further execute edge processing. This is a diagram illustrating the process of detecting. In step S201, the majority filter-processed binary image data created in steps S108 to S112 is multiplied by the mixed image to create a majority filter-processed color image. That is, if the binary image data of the majority filter processing is a single value, the mixed image is output as it is, and if the binary image data of the majority filter processing is a zero value, data of luminance 0, that is, black data is output. The majority filter-processed color image obtained here is used for an output image creation process described later.

Next, a description will be given of a process of blocking image data for removing worms in a person portion. In step S202, first, an image of one frame is divided into arbitrary blocks, and a binary image subjected to majority filtering on a block basis is obtained. In step S203 of FIG. 3, it is checked whether or not one value exists even in one pixel in the pixel data in the divided block. If even one pixel has one value, it is determined that the block includes a pixel regarded as person data, and the flow branches to step S204. In step S204, one value is stored in the block-processed binary image, and the block is regarded as a block including a person.

If it is determined in step S203 that none of the pixels in the block has a single value, that is, if the block does not include a pixel regarded as human data, the process branches to step S205, where the block-processed binary image is processed. To 0
The value is stored, and it is assumed that only the background data is contained in the block. In the next step S206, the above step S
It is determined whether or not the block processing of 203 to S205 has been completed for one frame. If the processing has not been completed for one frame, the process returns to step S203 to perform the block processing of the next block. In step S206, 1
When the processing has been completed for the number of frames, the processing proceeds to the next signal processing. FIG.
0 is an example of an image obtained by performing the blocking process in the above process.

With respect to the data of the block-processed binary image created in the processes of steps S202 to S206,
A method of detecting edges in the X direction (horizontal direction) and the Y direction (vertical direction) of an image will be described with reference to FIGS. Here, X and Y are addresses (coordinate values) indicating block positions.

First, proceeding to step S207 in FIG.
Edge detection in the X direction is performed on the data of the block processing binary image. Therefore, the values of the block-processed binary images adjacent in the X direction are compared. As shown in FIG. 8, an arbitrary block (X, Y) has one value, and the value of each adjacent block (X-1, Y) is compared with the value of (X + 1, Y). , Y) ≠ (X + 1, Y)
In the case of, the process branches to step S209. For example, FIG.
As shown in (a), (X, Y) = 1 value, (X-1, Y) = 0 value,
When (X + 1, Y) = 1, or as shown in FIG. 8B, (X, Y) = 1, (X-1, Y) = 1, (X + 1, Y) If the value is 0, the block (X, Y) is regarded as an edge block, and a single value is stored as the data of the edge-detected binary image in the X direction in step S209.
If this condition is not met, the process branches from step S208 to step S210, where the block is regarded as a non-edge, and a 0 value is stored as the data of the edge detection binary image.

In step S211, step S207
It is checked whether or not the X-direction edge detection processing in steps S210 to S210 has been completed for one frame. If the processing has not been completed for one frame, the process returns to step S207 to return to the processing of step S207.
Perform direction edge detection processing. In step S211, when the processing for one frame is completed, the process proceeds to the next Y-direction edge detection processing. As described above, when the edge detection in the X direction is performed on the block-processed binary image shown in FIG. 10, an image as shown in FIG. 11 is obtained.

Subsequently, the process proceeds to the process shown in FIG. 4, and first, edge detection is performed in the Y direction on the data in the block array. FIG.
The processing in steps S301 to S313 in FIG. 5 shows the edge detection processing in the Y direction and the output image creation processing. In step S301, the values of the block-processed binary images adjacent in the Y direction are compared. Therefore, the process proceeds to step S302, and the block of interest (X, Y) has one value,
It is checked whether (X, Y-1) ≠ (X, Y + 1) is satisfied in the adjacent block. If this condition is satisfied, the flow branches to step S303. If this condition is not satisfied, the flow branches to step S304.

In the example shown in FIG. 9, since the block of interest (X, Y) has one value, this block is compared with the values of the blocks (X, Y-1) and (X, Y + 1) adjacent thereto. Then (X, Y-1) ≠ (X, Y + 1)
In other words, as shown in FIG. 9A, (X, Y) = 1 value,
(X, Y-1) = 0 value, (X, Y + 1) = 1 value, or FIG.
As shown in (b), (X, Y) = 1 value, (X, Y-1) = 1 value,
If (X, Y + 1) = 0, the flow branches to step S303, where the block (X, Y) is regarded as an edge block, and is used as the data of the edge-detected binary image in the Y direction that has been blocked. 1 value is stored. If this condition is not satisfied, the flow branches to step S304, the block is regarded as a non-edge, and a 0 value is stored as data of the edge detection binary image.

In the next step S305, step S3
It is checked whether or not the Y-direction edge detection processing of 01 to S304 has been completed for one frame, and if it has not been completed for one frame, the process returns to step S301 to return to the processing of Y of the next block.
Perform direction edge detection processing. If one frame is completed in step S305, the process proceeds to step S306. When edge detection is performed in the Y direction on the block-processed binary image as shown in FIG. 10, an image as shown in FIG. 12 is obtained.

In step S306, a logical OR (OR) is performed on the two edge detection binary images detected in the X direction and the Y direction to create an entire edge detection binary image. When this processing is performed, an edge detection processing binary image in which edges are continuous as shown in FIG. 13 is obtained.

After the edge detection processing is completed, the output image color data is determined based on the values of the edge detection binary image and the blocking binary image. Therefore, step S3 in FIG.
At 08, it is checked whether the target block is a non-edge and this block has a value of 0. When the edge detection binary image has a 0 value as shown in the hatched block portion in FIG. 13 and the block processing binary image has a 0 value as shown in the hatched block portion in FIG. If the data is background data, the process branches to step S307, and a 0 value (black) is stored in the block as output color image data. If the condition in step S308 is not satisfied, the flow branches to step S310.

In step S310, it is checked whether or not the target block is a non-edge and whether or not this block has a value of one. In the case where the edge detection processing binary image has a 0 value as shown in a hatched block portion in FIG. 13 and the blocking processing binary image has a 1 value as shown in a white block portion in FIG. 10, that is, non-edge and If the data is person data, the flow branches to step S309 to store the value of the mixed image data as output color image data. If the condition in step S310 is not satisfied, the flow branches to step S311.

In step S311, it is checked whether the target block is an edge and this block has a value of one.
When the binary image of the edge detection processing is one-valued as shown in the white block part of FIG. 13, and the binary image of the blocking processing is one-valued as shown in the white block part of FIG. If it is the boundary portion of step S312,
And stores the majority filtered color image obtained in step S201 as data of the output image.

FIG. 14 shows an example of the output image obtained in the last step S313. In FIG. 14, the shaded block stores black data, the horizontal block stores mixed image data, and the white block stores majority filter processed color images. Although the output image is displayed in blocks in FIG. 14, the actual output image data is returned from the block unit to the pixel unit. Further, in the present embodiment, the background portion other than the person is replaced with black data, but may be replaced with another single color data, arbitrary image or graphic data, or the like.

[0035]

As described above, according to the present invention, it is possible to extract only a person from a photographed image, and there is no problem that noise is added to the background or the person becomes wormy.
An effect is obtained that it is possible to extract only a person clearly.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating signal processing (part 1) of a person extraction method according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating signal processing (part 2) of the person extraction method according to the embodiment of the present invention.

FIG. 3 is a flowchart illustrating signal processing (part 3) of the person extraction method according to the embodiment of the present invention.

FIG. 4 is a flowchart illustrating signal processing (part 4) of the person extraction method according to the embodiment of the present invention.

FIG. 5 is a flowchart showing signal processing (part 5) of the person extraction method according to the embodiment of the present invention.

FIG. 6 is an operation explanatory diagram showing an example (part 1) of majority filter processing in the present embodiment.

FIG. 7 is an operation explanatory diagram showing an example (part 2) of majority filter processing in the present embodiment.

FIG. 8 is an operation explanatory diagram showing an edge detection method in the X direction (horizontal direction) in the present embodiment.

FIG. 9 is an operation explanatory diagram showing a Y-direction (vertical direction) edge detection method according to the present embodiment.

FIG. 10 is an example of a block-processed binary image obtained by performing a block process in the present embodiment.

FIG. 11 is an example of an image obtained by performing edge detection in the X direction (horizontal direction) on the block-processed binary image according to the present embodiment.

FIG. 12 is an example of an image obtained by performing edge detection in the Y direction (vertical direction) on a block-processed binary image according to the present embodiment.

FIG. 13 is an example of a binary image subjected to a final edge detection process in the present embodiment.

FIG. 14 is an example of an output image obtained by the person extraction method according to the present embodiment.

[Explanation of symbols]

 x, y pixel address X, Y block address

 ────────────────────────────────────────────────── ─── Continued on the front page F term (reference) 5C064 AA01 AA02 AB04 AC02 AC08 AC12 AD08 AD14 5L096 AA02 AA06 BA08 DA04 EA43 FA06 FA19 GA08 GA19 GA28 GA55 9A001 BB06 HH23

Claims (6)

[Claims] 1. A background image captured and stored in advance, and two images of a mixed image in which the background and a person are imaged are compared by using a background subtraction method, and an equal portion is set as a background image. A first step of obtaining a background difference processing image using the portion that is present as a person image, and setting the pixels of the person image to a first value in order to binarize the background difference processing image obtained in the first step. A second step of generating a set background difference binary image by setting the pixels of the background image to a second value, and a specific pixel of the background difference binary image obtained in the second step. When the pixel of interest is used, the sum of the first values and the second value for a plurality of pixels surrounding the pixel of interest
Are calculated respectively, and based on the counting result,
A third step of performing majority filtering by setting the value of the pixel of interest to the first value or the second value to obtain a majority filtered binary image; A fourth step of obtaining a majority-filtered color image by multiplying the majority-filtered binary image by the pixels of the mixed image, and a majority-filter processing of one frame obtained in the third step. A fifth step of dividing the binary image into blocks composed of a plurality of pixels to obtain a block processing binary image, and in the block processing binary image obtained in the fifth step, A sixth step of performing edge detection to obtain an edge detection processing binary image indicating an edge or a non-edge; and an edge detection processing binary obtained in the sixth step. In the image, a block determined to be an edge includes the fourth
Replaces the majority-filtered color image obtained in step b) and replaces the mixed image with a block consisting of the first value determined as non-edge, and replaces the mixed image with the second value determined as non-edge. The block includes a seventh step of obtaining an output image by replacing a desired background color;
A person extraction method characterized by having: 2. The method according to claim 3, wherein the third step performs a majority process between a first value and a second value in pixels around the pixel of interest, and determines that the value of the pixel of interest is the larger value as a result of the majority process. 2. The method according to claim 1, wherein the filtering is a majority filtering process. 3. In the fifth step, if at least one of the pixels included in the predetermined block has a pixel of the first value, all the pixels of the predetermined block are converted into the first value. 2. The person extracting method according to claim 1, wherein the process is a process for obtaining a binary image. 4. The method according to claim 1, wherein the sixth step is to perform the second step in a vertical direction or a horizontal direction of each block.
When there is a pixel whose value changes from the first value to the first value or from the first value to the second value, it is determined that the pixel is an edge block, and the others are determined to be non-edge blocks. 2. The method according to claim 1, wherein the process is an obtaining process. 5. The method according to claim 6, wherein the sixth step is to perform the second step in a vertical direction and a horizontal direction of each block.
When there is a pixel whose value changes from the first value to the first value or from the first value to the second value, it is determined that the pixel is an edge block, and the others are determined to be non-edge blocks. 2. The method according to claim 1, wherein the process is an obtaining process. 6. In the fourth step, in the majority filtered binary image obtained in the third step, only pixels having the first value are replaced by pixels of the mixed image. 2. A method for extracting a person according to claim 1, wherein the process is a process of obtaining a majority-filtered color image.