JPH08153201A - Image area extraction method - Google Patents

Abstract

(57) [Summary] [Object] The present invention significantly improves the accuracy of chroma key processing in an image region extraction method. A color distribution obtained by arranging color information of pixel data for each predetermined unit in an image on a color space in which color information is arranged three-dimensionally is approximated by an arbitrary number of three-dimensional ellipsoidal surfaces, Divide the color space using the body surface. As a result, the accuracy of chroma key processing can be significantly improved.

Description

Detailed Description of the Invention

[0001]

[Table of Contents] The present invention will be described in the following order: Industrial field of prior art Problems to be solved by the invention Means for solving the problem Action Example (1) Principle of the present invention (FIGS. 1 to 6) ) (2) Application to chroma key processing (FIGS. 7 to 15)

[0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image area extracting method, and can be applied to, for example, generating a key signal (mask image) necessary for performing an image synthesizing process in a work related to video production.

[0003]

2. Description of the Related Art Conventionally, an image area dividing method for classifying each pixel by space division on a color space in which color information is arranged in a three-dimensional manner is applied to chroma key, which is one of image combining techniques. It is a technology. In general, this chroma key processing is a method that focuses on color difference signals, and from the viewpoint of color space, it is equivalent to dividing a color distribution projected on a plane perpendicular to the luminance axis into a fan shape on that plane.

The advantage of using this chroma key is that, firstly, since the color distribution of an object originally has the property of spreading in the luminance direction, the shape of the color distribution can be made simple and compact by projecting in the luminance direction. It is possible. Secondly, with a simple divided shape such as a fan shape, when a color distribution is given, a shape having a size and a position sufficient to surround it can be automatically calculated.

Further, the chroma key processing is used for an image photographed with a uniform background for the purpose of composition in advance, and it is classified into pixels of a uniform background which have been previously recognized and other pixels. Therefore, it is not necessary to pursue the diversity and precision of the divided shape of the color space for the purpose of use when it is possible to assume the uniformity of the background.

Also in the chroma key processing, when the reflection of the background color and the variation of the background color are problems, it is necessary to consider the difference in the luminance direction which cannot be dealt with by the color difference signal alone. For this reason, a method has been devised to perform division based on the shape of a polyhedron in the RGB space (“Chromakey software using polyhedron slices”, Yasushi Mishima, NICOGR
With this method, the more the number of vertices of a polyhedron, the greater the variety of color distribution shapes. Here, the RGB space refers to a space in which colors are represented by the intensities of three colors of red (R), green (G), and blue (B).

[0007]

By the way, in order to apply the chroma key to various images, that is, images of arbitrary backgrounds, it is necessary to use a space division method capable of coping with various combinations of background colors and object colors. However, in the conventional chroma-key space division method, since the color space is divided into a shape that does not match the three-dimensional shape of the color distribution, the luminance direction is ignored, and objects of similar colors with different luminance are separated from each other. There was a problem that I could not do it. Such a color combination cannot be ignored because it appears in an image of an arbitrary background as overlapping of similar color objects, reflection, shadow, and the like.

Further, as described above, when the dividing method based on the polyhedral shape in the RGB space is used, it is possible to separate color distributions having similar colors but different luminances, but the number of vertices of the polyhedron is large and the polyhedral shape is determined. Since it is difficult to uniquely determine the vertex coordinates, which is a parameter, from the color distribution, there is a problem that determination of the polyhedron shape becomes complicated. Further, there is a problem that it is necessary to determine whether these planes are inside or outside the polyhedron for a large number of planes, and this discrimination process is complicated.

The present invention has been made in consideration of the above points, and it is an object of the present invention to propose an image area extraction method capable of improving the accuracy of chroma key processing.

[0010]

In order to solve such a problem, according to the present invention, in an image area extracting method for extracting a desired area from an image, an image is extracted in a color space in which color information is three-dimensionally arranged. The color distribution obtained by arranging the color information of the pixel data for each predetermined unit is approximated by an arbitrary number of three-dimensional ellipsoidal surfaces, and the desired space is extracted by dividing the color space using the ellipsoidal surfaces.

[0011]

The color distribution obtained by arranging the color information of the pixel data for each predetermined unit in the image on the color space in which the color information is arranged three-dimensionally is approximated by an arbitrary number of three-dimensional ellipsoidal surfaces, A desired area is extracted by dividing the color space using the body surface. As a result, the accuracy of chroma key processing can be significantly improved.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the drawings.

(1) Principle of the present invention In the image region extraction method according to the present invention, the color distribution shape is approximated by an ellipsoid, and the ellipsoidal shape is used as a division shape to divide the color space, thereby performing chromakey processing. Is designed to improve the accuracy of. Hereinafter, it will be shown that the color distribution shape of the object is a shape that can be approximated by an ellipsoid, and then a calculation method for determining the ellipsoid shape from the color distribution information will be described.

First, an example of the color distribution of an object in the RGB space is shown in FIG. 1 (corresponding to reference drawing 1). Generally, a monochrome object has a rod-shaped distribution extending in the luminance direction or a banana-shaped distribution that bends as the luminance becomes higher. Next, FIG. 2 (corresponding to reference drawing 2) shows an example of the color distribution of the partial region of the object. These distribution shapes are substantially straight rod shapes or blob shapes that can be approximated by ellipsoids. 1 and 2
From (Reference Drawing 1 and Reference Drawing 2), it can be seen that the color distribution of the entire object may be bent like a banana, but if the object is divided into partial areas in the image, the distribution shape can be approximated by an ellipsoid. . Therefore, the method of approximating the color distribution by the combination of ellipsoids is effective.

A method of determining the ellipsoidal shape from the pixel data of a given area in the image will be described below. R
An ellipsoidal surface in the GB space is shown in FIG. The ellipsoidal surface is expressed by the following equations (1) and (2)

[Equation 1]

[Equation 2] Can be represented by In FIG. 3, x, y, z
Is a coordinate value parallel to the axis of the ellipsoid, and c is the center of the ellipsoid. Further, α, β and γ represent the ratio of the size of the three diameters of the ellipsoid, and ρ is a coefficient (hereinafter referred to as scale coefficient) that determines the size of the diameter.

From equations (1) and (2), it can be seen that 16 parameters must be determined to determine the shape of the ellipsoid. Of these, the method of determining 15 parameters excluding the scale factor ρ is shown in FIG. First step SP
Starting from 1, at step SP2, three variables r, g, and
Create the respective mean value μ and the variance-covariance matrix of b.

Next, in step SP3, step S
The three eigenvalues λ and the eigenvectors l, m, and n of the variance-covariance matrix obtained in P2 are calculated, and the obtained values are calculated in step SP4 by the following equations (3) and (4).

(Equation 3)

[Equation 4] Then, the processing is ended in step SP5. Here, an example in which an ellipsoid is determined with the scale coefficient ρ = 3 is shown in FIG. 5 (corresponding to reference drawing 3). In FIG. 5, the diamond shape of the wire frame (shown in yellow in Reference Drawing 3) indicates the direction of the axis of the ellipsoid and the size of each diameter.

Next, a method of adjusting the shape of the ellipsoid determined as described above according to the purpose will be described. Since the ellipsoidal surface is a quadratic surface, the determination as to whether the pixel exists in the inner region or the outer region of the ellipsoid is performed by making equation (1) an inequality and determining the direction of the inequality sign. FIG. 6 shows a method of determining an ellipsoidal surface that separates pixels that should be inside and pixels that should be outside the ellipsoid in a given region by using this discrimination.

First, in step SP2, starting from step SP1, in a given area in the image,
Pixels that should be inside and pixels that are outside of the ellipsoid are designated respectively, and the probability p that they should be inside is given for all the designated pixels. Next, at step SP3, the parameters other than the scale factor ρ are determined by using the pixel which should be inside, by the parameter determination method described in FIG. Next, in step SP4, while changing the value of the scale factor ρ, the following equation (5)

(Equation 5) Σ obtained by the evaluation formula is calculated, and the determination result is q (q = 1 in the case of inside, q = 0 in the case of outside). At step SP5, the scale factor ρ that minimizes σ is selected to determine the size of the ellipsoid.

(2) Application to Chromakey Processing As shown in FIG. 7, in the chromakey processing for extracting the object region in the input image by color, the color is calculated using the ellipsoidal shape determined as described above. The space is divided and the mask image is obtained. This embodiment requires input of a reference image to determine the ellipsoidal shape. The color distribution information of the background, the color distribution information of the object or the color distribution information of both the background and the object is obtained from the reference image, and the ellipsoidal shape is determined using the above-described method.

An image showing a color distribution close to that of the input image is used as the reference image. Specifically, (1) the input image itself, (2) an image temporally close to the input image,
(3) An image of only the background of the input image, and (4) an image of only the object of the input image are used. An image with only the background of the input image,
When the image of only the object of the input image is used as the reference image, the ellipsoidal shape is determined by using the color distribution of all the pixels of the reference image as the color distribution of the background or object, A processing method similar to the conventional chroma key processing for discriminating pixels is used.

When the input image itself or an image that is temporally close to the input image is used as the reference image, the background and the object are separated by designating the background and the area of the object in the reference image. The ellipsoidal shape that best separates can be determined. This process will be described later.

The output image obtained in this embodiment is a binary (mask) image showing the object area. The output image is obtained by the image area division processing by determining whether each pixel is the object area. The determination for each pixel can be performed by the direction of the inequality sign when the equation (1) is set to the inequality. The present invention is used to obtain equation (1).

The overall flow of the ellipsoidal shape determination process is shown in FIG.
Shown in As shown in FIG. 8, the shape of the ellipsoid is determined by two processes, that is, the determination of the outline of the ellipsoid by obtaining the color information and the principal component analysis of the color distribution, and the determination of the scale factor ρ by minimizing σ. It is determined. Here, in this embodiment, a mask image showing an object region in the reference image is used in order to automatically perform the two processes of determining the rough ellipse shape and determining the scale coefficient ρ.

First, FIG. 9 shows a schematic flow of a procedure for determining an ellipsoidal outline by obtaining color information and principal component analysis of color distribution. As shown in FIG. 9, the ellipsoidal outline is determined by data array generation processing, average value calculation, variance-covariance matrix generation, and eigenvalue and eigenvector calculation. Here, the input data is the values R (x, y), G (x, y), B (x, Y) of R, G, B at the pixel position (x, y) in the reference image.
y), the position (x, y) of the pixel in the mask image corresponding to the position (x, y) of the pixel in the reference image, flag flag
(0 or 1), and the sizes of the reference image on the x-axis and the y-axis are xsize and ysize.

First, FIG. 10 shows a processing procedure for data array generation. Starting from step SP1, in step SP2, the count number i is set to "0", and step SP
At 3, the coordinates (x, y) are set to (0, 0). Next, at step SP4, the coordinates (x, y) are (xsize
-1, ysize-1) is determined.

If it is "true", the process proceeds to step SP5, and a flag is set at the pixel position mask (x, y) of the mask image corresponding to the pixel position (x, y) in the reference image. ("1" or "0") is determined. If "true", that is, if the flag is set, the process proceeds to step SP6, where R (x, y), G (x,
y) and B (x, y) are r [i], g [i], and b, respectively.
Substitute in [i], increment the count number i and proceed to step SP7. If "false", that is, the flag is not set in step SP5, the process directly proceeds to step SP7.

In step SP7, the pixel position (x, y) is set to the next position (x, y), and then the process returns to step SP4. When the processing loop from step SP4 to step SP7 is completed for all pixels of the reference image in this way, the number of flagged pixels in the reference image, that is, the number of i, is known, and the number of i is stored in step SP8. The number of datanum is set, and the process ends in step SP9. The resulting output data is the number of data datanum (i), R, G for i pixels,
The values of B are r [i], g [i], and b [i].

Then, r obtained by the data array generation process
The average value μ of each of [i], g [i], and b [i] and the variance-covariance matrix vector V are obtained. FIG. 11 shows the procedure for calculating the mean value μ and the variance-covariance matrix vector V. First, in step SP2, starting from step SP1, the values r [i] of R, G, B obtained from i pixels,
average value vector μr of b [i], g [i],
Calculate μg and μb.

Next, in step SP3, step S
The average value vector μr, μg, μb obtained in P2 and r
The covariance matrix vector V is calculated based on [i], g [i], b [i], and the number of data datanum, and step SP
At 4, the process ends. Here, for example, the component Vrg of the variance-covariance matrix vector V can be obtained by the equation shown in step SP3 of FIG. 11, and the same applies to the other components.

FIG. 12 shows the procedure for calculating the eigenvalues of the variance-covariance matrix vector V. In this embodiment, the eigenvalue is calculated using the Jacobi's method. First, starting from step SP1, in step SP2, the eigenvalues λ1, λ2, and λ3 of the variance-covariance matrix vector V are obtained using the Jacobian method, and the eigenvector l,
After obtaining m and n, the processing is ended in step SP3.

When the outline of the ellipsoid is determined as described above, the scale factor ρ is then determined. This scale coefficient ρ is determined as the scale coefficient ρ when σ obtained by the equation (5) is the minimum. The calculation procedure of the scale factor ρ is shown in FIG.
Shown in Here, as shown in FIG. 8, the mask image is used as an input that gives a probability value that should be inside the ellipsoid, and σ minimization is calculated using this probability value. The input data is r [i], g [i], b obtained by the above-described procedure for determining the ellipsoidal outline.
[I], mean value vector μr, μg, μb, variance / covariance matrix vector V, eigenvalues λ1, λ2, λ3, eigenvectors 1, m, n, and data number datanum.

First, starting from step SP1, the maximum value maxρ of the scale factor ρ is calculated at step SP2. FIG. 14 shows the procedure for calculating the maximum value maxρ of the scale factor ρ. First, start from step SP1,
At step SP2, the count number i is set to "0", and at step SP3, the scale coefficient ρ is set to "0". Next, at step SP4, the count number i
Is less than the number of data datanum.

If it is "true", the process proceeds to step SP5 to obtain the coordinate values x, y, z for the i-th pixel using the equation (4), and the coordinate values x, y, z are (3 ) To obtain the scale factor tmpρ as a temporary variable. Next, in step SP6, the scale factor tmpρ
Is larger than the scale coefficient ρ set in step SP3. If true, step S
Proceed to P7 and substitute the value of tmpρ for the scale factor ρ. Next, in step SP8, the count number i is incremented, and the process returns to step SP4. In this way, the processing loop from step SP4 to step SP8 is executed until the count number i reaches the data number datanum.

As a result, the scale coefficient maxρ that maximizes the scale coefficient ρ among the i scale coefficients ρ is obtained,
The maximum value of the scale factor ρ max ρ in step SP9
The square root of is set to the maximum value maxρ of the scale factor ρ, and the processing is ended in step SP10.

Returning to FIG. 13, when the maximum value maxρ of the scale factor ρ is obtained in step SP2, step S
Go to P3 and substitute the maximum value maxρ of the scale factor ρ in tmpρ. Next, in step SP4, it is determined whether tmpρ is larger than the minimum value minρ of the scale coefficient ρ. If "true", proceed to step SP5 and
Is calculated. Here, the calculation procedure of σ is shown in FIG. The input data are eigenvalues λ1, λ2, λ3, eigenvalue vectors 1, m, n, average value vectors μr, μg, μb, R.
(X, y), G (x, y), B (x, y), mask (x,
y), xsise, ysise.

First, in step SP2, starting from step SP1, the total sum error of errors is set to "0", and the process proceeds to step SP3 to set the position (x, y) to (0,
Set to 0). Next, in step SP4, it is determined whether or not the position (x, y) is (xsise-1, ysise-1). If it is "true", the flow advances to step SP5 to set the values of R, G, B at the position (x, y) to (4).
The coordinate values u, v, w obtained by substituting in the formula are obtained. Next, in step SP6, it is determined whether or not tmpρ is larger than a scale coefficient ρ obtained by substituting the coordinate values u, v, w obtained in step SP5 into x, y, z of the equation (3). judge.

If "true", the process proceeds to step SP7 to set q to "1". If "false", the process proceeds to step SP8 to set q to "0". Next, in step SP9, a value obtained by squaring the error in the pixel at the position (x, y) of the reference image and the mask image is added to the total error sumerror. Next, in step SP10, the pixel position (x, y) is set to the next position (x, y), and then the process returns to step SP4. By executing the processing loop from step SP4 to step SP10 in this way, the sum sumerror of the squared values of the errors obtained for all the corresponding pixels of the reference image and the mask image is obtained.

When the total sum error of all the pixels is obtained in this way, the process proceeds to step SP11,
A square root of a value obtained by dividing the total error sumerror by the number of pixels N is obtained. That is, σ is obtained from equation (5) and step SP1
The process ends at 2.

When .sigma. Is obtained in this way, it proceeds to step SP6 in FIG. 13 and it is determined whether .sigma. Obtained in step SP5 is smaller than the minimum value min.sigma. "true"
If it is, the process proceeds to step SP7 and the minimum value of σ m
Substituting σ obtained in step SP5 for inσ and tmpρ for scale coefficient ρ, step SP8
Proceed to. If "false" is obtained in step SP6, the process directly proceeds to step SP8.

After reducing tmpρ by Δρ in step SP8, the process returns to step SP4. Thus tmpρ
Executes the processing loop from step SP4 to step SP8 until is less than the minimum value minσ of the scale coefficient ρ, and obtains the scale coefficient ρ when tmpρ is less than the minimum value minσ of the scale coefficient ρ and processes at step SP9. To finish.

The scale factor ρ when σ obtained as described above is the minimum is selected, and the size of the approximate shape of the ellipsoid obtained in FIG. 9 is determined. By applying the determined ellipsoidal shape to the chroma key processing as the divided shape, the accuracy of the chroma key processing can be improved.

According to the above construction, the ellipsoidal surface containing the object partial area in the RGB space has a rod-like shape, and the ellipsoidal surface is obtained, and the color space is divided by the ellipsoidal surface. As a result, the accuracy of chroma key processing can be further improved. Further, according to the above configuration, by using the ellipsoidal surface that is a quadric surface as the division shape that divides the color space, it is possible to significantly simplify the discrimination calculation of one ellipsoid.

Further, according to the above-mentioned configuration, an ellipsoid having 16 parameters for determining the shape is used as a division shape for dividing the color space, and these parameters are uniquely determined by analyzing pixel values. Thereby, the shape of the ellipsoid can be easily determined.

Further, according to the above configuration, the diameter in each axial direction can be calculated based on the evaluation value that can be calculated by giving information on whether or not each pixel should be inside or outside the ellipsoid. The shape of the ellipsoid can be adjusted according to the purpose by adjusting the scale factor that is multiplied by the ratio.

In the above embodiment, the case where the input image itself or an image temporally close to the input image is used as the reference image has been described, but the present invention is not limited to this.
An image of only the background of the input image or an image of only the object of the input image may be used as the reference image. In this case, the ellipsoidal shape is determined by using the color distribution of all the pixels of the reference image as the color distribution of the background or the object.

In the above embodiment, the case where the eigenvalues of the variance-covariance matrix are obtained by using the Jacobi method has been described. However, the present invention is not limited to this, and the eigenvalues of the variance-covariance matrix can be calculated by other methods. You may ask.

In the above embodiment, the case where the mask image showing the object area in the reference image is used is described, but the present invention is not limited to this, and the background area or the object area in the reference image is used. A mask image showing both the background areas may be used.

[0049]

As described above, according to the present invention, the color distribution obtained by arranging the color information of the pixel data for each predetermined unit in the image on the color space in which the color information is three-dimensionally arranged. Is approximated by an arbitrary number of three-dimensional ellipsoidal surfaces, and the ellipsoidal surface is used to divide the color space to extract a desired region, whereby the accuracy of chroma key processing can be significantly improved.

[Brief description of drawings]

FIG. 1 is a schematic diagram showing a color distribution of an object in an RGB space.

FIG. 2 is a schematic diagram showing a color distribution of a partial area of an object in an RGB space.

FIG. 3 is a schematic diagram showing the shape of an ellipsoid in RGB space.

FIG. 4 is a flow chart showing a procedure for determining an ellipsoidal shape.

FIG. 5 is a schematic diagram showing a diamond shape in the axial direction and the size of diameter of an ellipsoid that matches the color distribution shape obtained by the image region extraction method according to the present invention.

FIG. 6 is a flow chart showing a processing procedure of an ellipsoidal shape adjusting method.

FIG. 7 is a block diagram showing the flow of processing when an ellipsoidal shape obtained by the image region extraction method of the present invention is used for chroma key processing.

FIG. 8 is a block diagram for explaining an ellipsoidal shape determination process in the embodiment.

FIG. 9 is a flowchart showing a processing procedure from acquisition of color distribution information to principal component analysis.

FIG. 10 is a flow chart showing a processing procedure for data array generation.

FIG. 11 is a flow chart showing a procedure for calculating a variance-covariance matrix.

FIG. 12 is a flow chart showing a procedure for calculating an eigenvalue.

FIG. 13 is a flow chart showing a calculation procedure of σ minimization.

FIG. 14 is a flow chart showing a procedure for calculating a maximum value maxρ of ρ.

FIG. 15 is a flow chart showing a calculation procedure of σ.

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] April 13, 1995

[Procedure Amendment 1]

[Document name to be amended] Statement

[Name of item to be corrected] Brief description of the drawing

[Correction method] Change

[Correction content]

[Brief description of drawings]

FIG. 1 is a photograph showing the color distribution of an object in RGB space.

FIG. 2 is a photograph showing a color distribution of a partial area of an object in RGB space.

FIG. 3 is a schematic diagram showing the shape of an ellipsoid in RGB space.

FIG. 4 is a flow chart showing a procedure for determining an ellipsoidal shape.

FIG. 5 is a photograph showing a diamond shape in the axial direction and the size of diameter of an ellipsoid that matches the color distribution shape obtained by the image region extraction method according to the present invention.

FIG. 6 is a flow chart showing a processing procedure of an ellipsoidal shape adjusting method.

FIG. 7 is a block diagram showing the flow of processing when an ellipsoidal shape obtained by the image region extraction method of the present invention is used for chroma key processing.

FIG. 8 is a block diagram for explaining an ellipsoidal shape determination process in the embodiment.

FIG. 9 is a flowchart showing a processing procedure from acquisition of color distribution information to principal component analysis.

FIG. 10 is a flow chart showing a processing procedure for data array generation.

FIG. 11 is a flow chart showing a procedure for calculating a variance-covariance matrix.

FIG. 12 is a flow chart showing a procedure for calculating an eigenvalue.

FIG. 13 is a flow chart showing a calculation procedure of σ minimization.

FIG. 14 is a flow chart showing a procedure for calculating a maximum value maxρ of ρ.

FIG. 15 is a flow chart showing a calculation procedure of σ.

[Procedure Amendment 2]

[Document name to be corrected] Drawing

[Name of item to be corrected] Figure 1

[Correction method] Change

[Correction content]

FIG.

[Procedure 3]

[Document name to be corrected] Drawing

[Name of item to be corrected] Figure 2

[Correction method] Change

[Correction content]

[Fig. 2]

[Procedure amendment 4]

[Document name to be corrected] Drawing

[Name of item to be corrected] Figure 5

[Correction method] Change

[Correction content]

[Figure 5]

Claims (3)

[Claims] 1. An image area extracting method for extracting a desired area from an image, wherein color information of pixel data for each predetermined unit in the image is arranged in a color space in which color information is arranged three-dimensionally. An image region extracting method, characterized in that the obtained color distribution is approximated by an arbitrary number of three-dimensional ellipsoidal faces, and the desired region is extracted by dividing the color space using the ellipsoidal faces. 2. The shape of the ellipsoidal surface is obtained as an average value and a variance-covariance matrix of each of the color information of pixel data constituting a predetermined unit in the image, and the obtained variance-covariance matrix is obtained. 2. The image area extracting method according to claim 1, wherein the determination is made by obtaining the eigenvalue and the eigenvector of. 3. The coefficient for determining the size of the diameter of the ellipsoidal surface is determined by determining whether the pixel data is inside or outside the ellipsoidal surface while changing the coefficient. The image area extracting method according to claim 1 or 2, wherein the image area extracting method is determined as follows.