JP2013120504A - Object extraction device, object extraction method and program - Google Patents

Abstract

An object extraction apparatus capable of tracking and extracting a target object designated by a user with high accuracy.
An object extracting apparatus extracts an object region of interest from a time-series image. An object region and an object are extracted from an image of the object region on a reference image in the time-series image and an image of its peripheral region. Based on the correspondence between the feature point in the object region on the reference image in the time-series image and the feature point of the processing target image in the time-series image An object region estimation unit 30 that estimates an object region on the processing target image, calculates the feature amount of the image of the estimated object region and its surrounding region, and estimates by comparing the calculated feature amount and the identification feature amount The area of the object in the processed image is corrected and confirmed.
[Selection] Figure 1

Description

  The present invention relates to an object extraction apparatus and an object extraction method for extracting an object of interest desired by a user from a video.

  In recent years, terminals with an imaging function such as a digital video camera, a digital still camera, and a mobile phone are rapidly spreading. In addition, there is an apparatus that generates a new video by processing and processing video shot by these cameras. For example, a specific image area in a video can be extracted, and the extracted image area can be used as a part for video processing, or a video can be extracted for each individual image area, and the extracted video can be managed in the original video. It is known to use for search.

As a technique for tracking and extracting a target object designated by a user from a video, for example, there is a technique described in Patent Document 1.
In Patent Literature 1, a partial area of an area where the object exists is detected based on a silhouette image indicating an object area acquired based on a background difference and an image characteristic amount (color information) unique to the object, and the partial area An object extraction device is disclosed that extracts an entire object region by growing the region based on the silhouette image.

Japanese Patent Laid-Open No. 2003-44860

  However, since Patent Document 1 acquires a silhouette image indicating an object area based on a background difference, if the background changes due to camera work or a scene change, the background area is included in the silhouette image indicating the object area. It may be extracted as an object area by mistake.

  The present invention has been made in view of the above points, and provides an object extraction device, an object extraction method, and a program capable of tracking and extracting a target object designated by a user with high accuracy.

  In order to solve the above-described problem, a first technical means of the present invention is an object extraction device that extracts an object region of interest from a time-series image, the image of the object region on the reference image in the time-series image and its An identification feature amount calculating unit for calculating an identification feature amount for identifying an object region and an object peripheral region from an image of the peripheral region, a feature point in the object region on the reference image in the time-series image, and the time-series image An object region estimation unit for estimating an object region on the processing target image by correspondence with a feature point of the processing target image; and calculating a feature amount of the image of the estimated object region and its surrounding region; The region of the object in the estimated processing target image is corrected by comparing the calculated identification feature amount An object region determination unit for constant, is obtained by comprising: a.

  According to a second technical means of the present invention, in the first technical means, the identification feature amount calculation unit is configured to identify an object region and an object peripheral region based on a silhouette image indicating the object region in the reference image. A feature amount calculation range setting unit that sets a feature amount calculation range for calculating the image feature amount, and an image feature amount calculation unit that calculates the image feature amount for each small region of a predetermined size in the set feature amount calculation range A representative image feature amount calculation unit that clusters image feature amounts on the object region out of the image feature amounts and calculates K different representative image feature amounts expressing the image feature amount of the object region; Further, a similarity vector for calculating a similarity vector between the image feature quantity and the representative image feature quantity from the image feature quantity of the small region and K different representative image feature quantities. Identification for calculating an identification function representing a class boundary as to whether or not a small area belongs to the object area in the feature amount space from the calculation unit and the calculated similarity vector on the object area and the similarity vector of the object peripheral area A function calculation unit, and outputs the K different representative image feature values and the discrimination function as the discrimination feature values.

  According to a third technical means of the present invention, in the second technical means, a feature amount calculation range in which the object region determining unit calculates an image feature amount based on an estimated silhouette image indicating the estimated object region. A feature amount calculation range setting unit to be set, an image feature amount calculation unit that calculates an image feature amount for each small region of a predetermined size in the set feature amount calculation range, and an image feature of the small region for each small region A similarity vector calculation unit that calculates the similarity vector from an amount and K different representative image feature values in the reference image, and based on the calculated similarity vector and the identification function, Whether each small area belongs to the object area in the feature amount space is identified, and a set of small areas determined to be included in the object area is defined as an object on the processing target image. Is obtained is characterized in that determining the region.

  According to a fourth technical means of the present invention, in any one of the first to third technical means, the object region estimation unit includes the reference image and a silhouette image indicating an object region on the reference image. A feature point calculation unit for calculating a feature point in an object area on the reference image, a feature point in the object area on the reference image, a silhouette image indicating the object area on the reference image, and the processing target A corresponding point calculating unit that calculates a feature point on the processing target image corresponding to a feature point in the object area on the reference image from the image, a feature point in the object area on the reference image, and the feature point And a feature matrix on the image to be processed corresponding to a feature point in the object area on the reference image to a feature point on the image to be processed corresponding to the feature point A conversion matrix calculating unit for calculating, on the reference image, multiplying each pixel on the processing target image by the inverse matrix of the conversion matrix from the conversion matrix and the silhouette image indicating the object region on the reference image. It is determined whether it is included in the object area, and a set of pixels determined to be included in the object area is estimated as an object area on the processing target image.

  According to a fifth technical means of the present invention, there is provided an object extraction method for extracting an object region of interest from a time-series image, wherein the object region is obtained from an image of the object region on the reference image and its surrounding region in the time-series image. An identification feature amount calculating step for calculating an identification feature amount for distinguishing between the object peripheral region and the object peripheral region, a feature point in the object region on the reference image in the time-series image, and a feature point of the processing target image in the time-series image An object region estimation step for estimating an object region on the processing target image by correspondence; calculating a feature amount of the image of the estimated object region and its surrounding region; and the calculated feature amount and the calculated identification feature amount The object for correcting and confirming the object region in the estimated processing target image by comparing And transfected region determination step, it is obtained by comprising a.

  According to a sixth technical means of the present invention, in the fifth technical means, the identification feature amount calculating step is for identifying an object region and an object peripheral region based on a silhouette image indicating the object region in the reference image. A feature amount calculation range setting step for setting a feature amount calculation range for calculating the image feature amount, and an image feature amount calculation step for calculating the image feature amount for each small region of a predetermined size in the set feature amount calculation range; A representative image feature amount calculating step of clustering image feature amounts on the object region out of the image feature amounts and calculating K different representative image feature amounts expressing the image feature amount of the object region; Further, from the image feature amount of the small area and K different representative image feature amounts, the similarity vector between the image feature amount and the representative image feature amount is calculated. A class boundary indicating whether or not a small region belongs to the object region in the feature amount space from the similarity vector calculating step for calculating the similarity vector and the similarity vector on the object region and the similarity vector of the object peripheral region An identification function calculating step for calculating an identification function; and a step of outputting the K different representative image feature values and the identification function as the identification feature values.

  According to a seventh technical means of the present invention, in the sixth technical means, a feature amount calculation range in which the object region determination step calculates an image feature amount based on an estimated silhouette image indicating the estimated object region. A feature amount calculation range setting step to be set; an image feature amount calculation step of calculating an image feature amount for each small region of a predetermined size in the set feature amount calculation range; and an image feature of the small region for each small region A similarity vector calculating step for calculating the similarity vector from the amount and K different representative image feature values in the reference image, and each subregion based on the calculated similarity vector and the identification function Are included in the object area in the feature amount space, and a set of small areas determined to be included in the object area is defined as the processing target image. A step of determining an object region of the upper is obtained by comprising a.

  According to an eighth technical means of the present invention, in any one of the fifth to seventh technical means, the object region estimation step includes the reference image and a silhouette image indicating the object region on the reference image. A feature point calculating step for calculating a feature point in the object area on the reference image, a feature point in the object area on the reference image, a silhouette image showing the object area on the reference image, and the processing target A corresponding point calculating step for calculating a feature point on the processing target image corresponding to a feature point in the object region on the reference image from the image, a feature point in the object region on the reference image, and the feature point And a feature point on the processing target image corresponding to the feature point in the object area on the reference image A reference image obtained by multiplying a transformation matrix calculation step for calculating a transformation matrix to be projected, the transformation matrix, and a silhouette image indicating an object region on the reference image by multiplying an inverse matrix of the transformation matrix for each pixel on the processing target image. Determining whether or not the object area is included in the upper object area, and estimating a set of pixels determined to be included in the object area as the object area on the processing target image. It is.

  A ninth technical means of the present invention is an object extraction program for causing each processing unit of the object extraction device of any one of the first to fourth technical means to function.

  According to the present invention, an object of interest designated by a user can be tracked and extracted with high accuracy.

It is a block diagram showing the structure of the object extraction apparatus which concerns on one Embodiment of this invention. It is a flowchart showing operation | movement of the object extraction apparatus of FIG. It is a block diagram showing the structure of the identification feature-value calculation part of the object extraction apparatus of FIG. It is a flowchart showing operation | movement of the identification feature-value calculation part of the object extraction apparatus of FIG. It is a figure explaining the setting of the calculation range of the feature-value of an object area | region and an object peripheral area | region. It is a schematic diagram which calculates | requires an identification function in the identification feature-value calculation part of the object extraction apparatus of FIG. It is a schematic diagram showing operation | movement of the identification feature-value calculation part of the object extraction apparatus of FIG. It is a block diagram showing the structure of the object area | region estimation part of the object extraction apparatus of FIG. It is a flowchart showing operation | movement of the object area estimation part of the object extraction apparatus of FIG. It is a schematic diagram showing operation | movement of the object area | region estimation part of the object extraction apparatus of FIG. It is a block diagram showing the structure of the object area | region determination part of the object extraction apparatus of FIG. It is a flowchart showing operation | movement of the object area | region determination part of the object extraction apparatus of FIG. Time t ₁ of the image (target image) and the estimated silhouette image, and time t ₂ of the image (reference image) and an example showing the silhouette image. A feature quantity calculation range at time t ₁ of the image (processed image) is an example showing the results of identifying the object region in the feature calculation range. It is a schematic diagram showing the calculation method of a gradient histogram.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, portions having the same function are denoted by the same reference numerals, and repeated description is omitted.

  FIG. 1 is a block diagram showing a schematic configuration of an object extracting apparatus according to the present invention. In this figure, the object extraction apparatus 1 includes an identification feature amount calculation unit 20, an object region estimation unit 30, and an object region determination unit 40. FIG. 2 is a flowchart for explaining an operation example of the object extraction apparatus 1 of FIG.

The object extraction apparatus 1 in FIG. 1 includes an image F (t ₀ ) (hereinafter also referred to as a reference image) at time t ₀ and a silhouette image S indicating a region on the reference image of an object to be extracted that has been created in advance. (T ₀ ) is acquired from the outside and input to the discriminating feature amount calculation unit 20 (step S1-1 in FIG. 2). Here, the silhouette image S (t) is a set of pixels where “S (x, y | t) = 1” is an object area (also referred to as a foreground area), and “S (x, y | t) = 0. Is a binary image showing a set of pixels as a background region. Moreover, although the silhouette image is used as the information indicating the object area, the present invention is not limited to this. Instead of the silhouette image, contour information expressing the contour line of the object region with a chain code may be used. The chain code is a numerical value of a position of a point B adjacent to a certain point A, and a numerical value of a position of a point C (a point other than the point A) adjacent to the adjacent point B. , And so on, and a line is represented by the combination of those numerical values.

(About the identification feature amount calculation unit 20)
Based on the input image F (t ₀ ) and the silhouette image S (t ₀ ) indicating the region of the extraction target object _OFG on the image F (t ₀ ), the identification feature amount calculation unit 20 N representative image feature amounts (representative local feature amounts) representing the _FG and image feature amounts (local feature amounts) of each small area in the image F (t ₀ ) are calculated, and the representative local feature amounts and the respective small feature amounts are calculated. An identification function (boundary between the object area and the other area) that identifies whether or not a small area is an object to be extracted is calculated from the local feature quantity of the area, and the calculated identification feature quantity (Representative local feature value and discriminant function) are output to the object area determining unit 40 (step S1-2 in FIG. 2). Hereinafter, as an example of the image feature amount, the identification feature amount calculation unit 20 in the case of using a color histogram will be described with reference to FIGS.

  As illustrated in FIG. 3, the identification feature amount calculation unit 20 includes a feature amount calculation range setting unit 201, a local feature amount calculation unit 202, a representative local feature amount calculation unit 203, a similarity vector calculation unit 204, and an identification function calculation unit. 205. FIG. 4 is a flowchart for explaining an operation example of the identification feature amount calculation unit 20.

The feature amount calculation range setting unit 201 calculates local feature amounts of the object region and the object peripheral region based on the input silhouette image S (t) (silhouette image S (t ₀ ) in the example of FIG. 3). The set feature amount calculation range is output to the local feature amount calculation unit 202 and the discrimination function calculation unit 205 (step S20-1 in FIG. 4). The setting of the feature amount calculation range of the object area and the object peripheral area will be described with reference to FIG.

FIG. 5A is a silhouette image S (t ₀ ) representing the object area _OFG at time t ₀ . First, the feature quantity calculation range setting unit 201, as shown in FIG. 5 (B), is performed K times the expansion processing is the morphology operation the object area _{O FG} by a predetermined kernel object region to an extended region ^O _{K dilated} Ask for. Subsequently, feature calculation range setting unit 201 obtains the 5 area excluding the object area _{O FG} from Extended object area ^O _{K dilated} shown in (C) (Object peripheral region _{O Around).} At this time, the object peripheral area O _Around is set so as to be approximately the same as the area of the object area _{OFG in} order to make the number of samples of the feature amount acquired from the object area and other areas (object peripheral area) the same. It is desirable. For example, the area of the extended object region obtained by K-th expansion processing ^S _{(O k dilated),} if the area of the object area _{O FG} and S _{(O FG),} extended satisfying equation (20-1) in the first from the object area ^O _{K dilated,} it may acquire the area excluding the object area _{O FG.}

3 is based on the input image F (t) (F (t ₀ ) in the example of FIG. 3) and the feature amount calculation range set by the feature amount calculation range setting unit 201. Te, and calculates the local feature amount of each small area included in the object area _{O FG} and objects peripheral region _{O around} (step S20-2 in Fig. 4). Here, a calculation method will be described using a color histogram as an example of the local feature amount. It is assumed that the small area is a circle with a radius r centered on the pixel (x, y). First, the local feature amount calculation unit 202 acquires a pixel value C _Q (r, g, b) quantized with a quantization step size Q for each pixel in the small region (Formula (20-2)). In this example, the small region is described as a circle having a radius r centered on the pixel (x, y), but is not limited thereto. For example, it may be a rectangular area in units of N × M pixels with the pixel (x, y) as the center.

In Expression (20-2), the operator “//” is integer division. Subsequently, the appearance frequency of the quantized pixel value C _Q (r, g, b) is counted, and a color histogram H _C (x, y) in which each element of the histogram is normalized by the area of the small region is calculated. The local feature amount 202 is calculated using RGB as the color space of the image. However, the present invention is not limited to this, and the present invention is not limited to this. YCbCr (YUV), CIE L * a * b *, CIE L * u * V * or other color space may be used.

  The representative local feature value calculation unit 203 in FIG. 3 clusters the local feature values of the input object region and each small region of the object peripheral region, and calculates K representative local feature values representing the object region ( Step S20-3 in FIG. For example, the K-means method may be applied to the local feature amount to acquire K representative local feature amounts.

The similarity vector calculation unit 204 in FIG. 3 uses the local feature amount (color histogram) of each small region and the representative local feature amount (color histogram) representing the image feature amount of the object region to reduce the object region and the object peripheral region. The similarity between the local feature amount for each region and the K representative local feature amounts of the object region is calculated, and the similarity vector is output (step S20-4 in FIG. 4). Here, the color histogram _H c (x, y) of the small area representative color histogram _H c, k _{(O FG)} and similarity _{d (H C (x, y} ), H C, k (O FG)) Is calculated by, for example, a histogram intersection shown in Expression (20-3). The similarity value range is 0 to 1. The closer the value is to 1, the more similar the shape of the histogram is. The closer the value is to 0, the different the shape of the histogram is.

The discriminant function calculation unit 205 in FIG. 3 uses the similarity vector between the representative local feature amount of the input object region _{OFG and} the local feature amount of each small region of the object region _OFG and the object peripheral region _OAround . As shown in FIG. 6, an identification function “φ (x) = 0” for identifying (classifying) the object area _OFG and the object peripheral area O _Around in the feature amount space is calculated (step S20-5 in FIG. 3).

In FIG. 7, consider a problem of classifying each small region in the region surrounded by the dotted line U on the image F (t ₀ ) into two classes of an object region and an object peripheral region. The classification problem into two classes includes data D = {(x ^u , ^yu ), u = 1, 2,. . . , N} and is formulated as a problem for obtaining the discriminant function φ (x) that minimizes the sum-of-squares error shown in equation (20-4).

In Expression (20-4), sgn (•) is a sign function. Here, the data D, x ^u represents the feature vector of u-th small region on the region U in FIG. 7, y ^u is correct u-th small region of whether belonging to the object area O _FG Is a teacher signal represented by a 2 × 1 vector. Note that the teacher signal ^yu is obtained when the u-th small region belongs to the object region _OFG (the value of the silhouette image S (t) of the pixel (x, y) that is the center of the small region is “S (x, y | T) = 1 ”,“ y ^u = (1, 0) ”, otherwise (the value of the silhouette image S (t) of the pixel (x, y) that is the center of the small region is“ S (x, y | t) = 0), and “y ^u = (0, 1)”. Here, if it is assumed that the discriminant function φ (x) is given by the equation (20-5), the equation (20-4) becomes a problem for obtaining the weight w that minimizes the equation (20-6).

  When the equation (20-6) is solved by the least square method, the weight w is given as the equation (20-7).

As described above, according to the identification feature amount calculation unit 20, the N representative local feature amounts representing the image feature amount of the object region from the reference image F (t ₀ ) and its silhouette image S (t ₀ ), An identification function for identifying whether each small region belongs to the class of the object region or the object peripheral region can be calculated from the similarity vector with the local feature amount for each small region of the object region and the object peripheral region. . In this example, the identification feature amount calculation unit 20 has been described assuming that the identification function is linearly separable and is calculated by the least square method. However, the present invention is not limited to this. For example, the discriminant function may be calculated by machine learning such as a non-linear support vector machine, a decision tree, or a multilayer perceptron instead of the least square method.

In FIG. 7, a color histogram H _c (x, y), which is an example of a local feature amount in a small area, and a representative color histogram H _{c, 1} (an example of K representative local feature amounts in an object area) are shown. x, y) to H _{c, K} (x, y) are also shown.

Returning to FIG. 2, the object extraction apparatus 1 in FIG. 1 includes an image F (t ₁ ) at time t ₁ (hereinafter also referred to as a processing target image) and an object region at time t ₂ calculated immediately before time t _1. A silhouette image S (t ₂ ) indicating the above and an image F (t ₂ ) (hereinafter also referred to as a reference image) are acquired from the outside and input to the object region estimation unit 30 (step S1-3 in FIG. 2).

(About the object region estimation unit 30)
Returning to FIG. 2, the object region estimation unit 30 in FIG. 1 inputs the input image F (t ₁ ), the previous image F (t ₂ ) stored externally, and the image F (t ₂ ) From the silhouette image S (t ₂ ) showing the object region on the top, a transformation matrix H for projecting the feature points on the object region at time t ₂ onto the corresponding feature points at time t ₁ is obtained, and the transformation matrix From this, an estimated silhouette image S _pred (t ₁ ) in which the object region at time t ₁ is estimated is generated and output to the object region determining unit 40 (step S1-4 in FIG. 2).

Next, the object region estimation unit 30 in the present embodiment will be described in detail with reference to FIGS. FIG. 8 is a block diagram illustrating a schematic configuration of the object region estimation unit 30. As illustrated in FIG. 8, the object region estimation unit 30 includes a feature point calculation unit 301, a corresponding point calculation unit 302, a transformation matrix calculation unit 303, and an estimated silhouette image generation unit 304. FIG. 9 is a flowchart for explaining an operation example of the object region estimation unit 30. FIG. 10 is a schematic diagram illustrating an operation of estimating the object region at time t ₁ from the correspondence between the feature points at time t ₁ and time t ₂ .

In step S30-1 in FIG. 9, the feature point calculation unit 301 in FIG. 8 correlates the image between the object region of the previous image F (t ₂ ) and the input image F (t ₁ ). N feature points Ks (t ₂ ) (s = 1,..., N) in the object region in the image F (t ₂ ) are calculated, and the result is output to the corresponding point calculation unit 302. (Step S30-1).
A feature point is a point extracted as a part or vertex of a subject's edge based on a change in color or luminance between pixels. For example, it is represented using a gradient vector G _i (x, y) (i = x, y) of luminance in the x and y directions within the range of the local region S with a certain pixel (x, y) as the center. First eigenvalue λ ₁ and second eigenvalue λ ₂ of second moment matrix A (Equation (30-1)) are obtained, and pixel (x, y) satisfying the condition shown in Equation (30-2) is detected as a feature point. To do.

That is, a characteristic point is obtained when the smaller eigenvalue of the first eigenvalue λ ₁ and the second eigenvalue λ ₂ of the second moment matrix A is greater than (or greater than) the predetermined threshold λ _th (for example, see "J. Shi and C. Tomasi," Good Features to Track, "9 th IEEE Conference on Computer Vision and Pattern Recognition, June 1994 "). In the equation (30-1), the coefficient w (u, v) represents a weighting coefficient related to the pixel (x + u, y + v) separated from the pixel (x, y) by u in the x direction and v in the y direction. A value obtained by normalizing the value of the secondary Gaussian distribution within the range of the local region S, which is determined so as to satisfy the condition of Expression (30-3), is used.

Here, FIG. 10A shows an example of the result of detecting the feature point Ks (t ₂ ) from the object area of the image F (t ₂ ).
Proceeding to step S30-2 in FIG. 9, the corresponding point calculation unit 302 in FIG. 8 receives the input image F (t ₁ ), the previous image F (t ₂ ) read from the outside, and step S30. Based on the feature point information K (t ₂ ) of the image F (t ₂ ) acquired at −1, each feature point Ks (t ₂ ) (s = 1,..., N) of the image F (t ₂ ) corresponding image F _{(t 1)} point on Ks a _{(t 1)} is calculated by the optical flow for the time _{t 1} of the feature point Ks, time _t corresponding points describing the position in ₂ information Q _(t 1, t and ₂ ) is output to the transformation matrix calculation unit 303 (step S30-2). The FIG. 10 (A), the image F _{(t 2)} feature point Ks _{(t 2)} of the object area of the image F _{(t 1)} on the corresponding feature point Ks _{(t 1)} detected result is shown Has been. A dotted line connecting the feature point Ks (t ₁ ) on the image F (t ₁ ) that matches the feature point Ks (t ₂ ) on the object region of the image F (t ₂ ) is the corresponding point information Q (t ₁ , T ₂ ) is a visual representation.

Each feature point Ks of the image F _{(t 2) (t 2)} and the corresponding image F _{(t 1)} points on Ks _{(t 1),} for example, optical with a gradient method shown in Equation (30-4) It can be obtained by solving the flow constraint on Ks (t ₁ ) (for example, “BD Lucas and T. Kanade,“ An iterative image registration technique with an application to stereo vision, ”Proceedings of the 1981 DARPA imaging Understanding Workshop ( pp. 121-130), 1981 ”).

Here, in Expression (30-4), G _i (x, y | t ₁ ) (i = x, y, t) is the x direction, y direction, and t direction (time) regarding the luminance of the image F (t ₁ ). Direction), and S represents a local region of a predetermined size centered on the feature point Ks.

Proceeding to step S30-3 in FIG. 9, the transformation matrix calculating unit 303 in FIG. 8 uses the feature point Ks (s = 1,...) From the corresponding point information Q (t ₁ , t ₂ ) acquired in step S30-2. .., N) is calculated from a position on the previous image F (t ₂ ) to a position on the image F (t ₁ ), and information describing the conversion matrix H is estimated. It outputs to the silhouette image generation part 304 (step S30-3).

The coordinates of a point obtained by projecting the pixel (x, y) on the object area O (t ₂ ) on the image F (t ₂ ) onto the image F (t ₁ ) by the transformation matrix H are (x _pred , y _pred ). Then, the correspondence relationship between the estimated object region O _pred (t ₁ ) and the object region O (t ₂ ) is expressed by Expression (30-5) using the transformation matrix H (for example, “CG-ARTS Association, Digital image processing 2nd edition, 2009 ").

In the expression (30-5), the symbol “˜” represents an equivalence relationship, which means that they are equal by allowing a constant multiple difference. The transformation matrix H is called projective transformation because it can express general transformation. Here, assuming that the correspondence between the object region at time _{t 2} and time _{t 1} can be expressed as a translation, formula (30-5) is expressed as Equation (30-6).

Coefficient _t x in the formula (30-6), _{t y} is x-direction, respectively, represent the amount of movement in the y-direction. Assuming that the correspondence between images can be expressed as affine transformation including translation, rotation, and enlargement / reduction, Expression (30-5) is expressed as Expression (30-7).

Factor in the equation (30-7) a, b, c, d is scaled, and represents a rotation factor _t x, is _{t y} is the same as in the formula (30-6). Note that the coefficients h _ij (i, j = 1, 2, 3) of the transformation matrix H in the equations (30-5) to (30-7) are the transformation models (translation, affine transformation, projective transformation). This is calculated by solving simultaneous equations derived from the constraint conditions and the corresponding point information Q (t ₁ , t ₂ ) by the method of least squares. If there is not a sufficient number of corresponding points and the conversion matrix H cannot be calculated, a predetermined conversion matrix H ₀ is used. For example, as a specific example of the transformation matrix H ₀ , the calculation result of the previous transformation matrix is used.

Proceeding to step S30-4 in FIG. 9, the estimated silhouette image generation unit 304 in FIG. 8 determines that “the object region at time t ₁ is the previous one based on the transformation matrix H and the silhouette image S (t ₂ ). Assuming that the object area on the image F (t ₂ ) is an area projected onto the image F (t ₁ ) using the transformation matrix H calculated by the transformation matrix calculation unit 303 in FIG. The object area of t ₁ is estimated, and an estimated silhouette image S _pred (t ₁ ) representing the object area is generated (step S30-4). Specifically, for each pixel (x, y) (point Pn in FIG. 10B) on the image F (t ₁ ), the image F (t ₂ ) is multiplied by the inverse matrix H ⁻¹ of the transformation matrix H. It is determined whether the coordinates (x ′, y ′) (point Pn ′ in FIG. 10B) of the back-projected position are included on the object area O (t ₂ ) at time t ₂ . Note that the object region O (t ₂ ) at time t ₂ is a set of pixels with “S (x, y | t ₂ ) = 1” on the silhouette image S (t ₂ ).

As shown in Equation (30-8), if the image F _{(t 1)} on the pixel (x, y) was back projected to the image F _{(t 2)} position is included in the object area O _{(t 2),} The pixel (x, y) is determined to be a pixel constituting the estimated object region O _pred (t ₁ ), and “1” is set to the estimated silhouette image S _pred (x, y | t ₁ ). Further, when the pixel (x, y) on the image F (t ₁ ) is not included in the object area O (t ₂ ), the pixel (x, y) is not a pixel constituting the estimated object area O _pred (t ₁ ). It is determined that the estimated silhouette image S _pred (x, y | t ₁ ) is “0”. Here, FIG. 10B shows an example of the estimated silhouette image S _pred (t ₁ ) obtained from the silhouette image S (t ₂ ).

As described above, according to the object region estimation unit 30, based on the specific feature point of the object area, to determine the correspondence between the object region at time t ₁ and time t _2, the camera work, accompanied by changes in the background such as a scene change Even in this case, a robust object region can be estimated.

(About the object area determination unit 40)
Returning again to FIG. 2, the object region determination unit 40 in FIG. 1 sets a range for calculating the feature amount from the estimated silhouette image S _pred (t ₁ ), and performs local processing for each small region on the image F (t ₁ ). The feature amount is calculated, and based on the local feature amount and the identification feature amount (representative local feature amount and identification function φ (x)) of the reference image, the object region is identified. A silhouette image S (t ₁ ) is output (step S1-4 in FIG. 2).
Next, the object area determination unit 40 in this embodiment will be described in detail.

As shown in FIG. 11, the object region determination unit 40 includes a feature amount calculation range setting unit 201, a local feature amount calculation unit 202, a similarity vector calculation unit 203, and an object region identification unit 404. FIG. 12 is a flowchart for explaining an operation example of the object area determination unit 40. 13A is an image F (t ₁ ) at time t ₁ , FIG. 13B is an image F (t ₂ ) at time t ₂ , and FIG. 13C is an object region at time t _1. The estimated silhouette image S _pred (t ₁ ) and FIG. _13D are examples of the silhouette image S (t ₂ ) showing the object region at time t ₂ . Incidentally, in the object region at time t ₁ of FIG. 13 (A), the region R1 surrounded by a dotted line, a dotted line portion surrounded by a region R2 on the object region at time t ₂ shown in FIG. 13 (B) shape Suppose it has changed. Further, FIG. 14 (A) Fig. 13 (C) an enlarged view of a region B1 surrounded by a dotted line on FIG. 14 (B) is the feature quantity calculation range within the region B1 ^O _K, the object area from the _dilated and objects It is an example of the result of having identified the peripheral region.

The feature amount calculation range setting unit 201 in FIG. 11 uses the estimated silhouette image S _pred (t ₁ ) indicating the input estimated object region at time t ₁ as a range for calculating the local feature amount as shown in FIG. the expansion process is a morphological operation is performed K times with the estimated object region O _pred predetermined kernel as shown in), seeking region O ^K _dilated an extension of the region, local feature calculation section 202 and the calculation range, object It outputs to the area | region identification part 404 (step S40-1 of FIG. 12).

The local feature quantity calculation unit 202 in FIG. 11 calculates the local feature quantity of each small region within the input image F (t ₁ ) and the feature quantity calculation range set by the feature quantity calculation range setting unit 201. (Step S40-2 in FIG. 12).

  The similarity vector calculation unit 203 in FIG. 11 calculates the local feature amount for each small region of the object region and the object peripheral region based on the local feature amount of each small region and the representative local feature amount representing the image feature amount of the object region. The similarity with the K representative local feature quantities of the object area is calculated, and the similarity vector x is output (step S40-3 in FIG. 12).

Object area identifying unit 404 in FIG. 11, a similarity degree vector x of each small region is inputted, based on the identification function phi (x), the feature amount calculation range O ^K, each small area on the _dilated is an object region A silhouette image S (t ₁ ) indicating the determined object region is output as shown in FIG. 13D (step S40-4 in FIG. 12). Specifically, when a similarity vector x of a small area centered on the pixel (x, y) is input to the discrimination function φ (x) and the response value indicates an object area (φ (x)> 0) , The pixel (x, y) is determined to be a pixel constituting the object region _OFG , and “1” is set to the silhouette image S (x, y | t ₁ ). If the response value indicates the object peripheral area (φ (x) <0), it is determined that the pixel (x, y) is not a pixel constituting the object area _OFG , and the silhouette image S (x, y | t ₁ ) Set "0". When the response value indicates the boundary of the discriminant function (φ (x) = 0), it is randomly selected whether the region is the object region or the object peripheral region, and corresponds to the silhouette image S (x, y | t ₁ ). A value shall be set. It is assumed that the silhouette image S (t ₁ ) has been initialized with “0”.

Figure 14 (B) is the feature quantity calculation range ^O _K in FIG. 14 _(A), the respective small regions on the _dilated, the result of identifying whether the object region. In FIG. 14A, the estimated object region O _pred obtained by projective transformation of the object region at time t ₂ to time t ₁ is the shape change of the region R ₂ at time t ₂ among the object regions at time t _1. The region R1 is not included. In other words, the object region accompanying the shape change cannot be extracted with high accuracy simply by projective transformation of the object region at time t ₂ to time t ₁ . However, if it contains regions R1 to extend the estimated object region feature calculation range O ^K _dilated, by identifying whether the object region can be extracted area R1. That is, it is possible to accurately extract an object region that accompanies a shape change.
As described above, according to the object region determination unit 40, a feature amount calculation range is set from the estimated object region, and it is identified whether the feature region is an object region or an object peripheral region with respect to the set feature amount calculation range. The area can be determined. Therefore, an object region with a shape change can be extracted as long as it is within the feature amount calculation range. In addition, since the feature amount calculation range is limited to the peripheral portion of the object region, it is possible to reduce the amount of calculation required for calculating the feature amount.

Returning to FIG. 2, the object extracting apparatus 1 of FIG. 1, for performing object extraction processing in the image of the next time of the time _{t 1 (t} 1 +1), it returns to step S1-3 (in FIG. 2, step S1- 6). If there is no image at time (t ₁ +1), the object extraction process is terminated (No in step S1-6 in FIG. 2).

(effect)
As described above, the object extraction device uses the silhouette image indicating the object area in the first image (reference image) on the time-series image, and the image signal of the object area on the reference image and the object peripheral area to calculate each small area. The local feature amount of the region and the local feature amount representing the object region (representative local feature amount) are calculated, and each small region is converted into the object region based on the calculated representative local feature amount and the local feature amount of each small region. An identification function for identifying whether or not is calculated. Further, based on the second image (processing target image) on the time-series image and the silhouette image and reference image indicating the object area in the third image (reference image) on the time-series image, A correspondence relationship between the object areas between the reference images is calculated, and a transformation matrix for projectively transforming the object area on the reference image onto the processing target image is calculated from the correspondence relation. Based on the calculated transformation matrix and the silhouette image of the reference image, an object region on the processing target image is estimated to generate an estimated silhouette image. Further, the local feature amount of each small region is calculated from the estimated object region indicated by the estimated silhouette image on the processing target image and the image signal of the processing target image, the local feature amount, the representative local feature amount of the reference image, and the discrimination function Based on the above, it is determined whether or not each small area is an object area, and the object area on the processing target image is determined. Thereby, the object of interest designated by the user can be tracked and extracted with high accuracy.

(Modification of local feature amount calculation unit 202)
In the above embodiment, the local feature amount calculation unit 202 has been described using a color histogram as an example of an image feature amount, but the present invention is not limited to this. The local feature amount calculation unit 202 may calculate a gradient histogram in a small region. A method for calculating the gradient histogram in the local feature amount calculation unit 202 in this case will be described with reference to FIG. FIG. 15 is a schematic diagram illustrating a gradient histogram calculation method.

First, the local feature amount calculation unit 202 calculates a gradient vector G (x, y) = (G _x , G _y ) for the luminance component for each pixel in the small region. Next, the angle θ _Q of the gradient vector quantized by the quantization step size _Q is acquired from the calculated gradient vector G (x, y) of each pixel (formula (20-8)).

Subsequently, the appearance frequency of the angle θ _Q of the quantized gradient vector is counted, and a gradient histogram H _θ (x, y) is calculated by normalizing each element of the histogram shown in FIG. 15A with the area of the small region. . Subsequently, from the calculated gradient histogram H _θ (x, y), the angle θ _{m of the} gradient vector having the maximum appearance frequency is acquired, and the gradient vector having the maximum appearance frequency with the gradient histogram H _θ (x, y) is acquired. based on the angle .theta.m, calculated normalized gradient histogram H _.theta.m the angle theta _m shown in FIG. 15 (B) as the origin, the (x, y). By using a luminance gradient histogram as a local feature amount, it is robust against color changes such as illumination changes. In addition to the color histogram and the gradient histogram, the local feature amount calculation unit 202 may use a wavelet feature amount, a Harr-like feature amount, an Edgelet feature amount, an EOH feature amount, and an HOG feature amount in addition to the color histogram and the gradient histogram. . Each image feature amount may be used in combination with a different image feature amount.

(Modification of Object Extraction Device 1 (Object Extraction Device 1a))
In the above embodiment, the object extraction device 1 in FIG. 1 includes, in the reference image, a representative local feature amount that represents the image feature amount of the extraction target object from the silhouette image of the extraction target object and the image signal of the reference image, and a small region. Is used to calculate whether or not the object region is an object region. However, the present invention is not limited to this. For example, obtained from the object region determination unit 40, the time _{t 1} of the silhouette images showing the object area S _{(t 1)} and image F _{(t 1),} and fed back to the next time _(t 1 +1) at time _{t 1} , A reference image, and a silhouette image of the reference image. Even when the local feature amount of the object region at every time changes minutely, it is possible to accurately track and extract.

  A part of the object extraction device 1 in the above-described embodiment may be realized by a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” is a computer system built in the object extraction apparatus 1 and includes hardware such as an OS and peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  Moreover, you may implement | achieve part or all of the object extraction apparatus 1 in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each functional block of the stereoscopic image generating apparatus 1 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

DESCRIPTION OF SYMBOLS 1 ... Object extraction apparatus, 20 ... Identification feature-value calculation part, 30 ... Object area | region estimation part, 40 ... Object area | region determination part, 201, 401 ... Feature-value calculation range setting part, 202, 402 ... Local feature-value calculation part, 203 ... representative local feature amount calculation unit, 204 ... identification function calculation unit, 301 ... feature point calculation unit, 302 ... corresponding point calculation unit, 303 ... transformation matrix calculation unit, 304 ... estimated silhouette image generation unit, 403 ... object region identification unit .

Claims (9)

An object extraction device that extracts an object region of interest from a time-series image,
An identification feature amount calculating unit that calculates an identification feature amount for identifying the object region and the object peripheral region from the image of the object region on the reference image in the time-series image and the image of the peripheral region;
An object region estimation unit that estimates an object region on the processing target image by correspondence between a feature point in the object region on the reference image in the time-series image and a feature point of the processing target image in the time-series image;
The feature amount of the image of the estimated object region and its peripheral region is calculated, and the object region in the estimated processing target image is corrected and confirmed by comparing the calculated feature amount with the calculated identification feature amount. And an object area determination unit. The identification feature amount calculation unit sets a feature amount calculation range for calculating an image feature amount for identifying an object region and an object peripheral region based on a silhouette image indicating the object region in the reference image. A range setting section;
An image feature amount calculation unit that calculates the image feature amount for each small region of a predetermined size in a set feature amount calculation range;
A representative image feature amount calculation unit that clusters image feature amounts on the object region out of the image feature amounts and calculates K different representative image feature amounts expressing the image feature amount of the object region;
For each small region, a similarity vector calculation unit that calculates a similarity vector between the image feature amount and the representative image feature amount from the image feature amount of the small region and K different representative image feature amounts;
An identification function calculation unit for calculating an identification function representing a class boundary as to whether or not a small area belongs to the object area in the feature amount space from the calculated similarity vector on the object area and the similarity vector of the object peripheral area; The object extraction apparatus according to claim 1, wherein the K different representative image feature quantities and the discrimination function are output as the discrimination feature quantities. The object area determination unit
A feature amount calculation range setting unit for setting a feature amount calculation range for calculating an image feature amount based on an estimated silhouette image indicating the estimated object region;
An image feature amount calculation unit that calculates an image feature amount for each small region of a predetermined size in the set feature amount calculation range;
A similarity vector calculating unit that calculates the similarity vector from the image feature amount of the small region and K different representative image feature amounts in the reference image for each small region;
Based on the calculated similarity vector and the discrimination function, each small area is identified as belonging to the object area in the feature amount space, and a set of small areas determined to be included in the object area is determined. The object extraction apparatus according to claim 2, wherein the object area is determined as an object area on the processing target image. The object region estimation unit
A feature point calculation unit for calculating a feature point in the object region on the reference image from the reference image and a silhouette image indicating the object region on the reference image;
A process corresponding to a feature point in the object area on the reference image from the calculated feature point in the object area on the reference image, a silhouette image indicating the object area on the reference image, and the processing target image A corresponding point calculation unit for calculating feature points on the target image;
From the feature points in the object region on the reference image and the feature points on the processing target image corresponding to the feature points, the processing target image corresponding to the feature points in the object region on the reference image A transformation matrix calculation unit for calculating a transformation matrix to be projected onto the upper feature point,
From each of the transformation matrix and the silhouette image indicating the object region on the reference image, each pixel on the processing target image is multiplied by an inverse matrix of the transformation matrix to determine whether it is included in the object region on the reference image. The object extraction apparatus according to claim 1, wherein a set of pixels determined to be included in the object area is estimated as an object area on the processing target image. An object extraction method for extracting an object region of interest from a time series image,
An identification feature amount calculating step for calculating an identification feature amount for identifying the object region and the object peripheral region from the image of the object region on the reference image in the time-series image and the image of the peripheral region;
An object region estimation step for estimating an object region on the processing target image by correspondence between a feature point in the object region on the reference image in the time-series image and a feature point of the processing target image in the time-series image;
The feature amount of the image of the estimated object region and its peripheral region is calculated, and the object region in the estimated processing target image is corrected and confirmed by comparing the calculated feature amount with the calculated identification feature amount. And an object region determination step. The identification feature amount calculating step sets a feature amount calculation range for calculating an image feature amount for identifying an object region and an object peripheral region based on a silhouette image indicating the object region in the reference image. A range setting step;
An image feature amount calculating step for calculating the image feature amount for each small region of a predetermined size in the set feature amount calculation range;
A representative image feature amount calculating step of clustering image feature amounts on the object region out of the image feature amounts and calculating K different representative image feature amounts expressing the image feature amount of the object region;
A similarity vector calculation step for calculating a similarity vector between the image feature quantity and the representative image feature quantity from the image feature quantity of the small area and K different representative image feature quantities for each small area;
An identification function calculating step for calculating an identification function representing a class boundary as to whether or not a small area belongs to the object area in the feature amount space from the calculated similarity vector on the object area and the similarity vector of the object peripheral area; ,
6. The object extraction method according to claim 5, further comprising: outputting the K different representative image feature amounts and the discrimination function as the discrimination feature amounts. The object region determination step includes
A feature amount calculation range setting step for setting a feature amount calculation range for calculating an image feature amount based on the estimated silhouette image indicating the estimated object region;
An image feature amount calculating step for calculating an image feature amount for each small region of a predetermined size in the set feature amount calculation range;
For each small region, a similarity vector calculating step for calculating the similarity vector from the image feature amount of the small region and K different representative image feature amounts in the reference image;
Based on the calculated similarity vector and the discrimination function, each small area is identified as belonging to the object area in the feature amount space, and a set of small areas determined to be included in the object area is determined. The object extracting method according to claim 6, further comprising: determining an object region on the processing target image. The object region estimation step includes:
A feature point calculating step of calculating a feature point in the object area on the reference image from the reference image and a silhouette image indicating the object area on the reference image;
A process corresponding to a feature point in the object area on the reference image from the calculated feature point in the object area on the reference image, a silhouette image indicating the object area on the reference image, and the processing target image A corresponding point calculating step for calculating feature points on the target image;
From the feature points in the object region on the reference image and the feature points on the processing target image corresponding to the feature points, the processing target image corresponding to the feature points in the object region on the reference image A transformation matrix calculating step for calculating a transformation matrix to be projected onto the upper feature point;
From each of the transformation matrix and the silhouette image indicating the object region on the reference image, each pixel on the processing target image is multiplied by an inverse matrix of the transformation matrix to determine whether it is included in the object region on the reference image. And a step of estimating a set of pixels determined to be included in the object area as an object area on the processing target image. Extraction method.   The object extraction program for functioning a computer as each process part of the object extraction apparatus of any one of Claims 1-4.

Images