JP2016099980A - Image segmentation method, apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately segment an object area of an extraction target even when an image not including the extraction target is mixed in an input image group or when an image having a plurality of extraction target objects reflected thereon is mixed in one image.SOLUTION: An image segmentation device 10 extracts one or more object area candidates representing an extraction target object from images included in an image group; estimates an appearance model for each of the extracted object area candidates; estimates association of pixels between each of the extracted object area candidates and another similar object area; updates the object area candidates on the basis of the object area candidates, appearance models, and association; repeatedly performs estimation of the appearance models, estimation of association, and update of the object area candidates until predetermined convergence conditions are satisfied; and outputs the object area candidates of the images included in the image group as object areas.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image segmentation method, apparatus, and program, and in particular, image segmentation that automatically segments an object included in an input image group using an image group including a large percentage of objects to be extracted as an input. A method, an apparatus, and a program.

  When an image in which an object is reflected is given, the region in which the object is reflected is extracted from the image, that is, segmentation is performed by, for example, an image-based object search dictionary, image annotation learning data, and image editing material. It is an important element processing for constructing etc. In general, a large amount of images are required to construct the dictionary, learning data, materials, and the like, but it is expensive to manually segment these images one by one. In order to construct the above dictionary, learning data, materials, etc. at a low cost and realize such a large-scale image application service as described above, a technology for accurately segmenting an enormous amount of images without human intervention is indispensable. It can be said that there is.

  In order to automatically segment an enormous amount of images, structured information on the web is considered effective. For example, by using keyword image search provided by various search engines and image sharing sites, a plurality of images in which objects related to keywords (for example, “airplane”, “car”, etc.) are reflected can be easily obtained. is there. If an image group obtained as a result of searching with a certain keyword is input, and an object corresponding to the keyword included therein can be automatically segmented, an object region to which a label (that is, a search keyword) is assigned automatically. It can be acquired.

  Many of the image groups obtained as a result of the keyword search include an object region corresponding to the keyword. As a known technique for automatically segmenting an image group at the same time by using information that an object common to the input image group is reflected, Non-Patent Document 1 discloses a label of a similar adjacent pixel of each image. A term that evaluates the consistency of (foreground or background label) and a feature amount extracted from a pixel set having a foreground label in an image to a feature amount extracted from a pixel set having a background label. A method is disclosed for assigning labels to pixels in each image by minimizing an energy function comprised of terms to be evaluated. The technique disclosed in Non-Patent Document 1 is based on the premise that objects to be extracted are reflected in all images of the input image group in order to solve the label allocation problem.

  On the other hand, in Non-Patent Document 2, a pixel pair is associated by SIFT Flow with a pair of particularly similar images in the input image group, and each pixel is defined based on the obtained association. By solving an energy minimization problem that includes a term that evaluates the object-likeness of an object, it is possible to extract the object to be extracted while excluding objects other than the object to be extracted that are partially mixed in the input image group A method is disclosed. The technique disclosed in Non-Patent Document 2 is more advantageous than the technique of Non-Patent Document 1 in that the input image group can include an image in which an object to be extracted is not reflected at all. It can be said that this is a technique capable of accurately extracting an object region to be extracted from an input image group.

A. Joulin et al., Multi-Class Cosegmentation, in Proc. CVPR, 2012. Internet (URL: http://www.di.ens.fr/~fbach/joulin_cvpr2012.pdf) M. Rubinstein et al., Unsupervised Joint Object Discovery and Segmentation in Internet Images, in Proc. CVPR, 2013. Internet (URL: http://people.csail.mit.edu/mrub/papers/ObjectDiscovery-cvpr13.pdf)

  However, the method disclosed in Non-Patent Document 1 has the following problems. Since part of the image group obtained as a keyword search result often includes an image in which the object to be extracted is not reflected at all, all of the input image group assumed by Non-Patent Document 1 is assumed. Generally, the condition that the object to be extracted is shown in the image is not satisfied. For this reason, the image group obtained as a result of the keyword search is used as an input, and the result of segmentation using the technique disclosed in Non-Patent Document 1 has a problem that the original object region cannot be correctly captured and accuracy is low. there were.

  Further, the association processing between pixels that realizes the exclusion of objects other than the extraction target in Non-Patent Document 2 is executed on the assumption that one-to-one pixel correspondence is possible between image pairs. That is, the assumption is made that the number of objects to be extracted is the same between image pairs.

  Also, a part of the image group obtained as a result of the keyword search includes an image in which a plurality of objects to be extracted are included in one image. As general as it is. For this reason, the hypothesis disclosed in Non-Patent Document 2 that a one-to-one correspondence between pixels is possible does not generally hold, and in particular, an image in which a plurality of objects to be extracted are reflected in one sheet. When a large number of image groups are input, the result of segmentation using the technique disclosed in Non-Patent Document 2 has a problem in that the original object region cannot be correctly captured and accuracy is low.

  Furthermore, both methods of Non-Patent Document 1 and Non-Patent Document 2 have a problem that when a plurality of objects to be extracted appear in an image, there is no means for automatically separating them. It was. This is because the method disclosed in Non-Patent Document 1 and Non-Patent Document 2 determines only whether or not each pixel of each image included in the input image group is an object to be extracted. When there are a plurality of objects in the image, it does not have a function of determining which of them belongs.

  In the dictionary and learning data for realizing the above image application service, it is common to use each object as a unit of data. However, in the existing known technology, a plurality of objects included in an image are spatially separated. If so, a post-treatment is required to separate them. If individual objects are clearly separated spatially in the image, it is not difficult to automatically separate them, but if multiple objects are reflected spatially, they are automatically separated. It was very difficult to separate, and there was a problem that work with high cost was inevitable.

  As described above, in the existing known technique for simultaneously segmenting the object region corresponding to the keyword by using the image group obtained as a result of the keyword search, the “object to be extracted” which is a feature of the image group obtained as a result of the keyword search. Each feature included in the image is captured simultaneously with the feature that `` images that do not contain areas or objects that include objects other than the extraction target are mixed '' and the feature that `` images that include multiple objects to be extracted are mixed '' There is a problem that the object to be extracted cannot be separated and segmented with high accuracy.

  The present invention has been made in view of the above circumstances, and an input image group includes a mixture of images that do not include an object region to be extracted, or a plurality of objects to be extracted appear in one image. It is an object of the present invention to provide an image segmentation method, apparatus, and program capable of accurately segmenting an object region to be extracted even when there are mixed images.

  In order to achieve the above object, an image segmentation method according to the present invention includes an object region extraction unit, an appearance model estimation unit, an association estimation unit, an object region update unit, a convergence determination unit, and an object region output unit. In the image segmentation method, the object region extraction unit extracts one or more object region candidates representing an extraction target object from each image included in the image group, and the appearance model estimation unit includes the Estimating an appearance model for each of the object region candidates based on the object region candidate extracted by the object region extraction unit or the object region candidate updated by the object region update unit; and the association estimation unit Is the object region candidate extracted by the object region extraction unit or the object region update. For each of the object region candidates updated by the above, a step of estimating a pixel correspondence with another similar object region candidate, and the object region update unit is extracted by the object region extraction unit Based on the object region candidate or the object region candidate updated by the object region update unit, the appearance model estimated by the appearance model estimation unit, and the association estimated by the association estimation unit, the image group Updating the object region candidate of each image included in the image, and the convergence determination unit until the predetermined convergence condition is satisfied, the estimation by the appearance model estimation unit, the estimation by the association estimation unit, and the object A step of repeatedly performing the update by the region update unit, and the object region output unit includes each image included in the image group. Comprising a step wherein outputting the object region candidate object region of the.

  The step of extracting the object region candidates by the object region extraction unit may extract the object region candidates so that there are no overlapping regions in the image.

  Further, the step of estimating the appearance model by the appearance model estimation unit estimates the appearance model for each object region candidate of each image extracted by the object region extraction unit, and determines each object region candidate of each image. The appearance model may be estimated for a background region composed of a pixel set not included in any of them.

  Further, the step of estimating the association by the association estimation unit evaluates similarity of image feature amounts between pixels associated with other similar object region candidates for each of the object region candidates. And another term that evaluates the smoothness of the movement vector associated with the pixel association with respect to neighboring pixels in the object region candidate. You may make it estimate the correlation of the pixel between area | region candidates.

  Further, the step of updating the object region candidate by the object region update unit is a term that evaluates the likelihood of the extraction target object with respect to each pixel of each image included in the image group, and is extracted by the object region extraction unit. The term for evaluating spatial proximity from the object region candidate or the object region candidate updated by the object region update unit, and the appearance model estimation unit for each pixel of each image included in the image group A term that evaluates the estimated contribution to the appearance model, a term that evaluates similarity of image feature amounts between pixels associated by the association estimation unit, and an object region of pixels adjacent in the image A term that evaluates the smoothness of a label indicating whether or not the candidate is included in a candidate, and the label between the pixels associated by the association estimation unit A term evaluating the transmural properties, by optimizing the objective function including, may perform updating of the object region candidate of the images included in the image group.

  The image segmentation device of the present invention includes an object region extraction unit that extracts one or more object region candidates representing an extraction target object from each image included in an image group, and the object region extracted by the object region extraction unit. An appearance model estimation unit that estimates an appearance model for each of the object region candidates based on the candidate or updated object region candidate, and the object region candidate or the updated object extracted by the object region extraction unit For each of the area candidates, an association estimation unit that estimates pixel association with another similar object region candidate, and the object region candidate extracted by the object region extraction unit or the updated object region Candidates, the appearance model estimated by the appearance model estimation unit, and the correspondence estimation unit An object region update unit that updates the object region candidate of each image included in the image group based on the association, and an estimation by the appearance model estimation unit until a predetermined convergence condition is satisfied, the association A convergence determination unit that repeatedly performs estimation by the estimation unit and update by the object region update unit; and an object region output unit that outputs the object region candidate of each image included in the image group as an object region.

  The image segmentation program of the present invention is an image segmentation program for causing a computer to execute each step of the image segmentation method according to any one of claims 1 to 5.

  As described above, according to the image segmentation method, apparatus, and program of the present invention, images that do not include an extraction target are mixed in an input image group, or a plurality of extraction target objects exist in one image. Even in the case where captured images are mixed, there is an effect that the object region to be extracted can be accurately segmented.

It is a block diagram which shows the functional structural example of the image segmentation apparatus which concerns on this Embodiment. It is a flowchart which shows the image segmentation routine in this Embodiment. It is a figure which shows an example of the several object area | region extracted from the image. It is a figure which shows the example which removed the object area | region with a large ratio of the overlapping area | region among the several object area | regions extracted from the image. It is a figure which shows the example which labeled the pixel of the overlapping area | region to any object area | region. It is a figure which shows an example of the image group of the output image containing only the input image group and the object area | region extracted from the said image group. It is a figure which shows one object area | region extracted from the image containing the several object area | region in which some overlapped in the prior art. It is a figure which shows two object area | regions extracted from the image containing several object area | regions in which one part overlapped in this Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  In this embodiment, as an example, an image group obtained as a result of keyword search is used as an input, and object regions corresponding to the keyword included in the input image group are automatically segmented.

<System configuration>

  As shown in FIG. 1, the image segmentation device 10 according to the present embodiment includes an object region extraction unit 12, an appearance model estimation unit 14, an association estimation unit 16, an object region update unit 18, a convergence determination unit 20, and an object An area output unit 22 is provided.

  The object region extraction unit 12 extracts one or more object regions from each image included in the input image group that has been input. The input image group input to the object region extraction unit 12 is an image group obtained by inputting a keyword representing an object on an image search site or the like and searching for an image including the object corresponding to the keyword. Many of the images included in the image group include an object corresponding to the keyword, but not all images necessarily include an extraction target object corresponding to the keyword.

  The appearance model estimation unit 14 estimates an appearance model for each object region candidate based on the object region candidate extracted by the object region extraction unit 12 or the object region candidate updated by the object region update unit 18.

  The association estimation unit 16 associates pixels with other similar object regions for each of the object region candidates extracted by the object region extraction unit 12 or the object region candidates updated by the object region update unit 18. Is estimated.

  The object region update unit 18 includes the object region candidate extracted by the object region extraction unit 12 or the object region candidate updated by the object region update unit 18, the appearance model estimated by the appearance model estimation unit 14, and the association estimation unit. The object region candidate is updated based on the association estimated by 16.

  The convergence determination unit 20 determines whether or not a predetermined convergence condition is satisfied, and repeatedly performs appearance model estimation, association estimation, and object region candidate update until it is determined that the predetermined convergence condition is satisfied. .

  When the convergence determination unit 20 determines that the predetermined convergence condition is satisfied, the object region output unit 22 outputs the object region candidate of each image included in the image group as the object region. The object region output unit 22 transmits, for example, an image including only the extracted object region to another terminal device on the network, outputs it to a storage device, or outputs it to a display device.

  The object region extraction unit 12, appearance model estimation unit 14, association estimation unit 16, object region update unit 18, convergence determination unit 20, and object region output unit 22 are, for example, a CPU (Central Processing Unit) and a RAM (Random Access). And a ROM (Read Only Memory) that stores an image segmentation program for executing an image segmentation routine described later. Note that a nonvolatile memory may be used instead of the ROM. The computer constituting the image segmentation apparatus 10 may include a storage unit such as a hard disk drive, a communication interface, and the like. A program executed by the CPU may be stored in the hard disk drive. When the CPU reads and executes a program stored in a storage unit such as a ROM or a hard disk, functions described below are realized by cooperating the hardware and the program.

<Operation of image segmentation device>

  Next, an image segmentation routine executed in the image segmentation apparatus 10 according to the present embodiment will be described with reference to FIG.

  In step S <b> 100, the object region extraction unit 12 inputs the input image group.

Enter. N is the number of images. Each image may be used in the processing after step S102 with the original size, or may be resized. When resizing each image, the image

For example, each image is resized to a scale of l _max / max (height _i , width _i ), where the original height is height _i , the width is width _i , and l _max is a constant.

Next, the object region extraction unit 12 extracts object region candidates that appear in each image I _i included in the input image group. The object region candidates extracted here are extracted so that there are no overlapping regions in the image. Hereinafter, a method for extracting object region candidates so as not to overlap each other in an image will be specifically described.

  In the following, a case where a rectangular region is extracted as an object region candidate will be described, but the shape of the extracted region is not limited to a rectangle.

First, using a known technique for evaluating object-likeness that does not depend on an object category for an arbitrary rectangular region in an image, N _{proposals of} rectangular regions having a high object-likeness score are extracted.

N _proposal is a parameter, and a predetermined value is set. FIG. 3 shows an example of a rectangular area extracted from the image I _i , that is, an object area candidate when N _proposal = 5 as an example. As shown in FIG. 3, the object region candidate 30 ₁ overlaps the object region candidate 30 _2. Also, the object region candidate 30 ₂ overlaps the object region candidate 30 ₁ and the object region candidate 30 _3. The object region candidate 30 ₃ overlaps with the object region candidate 30 ₂ , the object region candidate 30 ₄ , and the object region candidate 30 ₅ . Also, the object region candidate 30 ₄ overlaps the object region candidate 30 ₃ and the object region candidate 30 _5. Also, the object region candidate 30 ₅ overlaps the object region candidate 30 ₃ and the object region candidate 30 _4.

In the following, object region candidates obtained by the object region extraction process for the image I _i are shown.

, The object-like score corresponding to each object region candidate

And Object region candidate

When pixel x is included in the object region candidate

Otherwise, it is a binary mask of the same size as the image that takes 0. As a known technique for evaluating object-likeness that does not depend on an object category for an arbitrary rectangular area in an image, for example, a method disclosed in Reference Document 1 below can be used.

(Reference 1) M.M. Cheng et al., BING: Binarized Normed Gradients for Objectness Estimation at 300fps, in Proc. CVPR, 2014.

Next, among the extracted object region candidates, object region candidates having a large proportion of regions overlapping with other object region candidates are removed using Non Maximum Suppression (NMS). FIG. 4 shows an example in which an object region candidate having a large proportion of regions overlapping with other object region candidates is removed using NMS. As shown in FIG. 4, object region candidates 30 ₂ , 30 ₃ are left, and object region candidates 30 ₁ , 30 ₄ , 30 ₅ are removed. As shown in FIG. 3, the object region candidates 30 ₁ , 30 ₄ , and 30 ₅ have a large ratio of regions that overlap with other object region candidates. As shown in FIG. 4, the object region candidates 30 ₂ and 30 ₃ are partially overlapped by a region 32. In this way, not all object region candidates having overlapping regions are removed.

  Below, the object region candidate set remaining as a result of NMS

And

Is the number of object region candidates extracted from the image I _i . Object region candidates remaining as a result of NMS may overlap each other even though the overlapping region is small. Therefore, in order to avoid overlapping of regions, for each pixel belonging to two or more object region candidates included in the object region candidate set shown in Equation 7, the color, position, and label consistency with adjacent pixels are consistent. By evaluating the property, assignment processing to any object region candidate included in the object region candidate set shown in the above equation 7 is performed. The above allocation process is performed by, for example, an object region candidate set that minimizes the energy function expressed by the following equation (1).

It is obtained by seeking.

... (1)

here,

Is a set of pixels included in at least one object region candidate among the object region candidates included in the object region candidate set shown in Equation 7, and N _x is a set of neighboring pixels of the pixel x (for example, the pixel x A set of pixels that exist in the top, bottom, left, and right).

  The first term on the right side of the above (1) is a term for evaluating the position of the pixel x, and can be defined as, for example, the following equation (2).

... (2)

here,

Is the object region candidate

The set of pixels contained in

Is the coordinate of pixel x. Further, κ _position is a parameter, and a predetermined value is set.

  The second term on the right side of equation (1) above is the object region candidate

This term is a term for evaluating the contribution of the pixel x to the appearance model, that is, a term for evaluating the color of the pixel x, and can be defined, for example, by the following equation (3).

... (3)

here,

Is an appearance model corresponding to the object region candidate shown by the above-mentioned equation 14

Luminance value of pixel x with respect to

The log-likelihood, kappa _color is a parameter, and sets the predetermined value. The appearance model shown in the above equation 19 can be constructed using, for example, a Gaussian Mixture Model.

  The third term on the right side of the above equation (1) is a term for evaluating the consistency of the label indicating whether or not adjacent pixels x and x ′ having similar luminance values are included in the object region candidate. 4) can be defined as:

... (4)

Here, κ _int is a parameter, and a predetermined value is set. [] Is an indicator function that becomes 1 when the condition in [] is satisfied and becomes 0 when the condition is not satisfied.

Is 0 when the pixel x is not included in any of the object region candidates included in the object region candidate set represented by Equation 9, and is j when it is included in the object region candidate R _ij .

Through the above processing, object region candidate sets that do not overlap with each other are obtained for each image I _i as shown in Equation 9 above. For example, as shown in FIG. 4, a pixel in a region 32 where a part of the object region candidates 30 ₂ and 30 ₃ partially overlaps is labeled as one of the object region candidate 30 ₂ and the object region candidate 30 ₃ as shown in FIG. Will be. For this reason, each object area candidate included in the object area candidate sets that do not overlap each other as shown in the above equation 9 is not necessarily a rectangular area.

  In step S102, the appearance model estimation unit 14 estimates an appearance model based on the object region candidate set of each image shown in the above equation 9 obtained in step S102. Here, the appearance model is for each object region candidate constituting the object region candidate set extracted from each image, and a region composed of a pixel set not included in any object region candidate, that is, a background region. presume.

In the following, the appearance model set estimated for the image I _i is

And Where the appearance model

Is the object region candidate

And a set of pixels included in the object region candidate R _ij

Can be estimated from The appearance model H _i0 is an appearance model estimated from a region that is not included in any of the object region candidates constituting the object region candidate set represented by _Equation 9 above, that is, a background region, and the object represented by Equation 9 above. It can be estimated from a pixel set that is not included in any object region candidate that constitutes the region candidate set. The appearance model can be constructed using, for example, a Gaussian Mixture Model.

  In step S104, the association estimation unit 16 performs pixel-to-pixel comparison between each object region candidate of each image obtained in step S102 and a particularly similar object region candidate among all object region candidates included in each image. Estimate close correspondence. Here, the close association between the pixels between the similar object region candidates is associated with the association between the term for evaluating the similarity of the image feature amount between the associated pixels and the neighboring pixels in the object region candidate. It is calculated by solving an optimization function composed of a term for evaluating the smoothness of the movement vector. For example, the following processing is executed.

  First, image feature values are extracted from each object region candidate, and each object region candidate is extracted.

The SIFT Flow disclosed in Reference Document 2 below is calculated between | N _R | object region candidates that are particularly similar with respect to the extracted image feature quantity. | N _R | is a parameter, and a predetermined value is set. As a method for extracting the image feature amount from each object region candidate, for example, the GIST feature amount disclosed in Reference Document 3 below can be used.

(Reference 2) C. Liu et al., SIFT Flow: Dense Correspondence across the Scenes and its Applications, TPAMI, 33 (5), 2011.

Internet (URL: http://people.csail.mit.edu/celiu/SIFTflow/SIFTflow.pdf)

(Reference 3) A. Oliva et al., Modeling the Spape of the Scene: A Holistic Representation of the Spatial Envelope, IJCV, 42 (3), pp145-175, 2001.

Internet (URL: http://cvcl.mit.edu/Papers/IJCV01-Oliva-Torralba.pdf)

  Below, the correspondence set obtained by the correspondence estimation process is

And w _{RR ′} (x) is associated with pixel x∈R

Pixels within.

  In step S106, the object region candidate update unit 18 obtains the object region candidate set as shown in the above equation 9 obtained in step S102 or the object region candidate set updated by executing the previous step S106, the appearance of step S102. The object region candidate is updated using the appearance model set obtained by the model estimation process and the result of the association set obtained by the association estimation process in step S104.

  Here, the update of the object region candidate includes the term for evaluating the object-likeness regarding each pixel of each image included in the input image group, and the object region candidate set as shown in the equation 9 obtained in step S102 or A term for evaluating the spatial proximity of object region candidates included in the object region candidate set updated by executing step S106 last time, and a contribution to the appearance model obtained by the appearance model estimation process in step S102 A term to be evaluated, a term to evaluate the similarity of image feature amounts between pixels associated in the association processing in step S104, a term to evaluate the smoothness of labels of adjacent pixels in the image, and step S104 A term that evaluates the consistency of the labels of the pixels associated with the mapping process, and by optimizing the objective function . Such an objective function can be defined as the following equation (5), for example.

... (5)

Here, the first term on the right side of the above equation (5) is an energy term defined for each pixel of each image, and is defined as the following equation (6).

... (6)

Here, β is a parameter, and a predetermined value is set. E _obj (x) in the above equation (6) is a term for evaluating the object-likeness of the pixel x. For example, the object region candidate represented by the above equation 3 obtained by the object region extraction process in step S100 and the above equation 4 Using the object-like score corresponding to each object region candidate shown in (5), it can be defined as the following equation (7).

... (7)

Here, λ _obj is a parameter, and a predetermined value is set. Of the above formula (6)

Is a term that evaluates the spatial proximity from the previously updated object region candidate for the position of the pixel x, and can be defined, for example, by the following equation (8).

... (8)

Here, λ _position is a parameter, and a predetermined value is set. Of the above formula (6)

Is a term for evaluating the similarity of image feature amounts between pixels associated in the association processing in step S104, and can be defined as, for example, the following equation (9).

... (9)

Here, α and λ _match are parameters, and predetermined values are set. d _x is a local image feature amount defined for the pixel x, and for example, a SIFT descriptor can be used.

  The second term on the right side of the above equation (5) is a term for evaluating the contribution of each pixel of each image to the appearance model, and can be defined as the following equation (10), for example.

... (10)

Here, logp (c _x | H _ij ) is a logarithmic likelihood of the luminance value of the pixel x expressed by the above equation 19 with respect to the appearance model H _ij corresponding to the object region candidate R _ij . Further, λ _color is a parameter, and a predetermined value is set.

  The third term on the right side of the above equation (5) is a term for evaluating the smoothness of the labels of adjacent pixels in the image, and can be defined as the following equation (11).

(11)

Here, λ _int is a parameter, and a predetermined value is set.

  The fourth term on the right side of the above equation (5) is a term for evaluating the smoothness of the label between the pixels associated in the association estimation process in step S104, and can be defined as the following equation (12). it can.

(12)

Here, λ _ext is a parameter, and a predetermined value is set. In addition, B _i (x) is

0 for 0, 1 otherwise.

The above equation (5) can be solved as follows, for example. First, an image I _i is selected, and all object region candidates included in other images are fixed. In this case, the above equation (5) is a function corresponding to the image I _i and having the object region candidate set shown in the above equation 9 as a variable, which can be solved by using a known algorithm such as loop belief propagation. . By executing this process for each image, the object region candidate set shown in Equation 9 can be updated.

  In step S108, the convergence determination unit 20 determines whether the result of the object region update process in step S108 satisfies the convergence condition. As the convergence condition, for example, the convergence condition may be that the amount of change in the value of the objective function defined in the object region update process is equal to or less than a certain value, and the number of times of repeating a series of processes from step S102 to S106 may be set in advance. The convergence condition may be that a series of processing from steps S102 to S106 is repeated a predetermined number of times. When the convergence determination unit 20 determines that the convergence condition is satisfied, the process proceeds to step S108, and when it is determined that the convergence condition is not satisfied, the process proceeds to step S102.

  In step S108, the object area output unit 22 outputs an image including the object area obtained by the object area update process in step S106. FIG. 6 shows the input image group 40 and the output image group 50 including only the object region obtained by executing the processes of steps S100 to S108 on the input image group 40. As shown in FIG. 6, the input image group 40 is an image group in which the extraction target is an airplane.

As shown in FIG. 6, the input image 42 included in the input image group 40 includes one airplane, and an output image 52 including only one airplane is obtained from the input image 42. It also includes a two planes opposed to the input image 44, from the input image 44, an output image 54 1 containing only airplane _right, obtained output image ₅₄₂ including only airplane left It is done. The input image 46 includes a plurality of large and small airplanes, and output images 56 ₁ , 56 ₂ , 56 ₃ ,... Including only the respective airplanes are obtained from the input image 46. Since the input image 48 does not include one airplane, an output image from which the airplane is extracted cannot be obtained from the input image 48.

Here, for example, as shown in FIG. 7, if it contains some two object region that overlaps candidate ₆₂ 1, 62 ₂ to the input image 60, in the prior art, the two object region candidate ₆₂ 1, 62 ₂ Cannot be segmented with high accuracy, and the output image 70 is an output image including only one object region candidate 72.

In contrast, in the present embodiment, as shown in FIG. 8, the two object region candidate ₆₂ 1 partially overlaps the input image 60, 62 _{if 2} is contained, two object region candidate ₆₂ 1, 62 ₂ can be accurately segmented, and the output image 70 is an output image including _two object region candidates 72 ₁ , 722 corresponding to the two object region candidates 62 ₁ , 62 ₂ .

  In this way, even when an input image group includes a mixture of images in which a plurality of objects to be extracted are included in one image or an image that does not include an object region to be extracted, The object region to be extracted can be segmented with high accuracy.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  Further, the image segmentation apparatus 10 described above has a computer system therein, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. .

  Further, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium such as a CD-ROM or a memory card. It is.

DESCRIPTION OF SYMBOLS 10 Image segmentation apparatus 12 Object area extraction part 14 Appearance model estimation part 16 Association estimation part 18 Object area update part 20 Convergence determination part 22 Object area output part

Claims (7)

An image segmentation method in an image segmentation device including an object region extraction unit, an appearance model estimation unit, an association estimation unit, an object region update unit, a convergence determination unit, and an object region output unit,
The object region extraction unit extracting one or more object region candidates representing an extraction target object from each image included in the image group; and
The appearance model estimation unit estimates an appearance model for each of the object region candidates based on the object region candidate extracted by the object region extraction unit or the object region candidate updated by the object region update unit. Steps,
For each of the object region candidate extracted by the object region extraction unit or the object region candidate updated by the object region update unit, the association estimation unit is a pixel between other similar object region candidates. Estimating the correspondence of
The object region update unit is the object region candidate extracted by the object region extraction unit or the object region candidate updated by the object region update unit, the appearance model estimated by the appearance model estimation unit, and Updating the object region candidate of each image included in the image group based on the association estimated by the association estimation unit;
The convergence determination unit repeatedly performs estimation by the appearance model estimation unit, estimation by the association estimation unit, and update by the object region update unit until a predetermined convergence condition is satisfied;
The object region output unit outputting the object region candidate of each image included in the image group as an object region;
An image segmentation method comprising: The image segmentation method according to claim 1, wherein the step of extracting the object region candidate by the object region extraction unit extracts the object region candidate so that there are no overlapping regions in the image. The step of estimating the appearance model by the appearance model estimation unit estimates the appearance model for each object region candidate of each image extracted by the object region extraction unit, and to which of the object region candidates of each image The image segmentation method according to claim 1, wherein the appearance model is estimated for a background region including a pixel set that is not included. The step of estimating the association by the association estimation unit includes:
For each of the object region candidates,
A term for evaluating the similarity of image feature amounts between pixels associated with other similar object region candidates;
A term for evaluating the smoothness of a movement vector associated with the pixel association with respect to neighboring pixels in the object region candidate;
The image segmentation method according to any one of claims 1 to 3, wherein a pixel association with another similar object region candidate is estimated by solving an optimization function composed of: The step of updating the object region candidate by the object region updating unit includes:
A term for evaluating the likelihood of the extraction target object regarding each pixel of each image included in the image group;
A term for evaluating spatial proximity from the object region candidate extracted by the object region extraction unit or the object region candidate updated by the object region update unit;
A term for evaluating a contribution to the appearance model estimated by the appearance model estimation unit for each pixel of each image included in the image group;
A term for evaluating the similarity of image feature amounts between pixels associated by the association estimation unit;
A term for evaluating the smoothness of a label indicating whether or not an adjacent pixel in an image is included in a candidate object region;
A term for evaluating the consistency of the label between pixels associated by the correspondence estimation unit;
The image segmentation method according to claim 1, wherein the object region candidate of each image included in the image group is updated by optimizing an objective function including the image function. An object region extraction unit that extracts one or more object region candidates representing an extraction target object from each image included in the image group;
An appearance model estimation unit that estimates an appearance model for each of the object region candidates based on the object region candidate extracted by the object region extraction unit or the updated object region candidate;
An association estimation unit that estimates pixel association with other similar object region candidates for each of the object region candidates extracted by the object region extraction unit or the updated object region candidates;
Based on the object region candidate extracted or updated by the object region extraction unit, the updated object region candidate, the appearance model estimated by the appearance model estimation unit, and the association estimated by the association estimation unit , An object region updating unit that updates the object region candidate of each image included in the image group;
A convergence determination unit that repeatedly performs estimation by the appearance model estimation unit, estimation by the association estimation unit, and update by the object region update unit until a predetermined convergence condition is satisfied;
An object region output unit that outputs the object region candidate of each image included in the image group as an object region;
Image segmentation device including:   The image segmentation program for making a computer perform each step of the image segmentation method of any one of Claims 1-5.