JP2017084006A - Image processor and method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a calculation cost when extracting feature amounts of an integrated area obtained by integrating partial areas.
A dividing unit divides an image into partial areas. The extraction unit 103 extracts the first feature amount from the partial region. Based on the first feature amount, the integration unit 104a integrates the partial areas and generates an integrated area. The calculation unit 104b calculates at least a statistical value of the first feature amount of the partial region included in the integrated region as the integrated region feature amount.
[Selection] Figure 1

Description

  The present invention relates to image processing such as image recognition.

  When learning and recognizing a subject in an image, a method is used in which learning and recognition are performed based not only on information on a local area in the image but also on information on a certain area. For example, when an image is segmented into meaningful groups (hereinafter referred to as “semantic region segmentation”), if a recognition method that focuses only on local regions is used, a part of the white wall is identified as “empty”. In some cases, accurate discrimination is difficult. Therefore, a method is used in which recognition is performed using information on a region wider than a local region.

  When performing recognition using a wider area, as shown in Patent Documents 1 and 2, a method of using an integrated area obtained by integrating partial areas is conceivable. However, a method for extracting feature amounts directly from an integrated region, such as scanning all pixels included in the integrated region and calculating feature amounts, has a high calculation cost.

  Also, since the ideal size and shape of the integrated region varies depending on the subject to be recognized, it is difficult to prepare an appropriate integrated region in advance. Patent Document 3 discloses a method in which a large number of integrated regions are generated by changing the scale and the like, and recognition is performed using these simultaneously. Such a method further increases the calculation cost of feature quantity extraction.

JP 2013-027637 JP 2009-212750 A US Patent Application Publication No. 2014/0037198

R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk `` SLIC Superpixels Compared to State-of-the-art Superpixel Methods '' IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, Issue 11, 2274-2282, 2012 T. Ojala, M. Pietikainen, and D. Haywood "A comparative study of texture measures with classification based on featured distributions" Pattern Recognition, Vol. 29, No. 1, pp. 51-59, 1996

  An object of the present invention is to reduce the calculation cost when extracting a feature amount of an integrated area obtained by integrating partial areas.

  The present invention has the following configuration as one means for achieving the above object.

  The image processing according to the present invention divides an image into partial regions, extracts a first feature amount from the partial region, and integrates the partial regions based on the first feature amount to generate an integrated region. The statistical value of at least the first feature amount of the partial region included in the integrated region is calculated as the integrated region feature amount.

  ADVANTAGE OF THE INVENTION According to this invention, the calculation cost at the time of extracting the feature-value of the integrated area | region which integrated the partial area | region can be reduced.

FIG. 2 is a block diagram for explaining a configuration example of an image processing apparatus according to the first embodiment. 6 is a flowchart for explaining image recognition processing by the image processing apparatus. The figure explaining the discrimination | determination process of the category of an attention partial area. The figure which shows the example of a recognition result by the image recognition process of an Example. 1 is a block diagram illustrating a configuration example of a computer device that functions as an image processing device according to an embodiment. The figure which shows the feature-value which can be utilized as an integrated area | region feature-value. The figure explaining the aspect of the calculation method of an integrated area | region feature-value. FIG. 3 is a block diagram for explaining an example configuration of an image processing apparatus according to a second embodiment. 9 is a flowchart for explaining image recognition processing according to the second embodiment. FIG. 6 is a block diagram for explaining an example configuration of an image processing apparatus according to a third embodiment. 10 is a flowchart for explaining object detection processing according to the third embodiment. The figure explaining an object detection process. FIG. 9 is a block diagram illustrating an example of the configuration of an image processing apparatus according to a fourth embodiment. 9 is a flowchart for explaining main subject detection processing according to a fourth embodiment.

  Hereinafter, an image processing apparatus and an image processing method according to embodiments of the present invention will be described in detail with reference to the drawings. In addition, an Example does not limit this invention concerning a claim, and all the combinations of the structure demonstrated in an Example are not necessarily essential for the solution means of this invention.

  The present invention relates to image recognition processing such as detection of a subject in an input image, region division for dividing a region for each subject, image search for searching for a similar image, and scene determination for determining a scene of an image. The input image may be a still image or a moving image. The subject includes all objects such as creatures such as people and dogs, artifacts such as buildings and tools, nature and landscapes such as mountains and the sky.

  In the following, an image recognition process for determining a category of a subject included in an input image and dividing the image into semantic regions will be described. General C categories such as sky, human body, vegetation, buildings, cars, and roads are applied as subject categories.

[Device configuration]
A configuration example of the image processing apparatus according to the first embodiment will be described with reference to the block diagram of FIG. In the image processing apparatus, the acquisition unit 101 acquires an image to be recognized. The dividing unit 102 divides the acquired image into regions. The extraction unit 103 extracts a feature amount (hereinafter “first feature amount”) for each of the divided partial areas.

  The integrated region generation unit 104 generates an integrated region obtained by integrating partial regions and a feature amount thereof (hereinafter, “integrated region feature amount”). The integrated region generation unit 104 calculates a statistical value indicating a feature of each integrated region, and outputs the statistical value as an integrated region feature amount, and an integration unit 104a that integrates the partial regions based on the first feature amount Part 104b.

  The integrated area recognition unit 105 recognizes the category of the corresponding integrated area based on the integrated area feature amount, and outputs a discrimination score for C types of categories, as will be described in detail later. The partial region recognition unit 106 recognizes the category of the corresponding partial region based on the first feature amount, and outputs a discrimination score for C types of categories, as will be described in detail later.

  Based on the partial region discrimination score input from the partial region recognition unit 106 and the integrated region discrimination score including the partial region input from the integrated region recognition unit 105, the feature amount generation unit 107 calculates the feature amount of the partial region (Hereinafter referred to as “second feature amount”). The category determination unit 108 outputs the category of the corresponding partial area recognized based on the second feature amount.

[Image recognition processing]
Image recognition processing by the image processing apparatus will be described with reference to the flowchart of FIG. The acquisition unit 101 acquires an image to be recognized from an imaging device such as a camera or a server device (S11). Note that the acquired image is a still image or an image of one frame in a moving image.

  Next, the dividing unit 102 divides the acquired image into regions (S12). For example, an image is divided into “Superpixel (SPx)”, which is a cluster of pixels having similar colors, using a method described in Non-Patent Document 1. That is, SPx corresponds to the partial area.

  Next, the extraction unit 103 extracts a first feature amount from each partial region (S13). For example, feature quantities such as a color distribution histogram, Local Binary Pattern (LBP) described in Non-Patent Document 2, area moments, and higher-order statistics are extracted. The first feature quantity is obtained by normalizing the feature quantity for each dimension in order to connect these plural kinds of feature quantities and absorb the difference in the scale of the feature dimension.

  Next, K iteration processes are executed. In other words, integrated regions of various sizes and shapes are generated from the partial regions, and the category of the integrated region is recognized based on the integrated region feature amount generated from each integrated region.

  In the iterative process, the integration unit 104a generates an integrated region obtained by integrating a plurality of partial regions based on the first feature amount (S14). For example, the first feature amount is clustered by a clustering method such as the k-means algorithm, and the partial regions belonging to the cluster are integrated to form an integrated region. Here, an integrated area obtained by integrating SPx corresponding to the partial areas is referred to as “Super-Superpixel (SSPx)”.

  The calculation unit 104b calculates a statistical value of the first feature amount of the partial region (hereinafter, included region) included in the integrated region as the integrated region feature amount (S15). In other words, the calculation unit 104b votes for the feature amount (Visual Word) of the codebook closest to the first feature amount of the inclusion region, and uses the frequency histogram (Bag-of-Fearures) indicating the voting result as the integrated region feature amount. Calculate as

  The code book is created in advance. That is, various learning images corresponding to C types of categories are prepared. Then, the learning image is divided into regions by the dividing unit 102, the feature amount is extracted for each partial region by the extracting unit 103, the feature amounts are clustered, and a code that is a set of feature amounts (Visual Word) at the center of the cluster Create a book.

  The integrated region recognition unit 105 recognizes the category of the corresponding integrated region based on the integrated region feature amount, and outputs C category discrimination scores indicating the recognition result (S16). The processing of steps S14 to S16 is repeated K times (predetermined number of times) while changing the k value of the k-means algorithm. By this iterative process, the category discrimination score of the integrated area generated in various sizes and shapes can be obtained.

  When the repetition process ends, the partial area recognition unit 106 recognizes the category of the corresponding partial area based on the first feature value, and outputs C category discrimination scores indicating the recognition result (S17). The feature quantity generation unit 107 connects the category determination score of the partial region output from the partial region recognition unit 106 and the category determination score output from the integrated region recognition unit 105 of the integrated region including the partial region. Is generated (S18).

  Next, the category determination unit 108 recognizes the category of the corresponding partial area based on the second feature amount, and outputs the recognized category as the category of the partial area (S19). Each of the integrated region recognition unit 105, the partial region recognition unit 106, and the category determination unit 108 includes a support vector machine (SVM) classifier. The discriminator is trained in advance to output the correct category for the input variable, with the integrated region feature value, the first feature value, or the second feature value as the input variable and the correct answer category as the target variable. ing.

  The SVM is basically a two-class discriminator, learning for each category with the target category as a positive case and all other categories as negative cases, and preparing C SVMs corresponding to C types of categories. Accordingly, C category discrimination scores are obtained for one integrated region or one partial region as a result of the discrimination processing in steps S16, S17, and S19. In step S19, the category having the highest discrimination score among the C category discrimination scores is output as the partial region category.

With reference to FIG. 3, the process of determining the category of the target partial area will be described. Dividing unit 102, an input image divided into regions, and generates the SP _N from the partial area SP _1. In the following, the category determination process will be described using SPn, which is the _nth partial region SPx, of the partial regions SPx as a target partial region.

Extraction unit 103 extracts the first feature quantity from the partial area SP ₁ SP _N, respectively. Partial area recognition unit 106 calculates the category likelihood of partial regions SP _n based on the first characteristic amount of partial regions SP _n. The category likelihood is obtained for each of the C types of categories.

The integration unit 104a generates an integrated region SSPx in which the partial regions SPx are integrated based on the first feature amount. Assume that SSP _M including, for example, M integrated regions SSP ₁ including the target partial region SP _n is generated. The calculating unit 104b calculates the statistical value of the first feature amount of the partial region SPx included in the generated integrated region SSPx as the integrated region feature amount. The integrated region recognition unit 105 calculates the category likelihood of each of the integrated regions SSP ₁ to SSP _M based on the integrated region feature amount. As described above, the category likelihood is obtained for each of the C types of categories.

Feature amount generating unit 107 generates a category likelihood of partial regions SP _n, the second feature quantity linked category likelihood of SSP _M from the integrated area SSP _1. Category determining unit 108 calculates the second feature quantity categories likelihood of partial regions SP _n, the likelihood outputs the highest category as a category of partial regions SP _n.

The feature amount of the integrated region SSPx is usually extracted based on the pixel and edge information of the integrated region. Such a feature quantity extraction method is computationally expensive. On the other hand, according to the embodiment, the extraction of the integrated region feature value is performed based on the feature value of the partial region calculated in advance, and the calculation cost of the integrated region feature value is reduced. For example, the feature amount of the consolidated region SSP ₁ shown in FIG. 3, the inclusion area SP _n of the combined area _{SSP 1, SP o, SP p} , is calculated as a statistical value of the first characteristic amount of SP _q respectively, combined areas The calculation cost for generating the feature value can be kept low.

  Also, when determining a subject category based on a partial area of a subject, it is difficult to correctly determine the subject category if a certain partial area is similar to a category different from the category of the subject. Ideally, the partial area is generated according to the shape of the subject, but generating such a partial area is equally difficult to solve the problem of semantic area division.

  In the embodiment, when generating integrated regions of various shapes and sizes, an appropriate integrated region may be obtained or an inappropriate integrated region may be obtained. However, by generating a plurality of integrated regions having different shapes and sizes, there is a high possibility that an appropriate integrated region is generated for the subject in the image. Therefore, by integrating the discrimination results of various integrated areas, the recognition result is stabilized, and a more correct recognition result can be easily obtained.

  FIG. 4 shows an example of recognition results obtained by the image recognition processing of the embodiment. If the category of the partial area obtained by dividing the input image showing a person and a vehicle is determined, the vehicle's window glass and bonnet are similar to the sky, so it is erroneously determined as “sky” as in the partial areas 401 and 402. Is done.

  On the other hand, when integrated regions of various shapes and sizes are generated by integrating partial regions and the categories of each integrated region are discriminated, misclassifications such as the integrated regions 404 and 406 are included, but correct discrimination results are obtained. There are integrated areas 403 and 405 to be created. By integrating the determination results of these various integrated regions, the recognition result is stabilized, and it becomes easier to obtain a correct recognition result than focusing only on the partial region.

[Configuration of information processing device]
The block diagram of FIG. 5 shows a configuration example of a computer device that functions as the image processing apparatus of the embodiment. The CPU 201 uses the RAM 202 as a work memory, executes a program stored in the ROM 203 or the storage unit 204, and controls a configuration described later via the system bus 208. The storage unit 204 is a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like, and stores an OS and a program that realizes the above-described image recognition processing.

  The general-purpose interface 205 is a serial bus interface such as a USB, for example, and is connected to an operation unit 211 such as a mouse and a keyboard and a digital camera 212 which is one source of images to be recognized. The recognition target image can be input from the storage unit 204, a recording medium of a drive connected to the general-purpose interface 205, a server device connected to the network 213, or the like.

  The video interface 206 is a video interface such as HDMI (registered trademark) or DisplayPort (trademark), to which the monitor 106 is connected. The network interface 207 is an interface for connecting to a wired or wireless network 213. User operation and connection with the digital camera 212 may be performed via the network interface 207.

  Thus, the recognition accuracy of the category of the partial area can be improved by connecting the recognition result of the category of the partial area and the recognition result of the category of the integrated area including the partial area. In addition, since the feature amount for recognizing the category of the integrated region is calculated as a statistic amount of the feature amount of each partial region included in the integrated region, it is possible to reduce the calculation cost in feature amount extraction. In other words, semantic area division with low calculation cost and high accuracy is realized.

[Modification]
In the above, an example in which the integration unit 104a uses the k-means algorithm has been described. However, a clustering algorithm such as Mean-Shift or spectral clustering may be used. In addition to the clustering algorithm, any algorithm that integrates regions may be used.

  In the above description, the example in which the integration unit 104a clusters the first feature amount in the feature space and integrates the partial regions has been described. However, an integrated region may be generated by comparing the first feature amounts of adjacent partial regions and connecting the partial regions when the similarity of the first feature amounts is equal to or greater than a predetermined threshold. Absent. Alternatively, when the difference between pixel values near the boundary between two partial areas is smaller than a predetermined threshold, the integrated areas may be generated by connecting the partial areas.

  A plurality of integrated areas may be generated hierarchically. In the above, an example has been described in which the integration unit 104a integrates partial areas based on the colors and LBP features of the entire partial areas, but integration of partial areas based only on color features, integration of partial areas based only on LBP features, etc. The integrated region may be generated by changing various feature quantities used for integration.

  FIG. 6 shows feature quantities that can be used as integrated area feature quantities. In the above description, an example has been described in which the calculation unit 104b generates a frequency histogram (FIG. 6A) of feature amounts that are converted into codebooks of partial regions included in the integrated region as integrated region feature amounts. However, higher-order statistics (FIG. 6B) such as the mean, variance, skewness, and likelihood of feature quantities in the inclusion area may be used as the integrated area feature quantity. Further, the weighted linear sum of the inclusion region (FIG. 6C) may be used as the integrated region feature amount.

  An aspect of the calculation method of the integrated region feature value will be described with reference to FIG. In the above, an example has been described in which the calculation unit 104b calculates the statistical value from all the included regions when generating the integrated region feature amount. However, as shown in FIG. 7, a statistical value may be calculated by distinguishing between a partial region 601 located near the boundary of the target integrated region and a partial region 602 located outside the boundary.

  Further, a statistical value is calculated from the first feature amount of the inclusion region 603 and the first feature amount of the inclusion region 604 of the integration region adjacent to the attention integration region, and the statistical values are integrated into the attention integration region. It may be an area feature amount.

  Further, the statistical value of the attention integration region 605 and the statistical value of the integration region 606 adjacent to the upper side of the attention integration region, and the statistical value of the attention integration region 605 and the statistical value of the integration region 607 adjacent to the lower side of the attention integration region May be the integrated region feature amount of the target integrated region. In the attention integration region, the statistical value of the partial region 609 in contact with the boundary, the statistical value of the partial region 608 in contact with the boundary, and the statistical value of the partial region 610 in contact with the attention integration region (or in the vicinity of the attention integration region) Alternatively, the integrated region feature amount of the attention integrated region may be used.

  As described above, various methods of calculating the integrated region feature value in the calculation unit 104b can be considered, and the calculation method is not limited to one.

  In the above description, the SVM is used as the partial region recognition unit 106, the integrated region recognition unit 107, and the category recognition unit 109. However, another classifier can be used. For example, logistic regression, neural network, random forest, etc. can be used. Further, the category discrimination may be performed by incorporating the category discrimination score of the partial area and the integrated area into the framework of a conditional random field.

  In the above description, an example has been described in which the determination result of the category of the partial region based on the first feature amount and the determination result of the category of the integrated region including the partial region are connected to generate the second feature amount. However, the second feature value may be generated by connecting the first feature value and the integrated region feature value. Alternatively, the result of determining the category of the integrated region, or basic statistics such as the average, variance, skewness, kurtosis of the integrated region feature value are connected to the first feature value of the corresponding partial region, and the second A feature amount may be generated.

  Hereinafter, an image processing apparatus and an image processing method according to a second embodiment of the present invention will be described. Note that the same reference numerals in the second embodiment denote the same parts as in the first embodiment, and a detailed description thereof may be omitted.

  In the second embodiment, an example in which the present invention is applied to an image recognition task different from the first embodiment will be described. In the image recognition processing according to the second embodiment, a still image is input and a scene category of the input image is determined. The category is a category of C types of scenes classified in advance by the user, such as mountain scenery, city scenery, and portraits.

  A configuration example of the image processing apparatus according to the second embodiment will be described with reference to the block diagram of FIG. The image processing apparatus according to the second embodiment includes an acquisition unit 101, a division unit 102, an extraction unit 103, and an integrated region generation unit 104 that are the same as those of the first embodiment, and further includes a first determination unit 111 that determines a scene in the integrated region. And a second discriminating unit 112 for discriminating a scene of the input image.

  The image recognition process of the second embodiment will be described with reference to the flowchart of FIG. The processing from step S11 to S15 is the same as that in the first embodiment. In the repetitive processing, the first determination unit 111 determines a scene category of a corresponding integrated region using a frequency histogram that is an integrated region feature amount as an input variable and using a discriminator such as SVM (S21). Note that the SVM outputs C category discrimination scores for C types of scenes, and C category discrimination scores are obtained for one integrated region.

  The processes in steps S14, S15, and S21 are repeated K times with various changes in the shape and size of the integrated region. By this iterative process, K category discrimination scores are obtained for one input image.

  When the iterative process is completed, the second identification unit 112 determines the scene of the input image using a classifier such as SVM, using the feature quantity obtained by connecting the K category determination scores as an input variable (S22). The SVM calculates a discrimination score indicating which of the C scene categories the input image is, and C category discrimination scores are obtained for the input image. The second identification unit 112 outputs the category of the scene corresponding to the category having the highest discrimination score among the C category discrimination scores as the category of the scene of the input image.

  Hereinafter, an image processing apparatus and an image processing method of Example 3 according to the present invention will be described. Note that the same reference numerals in the third embodiment denote the same parts as in the first and second embodiments, and a detailed description thereof may be omitted.

  In the third embodiment, an object detection process for inputting an image and detecting an object shown in the input image will be described. The object to be detected is a category of C types of objects designated in advance by the user, such as a person or a vehicle.

  A configuration example of the image processing apparatus according to the third embodiment will be described with reference to the block diagram of FIG. The image processing apparatus according to the third embodiment includes an acquisition unit 101, a division unit 102, an extraction unit 103, and an integrated region generation unit 104 that are the same as those in the first embodiment. Furthermore, the estimation unit 121 that estimates the object-likeness of the integrated region (hereinafter, “object likelihood”), the determination unit 122 that determines the integrated region corresponding to the object, and the extraction unit 123 that extracts the feature amount of the rectangular region surrounding the integrated region The detection unit 124 detects an object based on the feature amount of the rectangular area.

  The object detection process of the third embodiment will be described with reference to the flowchart of FIG. The processing from step S11 to S14 is the same as that in the first embodiment. The process of step S15 is substantially the same as in the first embodiment, but in the third embodiment, a partial region 601 located near the boundary of the integrated region and a partial region 602 located outside the vicinity of the boundary shown in FIG. A statistical value is calculated for each distinction. As a result, it is possible to recognize the object shape that is likely to occur at the boundary portion of the integrated region in consideration of defects and burrs, and the detection result is stabilized.

  In the iterative process, the estimation unit 121 estimates the object likelihood of the corresponding integrated region using a discriminator such as SVM using the integrated region feature quantity as an input variable (S31). The discriminator learns in advance using the integrated region feature quantity as an input variable, an integrated region that is an object as a positive example, and an integrated region that is not an object as a negative example. The processes in steps S13, S14, and S31 are repeated K times by changing the shape and size of the integrated region in various ways, and the object likelihood is estimated for the integrated regions of various shapes and sizes.

  When the iterative process ends, the determination unit 122 rejects the integrated region having the object likelihood less than the predetermined threshold as not corresponding to the object (S32). In other words, the integrated area corresponding to the object is determined. This process reduces subsequent processes related to the integrated region that is estimated not to be an object.

  Next, the extraction unit 123 extracts a second feature amount of a rectangular area (hereinafter referred to as an inclusion area) surrounding the integrated area determined to correspond to the object (S33). The second feature amount is, for example, a gradient direction histogram (histograms of oriented gradients: HOG), which is a general feature amount in object detection.

  Next, the detection unit 124 uses the second feature amount as an input variable, detects a category of the object in the corresponding integrated region using a discriminator such as SVM (S34), and outputs the category of the object (S34). ). When the determination score obtained in step S34 is small for all categories, the detection unit 124 determines that the integrated region does not correspond to an object.

The object detection process will be described with reference to FIG. The dividing unit 102 divides the input image into regions. The extraction unit 103 extracts the first feature amount for each partial region. The integration unit 104a generates an integrated region SSPx in which the partial regions SPx are integrated based on the first feature amount. Assume that integrated regions having various shapes and sizes are generated, and, for example, integrated regions SSP ₁ to SSP _M are obtained.

The calculation unit 104b calculates a statistical value of the feature amount of the included partial region as the integrated region feature amount. The estimation unit 121 estimates the object likelihood of the corresponding integrated region based on the integrated region feature amount. That is, the object likelihood of the integrated areas SSP ₁ to SSP _M is obtained. The estimation unit 121 rejects the integrated region having the object likelihood less than the threshold value as not an object.

  The extraction unit 122 extracts a feature quantity such as HOG from a rectangular area (inclusion area) surrounding an integrated area having an object likelihood equal to or greater than a threshold. The detection unit 123 estimates the likelihood for each object category based on the feature amount of the inclusion region, and outputs the object category having the maximum likelihood as the category of the object detected from the input image.

  In this way, the candidate area of an object is limited by a feature quantity extraction method with a low calculation cost, and the object is discriminated using the feature quantity with the next highest calculation cost, thereby achieving a balance between accuracy and calculation quantity. Object detection processing can be performed.

  Hereinafter, an image processing apparatus and an image processing method according to a fourth embodiment of the present invention will be described. Note that the same reference numerals in the fourth embodiment denote the same parts as in the first to third embodiments, and a detailed description thereof may be omitted.

  In the fourth embodiment, a process of inputting an image and recognizing the main subject from the input image will be described. A configuration example of the image processing apparatus according to the fourth embodiment will be described with reference to the block diagram of FIG. The image processing apparatus according to the fourth embodiment includes an acquisition unit 101, a division unit 102, an extraction unit 103, and an integrated region generation unit 104 that are the same as those in the first embodiment. Furthermore, an estimation unit 131 that estimates the main subject-likeness (hereinafter, saliency) of the integrated region and a detection unit 132 that detects the main subject are included.

  The main subject detection process of the fourth embodiment will be described with reference to the flowchart of FIG. The processing from step S11 to S14 is the same as that in the first embodiment. The processing in step S15 is substantially the same as in the first embodiment, but in the fourth embodiment, statistical values are calculated from the partial area 608, the partial area 609, and the partial area 610 shown in FIG.

  The partial area 608 is a partial area located inside the attention integration area, is included in the attention integration area, and is a partial area that does not touch the boundary of the attention integration area at all, or a part that touches the boundary of the attention integration area. This is a partial region whose length is less than a predetermined value. The partial region 609 is an adjacent partial region that is included in the attention integration region and is in contact with the boundary of the attention integration region. The partial area 610 is a partial area that is not included in the focused integrated area and touches the boundary of the focused integrated area.

  That is, the calculation unit 104b calculates the statistical value calculated from the partial region located inside, the statistical value calculated from the partial region located inside and the boundary, the partial region located inside and the boundary, and the adjacent partial region Calculate statistics. As a result, not only the focused integrated area but also the statistical value of the feature value including the partial area adjacent to the focused integrated area can be calculated, and the statistical value considering the relationship with the background can be obtained. Improvement in accuracy can be expected.

  In the iterative process, the estimation unit 131 estimates the saliency indicating the similarity between the target integrated region and the surrounding integrated region (S41). As the saliency, values such as the Kullback-Leibler divergence between the feature value of the target integrated region and the feature values of the surrounding integrated region and histogram intersection are used. It seems that it seems. The processes of steps S13, S14, and S41 are repeated K times by changing the shape and size of the integrated region in various ways, and the saliency is estimated for the integrated regions of various shapes and sizes.

  When the repetition process ends, the detection unit 132 calculates the saliency of the partial area by adding the saliency of the integrated areas of various shapes and sizes for each inclusion area (S42). When the degree of saliency of the integrated area including a certain partial area is high and a large number of integrated areas having high saliency including the partial area are generated, the degree of saliency of the partial area is increased by adding the saliency. The detection unit 132 outputs main subject information indicating a set of partial areas with high saliency as the main subject (S43).

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  102 ... Dividing unit, 103 ... Extracting unit, 104a ... Integration unit, 104b ... Calculation unit

Claims (14)

A dividing means for dividing the image into partial areas;
Extraction means for extracting a first feature amount from the partial region;
An integration means for generating an integrated region by integrating the partial regions based on the first feature amount;
An image processing apparatus comprising: a calculation unit that calculates at least a statistical value of a first feature amount of a partial region included in the integrated region as the integrated region feature amount. An integrated region recognition means for recognizing a category of the corresponding integrated region based on the integrated region feature amount;
Partial area recognition means for recognizing the category of the corresponding partial area based on the first feature amount;
Generating means for connecting the recognition results of the integrated area recognition means and the partial area recognition means to generate a second feature quantity;
2. The image processing apparatus according to claim 1, further comprising a determination unit that determines a category of a corresponding partial region based on the second feature amount.   3. The image processing apparatus according to claim 2, wherein the processing of the integrating unit, the calculating unit, and the integrated region recognizing unit is repeated a predetermined number of times by changing the shape or size of the integrated region. First determining means for determining a scene category of a corresponding integrated region based on the integrated region feature amount;
2. The image processing apparatus according to claim 1, further comprising: a second discriminating unit that discriminates a scene category of the image based on a feature amount obtained by connecting the discrimination results of the first discrimination unit.   5. The image processing apparatus according to claim 4, wherein the processing of the integration unit, the calculation unit, and the first determination unit is repeated a predetermined number of times by changing the shape or size of the integration region. Estimating means for estimating the object likelihood of the corresponding integrated region based on the integrated region feature amount;
Determination means for determining an integrated region corresponding to an object based on the object likelihood;
Means for extracting a second feature amount of an area including an integrated area determined to correspond to the object;
2. The image processing apparatus according to claim 1, further comprising a detecting unit that detects a category of the object in the integrated region based on the second feature amount.   7. The image processing apparatus according to claim 6, wherein the calculating unit calculates the statistical value by distinguishing between a partial region located near the boundary of the integrated region and a partial region located outside the boundary.   8. The image processing apparatus according to claim 6, wherein the processing of the integration unit, the calculation unit, and the estimation unit is repeated a predetermined number of times by changing the shape or size of the integration region. Estimating means for estimating the saliency of the corresponding integrated region based on the integrated region feature amount;
2. The image processing according to claim 1, further comprising detection means for adding a saliency of the integrated area for each partial area included in the integrated area and detecting a set of partial areas corresponding to the main subject of the image. apparatus.   The calculation means distinguishes a partial area located inside the integrated area, a partial area included in the integrated area and located at a boundary of the integrated area, and a partial area adjacent to the integrated area. 10. The image processing device according to claim 9, wherein the statistical value is calculated.   11. The image processing apparatus according to claim 9, wherein the processing of the integration unit, the calculation unit, and the estimation unit is repeated a predetermined number of times by changing the shape or size of the integration region.   12. The statistical value according to claim 1, wherein the statistical value is a frequency histogram indicating a result of voting for each feature amount of a codebook prepared in advance based on the first feature amount. Image processing device. Divide the image into partial areas,
Extracting a first feature value from the partial region;
Based on the first feature amount, the partial areas are integrated to generate an integrated area,
An image processing method for calculating a statistical value of at least a first feature amount of a partial region included in the integrated region as an integrated region feature amount.   13. A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.