JP2012003623A - Face image search device and face image search method - Google Patents

Abstract

In addition to outputting a list of face image candidates similar to the input face image, a list of similar face image candidates from the viewpoint based on the attribute information specified in advance is also output at the same time. A face image search device and a face image search method that can be confirmed automatically.
In a face image search device that searches for a face image similar to an input face image from a plurality of face images registered in advance in a face image database, and displays and outputs the face image, a personal image of the face image is obtained. In addition to the first search result that displays the search results in the order of similarity using the feature information, the search results are obtained in the order in which the attribute features of at least one person specified in advance are similar to the input image, and the attribute features The first search result is obtained by using a search result obtained by using the search result or a search result obtained in the order of similarity in consideration of both personal feature information and attribute features of the face image. Display separately.
[Selection] Figure 1

Description

  Embodiments of the present invention include, for example, a face image search device and a face image that search for and display a face image similar to an input face image from a plurality of face images registered in advance in a face image database. It relates to the search method.

  For example, for a plurality of face images registered in advance in a face image database, there is a technique of inputting a predetermined face image, searching for face images similar to the face image, and displaying a list in the order of similarity.

  However, if a large number of face images are registered in the face image database, if the search accuracy by the face recognition technology is low, it will not be displayed as a top candidate, and the candidate will be visually checked while continuing the operation on the screen. Therefore, a technique for efficiently narrowing down when searching a large amount of face image database has been developed.

  For example, in addition to the face image, a search mechanism that searches at high speed by using the name, member ID, and membership date as keys, and attribute information other than the main biological information such as the face (person's height, weight, gender, Techniques for narrowing down recognition dictionaries using age etc. are known.

  On the other hand, a technique for searching a face image having a specified feature from a face image database by specifying a facial feature of a person to be searched as a search condition is also known. For example, there is a technique in which candidate images are arranged by setting a person's skin color, wrinkles, wrinkles, moles, glasses, gender, and estimated age as search conditions using check boxes, logical products, and logical sums.

JP 2007-310646 A JP 2000-090264 A JP 2006-318375 A

Using the attribute information (for example, age, gender, etc.) input in advance from the face image database in association with the face image in the face image database as a key, the search result is obtained by text matching processing separately from the face recognition technology result. Although there is a technology to narrow down, it is necessary to input attribute information as text information in the face image database in advance, and attribute information is not given in advance when a large amount of new face images enter There is a problem that it is not possible to narrow down, and there is also a problem that it is not possible to simultaneously view search results from different viewpoints using attribute information.
Registering attribute information in the face image database is very time-consuming and may cause errors, and it is highly possible that the registered attribute information is not noticed even if it is incorrect.

  Therefore, the embodiment outputs a list of face image candidates similar to the input face image when searching for a face image similar to the input face image from a plurality of face images registered in advance. In addition to pre-specified attribute information, or a candidate that is similar to a face image candidate from the viewpoint of the attribute information that automatically selected a search result that seems to represent the search result closest to the input image It is an object of the present invention to provide a face image search device and a face image search method that can simultaneously output a list and visually confirm the list.

  The face image search device according to the embodiment includes an image input unit that inputs an image including at least a person's face image, a face detection unit that detects at least one face region from the input image input by the image input unit, Search condition setting means for setting the type of attribute to be searched and search result output method for setting search conditions, and facial features indicating personal features of the face obtained from the face area detected by the face detection means Face feature extracting means for extracting information, attribute feature extracting means for extracting at least one person's attribute feature information obtained from the face area detected by the face detecting means, a pre-registered face image and corresponding Personal information management means for managing facial feature information and attribute feature information in association with each other; and facial feature information for the input image extracted by the facial feature extraction means; By comparing with the facial feature information managed by the personal information management means, the facial feature information for the input image is similar to any facial feature information in the facial feature information managed by the personal information management means The person information managing means retrieves a plurality of corresponding face images determined in advance in descending order of the degree of similarity, and the attribute feature extracting means extracts the corresponding face images. By comparing the attribute feature information with respect to the input image and the attribute feature information managed by the person information management means, the similarity of both attribute feature information is obtained, and a plurality of predetermined numbers in order from the highest similarity are obtained. Attribute search means for searching the corresponding face image from the person information management means, a plurality of search results obtained by the attribute search means, and this An optimum search result automatically selected as necessary from a plurality of search results and a content set by the search condition setting means from the search results obtained by the face search means Search result management means for selecting a search result and output means for outputting the search result selected by the search result management means.

1 is a block diagram schematically showing the configuration of a face image search device according to a first embodiment. The figure which shows the example of a setting screen of the search condition setting part which concerns on 1st Embodiment. The figure which shows the structural example of the management information of the person information management part which concerns on 1st Embodiment. The flowchart explaining the flow of the whole process which concerns on 1st Embodiment. The figure which shows the example of a screen display of the output part which concerns on 1st Embodiment. The block diagram which shows schematically the structure of the face image search device which concerns on 2nd Embodiment. The figure which shows the example of a setting screen of the search condition setting part which concerns on 2nd Embodiment. The figure which shows the example of a composite image creation screen of the image composition part which concerns on 2nd Embodiment. The flowchart explaining the flow of the whole process which concerns on 2nd Embodiment. The figure which shows the example of a screen display of the output part which concerns on 2nd Embodiment. The block diagram which shows roughly the structure of the face image search device which concerns on 3rd Embodiment. The figure which shows the example of a setting screen of the search condition setting part which concerns on 3rd Embodiment. The figure which shows the structural example of the management information of the person information management part which concerns on 3rd Embodiment. The flowchart explaining the flow of a process of the attribute discrimination | determination part concerning 3rd Embodiment. The figure for demonstrating the age estimation method in the attribute discrimination | determination part which concerns on 3rd Embodiment. The flowchart explaining the flow of the age estimation method in the attribute discrimination | determination part which concerns on 3rd Embodiment. The flowchart explaining the flow of the process at the time of registration which concerns on 3rd Embodiment. 9 is a flowchart for explaining a flow of processing during a search according to the third embodiment. The figure which shows the example of a screen display of the output part which concerns on 3rd Embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
As an object to which the embodiment is applied, it is assumed that a face image (person) that is the same as the face image currently at hand is searched from a large number of face images registered in advance. For example, it is possible to search for a customer from customer data, search for a corresponding person from a database of a suspicious person, verify an identity when updating an ID certificate, and prevent double issuance at the time of new issuance.

  Further, in the embodiment, as a method of discriminating the person based on the attribute information, the face feature area obtained from the photographing target is detected and the face position and the face feature are used to extract the person attribute feature information, Search for a face image close to the attribute feature information from a pre-registered face database, and determine whether it matches attribute information such as age and gender (entered as text information on a terminal device, etc.) Use of the results increases the efficiency of searching for a person (face image).

  In the following description, a procedure for realizing a problem by detecting a face area as a person area and using facial feature information will be described. However, a technique for detecting a whole person area other than a face (Watanabe et al., “ Co-occurrence Histograms of Oriented Gradients for Pedestrian Detection, In Proceedings of the 3rd Pacific-Rim Symposium on Image and Video Technology ”(PSIVT2009), pp. 37-47.) It can also be realized by measuring height information, and is limited to the technique described in the present embodiment as long as it is a technique for detecting a person's area and a technique for measuring characteristic information in the person area. It is clear that it is not content.

First, the first embodiment will be described.
FIG. 1 schematically shows a configuration of a face image search apparatus according to the first embodiment. The face image search device according to the first embodiment includes an image input unit (image input unit) 110 that inputs an image including at least a face of the person M to be imaged, and at least one input image input by the image input unit 110. A face detection unit (face detection unit) 120 that detects a face area, and a search condition setting unit (search condition setting unit) 130 that sets the type of attribute to be searched for setting search conditions, a search result output method, and the like. A face feature extraction unit (face feature extraction means) 151 for extracting facial feature information indicating personal features of the face obtained from the face area detected by the face detection unit 120; obtained from the face region detected by the face detection unit 120; Attribute feature extraction unit (attribute feature extraction means) that extracts attribute feature information for identifying attribute information (age, sex, presence / absence of glasses, presence / absence of hat, presence / absence of race, race, etc.) 52. A person information management unit (person information management means) 160 for managing a face image registered in advance in association with face feature information and attribute feature information for the face image, and an input image extracted by the face feature extraction unit 151 By comparing the face feature information with the face feature information managed by the person information management unit 160, which face feature in the face feature information managed by the person information management unit 160 is the face feature information for the input image. A face search unit (face search means) 171 that obtains a similarity degree that is most similar to information and searches the person information management unit 160 for a plurality of corresponding face images that are predetermined in descending order of similarity. , Attribute feature information for the input image extracted by the attribute feature extraction unit 152 and attribute feature information managed by the person information management unit 160. An attribute search unit (attribute search unit) 172 that obtains a similarity between both attribute feature information by comparison and searches the person information management unit 160 for a plurality of corresponding face images determined in advance in descending order of similarity. , A plurality of search results obtained by the attribute search unit 172, an optimum search result automatically selected from the plurality of search results as necessary, and a search obtained by the face search unit 171 A search result management unit (search result management unit) 180 that selects a search result that matches the content set by the search condition setting unit 130 from among the results, and an output unit that outputs the search result selected by the search result management unit 180 (Output means) 190 is comprised.

Hereinafter, each part will be described in detail.
The image input unit 110, for example, captures and inputs an image including at least the face of the person M with the ITV camera 109, and digitizes an image signal obtained from the ITV camera 109 with an A / D converter as image information. input.

  The search target may be a digital face image as a result, so an image file captured by a digital still camera may be captured via a memory medium, or from a paper medium or photograph using a scanner. It may be a scanned digital image.

  For example, the face detection unit 120 obtains the position that gives the highest correlation value in the face region by obtaining a correlation value while moving a template prepared in advance in the image in the input image input by the image input unit 110. And In addition, it can be realized by a face extraction method using an eigenspace method or a subspace method.

  In addition, the position of a face part such as an eye or nose is detected from the detected face area. The detection method is, for example, Document 1 (Kazuhiro Fukui, Osamu Yamaguchi: “Face Feature Point Extraction by Combination of Shape Extraction and Pattern Matching”, IEICE Transactions (D), vol. J80-D-II, No. 8, pp2170-2177 (1997)).

  In the present embodiment, an example of using the feature of the face region has been described. However, even in the case of an apparatus that determines whether the face is an iris, retina, or eye image, it is detected by the above processing. Detection is possible by zooming the camera based on the eye area.

  In addition to the above eye and nose detection, the detection of the mouth area is described in, for example, Reference 2 (Mayumi Yuasa, Ayako Nakajima: “Digital Make System Based on High-Precision Facial Feature Point Detection”, 10th Image Sensing Symposium Proceedings , Pp 219-224 (2004)) can be easily realized.

  In any case, it is possible to acquire information that can be handled as a two-dimensional array image, and to detect a facial feature region from the acquired information. Also, in these processes, in order to extract only one facial feature from one image, a correlation value with the template is obtained for all images, and the maximum position and size may be output. In order to extract the facial features, the local maximum value of the correlation value for the entire image is obtained, the candidate positions of the face are narrowed down in consideration of the overlap in one image, and finally the past images that have been input continuously It is also possible to finally find a plurality of facial features at the same time in consideration of the relationship (time transition).

  In addition, in order to enable detection even when wearing a mask, wearing sunglasses, or wearing a hat as in this embodiment, a template with a face pattern when a part of the face is hidden in advance by sunglasses, a mask, or a hat. Even if not all points can be detected when face feature points are detected, if some face feature points are detected with sufficient evaluation values, the remaining feature points are 2 This can be realized by estimating the remaining feature points from the feature points detected using a three-dimensional face model or a three-dimensional face model.

  If the feature point is hidden by a mask, hat, or sunglasses so that the feature point cannot be detected at all, it is learned as a pattern of the entire face and the position of the entire face is detected. It can be realized by guessing from the above.

  If there are a plurality of faces in the image, an instruction as to which face is to be searched may be specified by the search condition setting unit 130 or the output unit 190. It is also possible to output the face-like index so that the search target can be automatically selected in the order of the obtained face-like index.

  The search condition setting unit 130 integrates the results of the facial features and each attribute to be used for the search, the order in which the search is performed, whether the search results are displayed on the screen, and the similarity obtained from the facial features. Items such as whether or not to set are set before the search process. This does not require processing every time. If the user of the system sets in advance, it is only necessary to store the parameters and call the information.

  An example of a setting screen for actually setting is shown in FIG. Here, it is assumed that the selected item is a menu indicated by a black rectangular mark or black circle mark. In this example, it is set to perform a search using attribute information of “age” and “mask” separately from the search process using face feature information. For the "similarity integration" check, whether the similarity is calculated based on the feature information of various attributes as a standard, the similarity of the face feature information and the result are output, or the search result is output independently. This is for switching whether to output the search result only by the feature information.

  In the case of FIG. 2, since “age” is not integrated, an image whose age feature is similar to the input face image is displayed at the top regardless of the similarity of the face. On the other hand, for mask features, the search result is output considering both the search result of the mask attribute feature and the search result of the facial feature information, so the input facial image is masked. In addition, it is set to display people who have similar faces in order.

  With regard to “importance”, it is possible to adjust the weight when the similarity is integrated. By specifying here, it is possible to change the weight when the output unit 190 displays the search result.

  “Screen display” is a setting for whether or not the search result is displayed on the screen, but the setting may be changed later by the output unit 190 separately.

  In addition, the “order of arrangement” is basically “highest order”, which is output in descending order of similarity, but the input image has a mask, but the face image database has a mask. You can select “Lowest” when searching for face images that do not exist.

  In this embodiment, only the order of arrangement is changed as “highest order” and “lowest order”, but there are only two types whether or not a specific condition such as the presence or absence of wearing of a mask, glasses, a hat, or the like is satisfied. It is also possible to select “match or not” for a case where there is no state. In this case, only the matching items will be arranged in order, but if “similarity integration” is performed, the ones that satisfy the condition “matching” with the highest facial similarity will be ordered. It can also be displayed.

  Furthermore, the search condition setting by the search condition setting unit 130 can be set to “automatic”. In this case, the mode is such that the search result that is most likely to represent the candidate (candidate face image) is automatically selected and output while looking at the numerical value of the search result. Automatically select the optimum attribute type, face-attribute combination, and attribute-to-attribute combination.

  In the example shown in FIG. 2, the operator does not select an attribute for displaying the search result in advance, but automatically selects attribute information that is closer to the input image as a higher order search result from a predetermined index by the system. It is also possible to display it as "automatic selection". Here, the search result using attribute information with the highest index is set to “automatic selection 1”, and the search result using attribute information with the next highest index is set to “automatic selection 2”.

  In the present embodiment, the top two search results that can be selected based on the optimum index have been selected. However, this number can be increased according to the number of searchable attributes and the screen configuration. That is obvious.

  The face feature extraction unit 151 outputs face feature information (face feature information) for identifying an individual from the face area information as a numerical value. First, based on the position of the face part detected by the face detection unit 120, the face area is cut into a certain size and shape, and the shading information is used as a feature amount. Here, the gray value of an area of m pixels × n pixels is used as information as it is, and m × n-dimensional information is used as a feature vector.

  These are normalized so that the vector and the length of each vector are set to “1” by a method called a simple similarity method, and a similarity indicating the similarity between feature vectors is obtained by calculating an inner product. Specifically, for example, by using the subspace method as described in Reference 3 (by Elkki Oya, Hidemitsu Ogawa, Makoto Sato, “Pattern Recognition and Subspace Method”, Sangyo Tosho, 1986) realizable.

Further, as disclosed in Document 4 (Japanese Patent Laid-Open No. 2007-4767), by creating an image in which the orientation and state of the face are intentionally changed using a model for one face image. A technique for increasing accuracy may be applied.
When the facial feature information is obtained from one image, the facial feature extraction is completed by the processing so far.

  On the other hand, a more accurate recognition process can be performed by calculating with a moving image using a plurality of continuous images for the same person. Specifically, for example, Document 5 (Kazuhiro Fukui, Osamu Yamaguchi, Kenichi Maeda: “Face Recognition System Using Moving Images” IEICE Research Report PRMU, vol 97, No. 113, pp 17-24 (1997)). , Reference 6 (Kenichi Maeda, Sadaichi Watanabe: “Pattern Matching Method Introducing Local Structure”, IEICE Transactions (D), vol. J68-D, No. 3, pp 345-352 (1985)). The method using the mutual subspace method in FIG.

  An image of m × n pixels is cut out from images continuously obtained from the image input unit 110 in the same manner as the face feature extraction unit 151, a correlation matrix of feature vectors is obtained from these data, and an orthonormal vector by KL expansion is obtained. To calculate a subspace indicating the facial features obtained from successive images.

The subspace calculation method calculates a subspace by obtaining a correlation matrix (or covariance matrix) of feature vectors and obtaining an orthonormal vector (eigenvector) by KL expansion. In the subspace, k eigenvectors corresponding to eigenvalues are selected in descending order of eigenvalues, and expressed using the eigenvector set. In this embodiment, the correlation matrix Cd is obtained from the feature vector, and the correlation matrix
Cd = Φd Λd Φd T
And a matrix Φ of eigenvectors is obtained. This information becomes a partial space indicating the characteristics of the face of the person currently recognized.

  Characteristic information such as a partial space output by such a method is used as personal face characteristic information for the face detected in the input image.

  The attribute feature extraction unit 152 calculates feature information for obtaining the attribute information of the person using the information on the face area detected by the face detection unit 120. As attribute information described in the present embodiment, the age, sex, glasses type, mask type, and hat type are advanced, but in addition, the race, the presence / absence of glasses (1 or 0 information), The presence / absence of a mask (information about 1 or 0), the presence / absence of a hat (information about 1 or 0), items worn on the face (piercings, earrings, etc.), clothes, facial expressions, obesity, wealth, etc. In any case, it is obvious that correspondence is possible by learning for each category in advance as in the attribute determination method described later.

  The method for calculating the feature information used to determine the attribute can be realized by the same subspace method as the face feature extraction unit 151. Therefore, the detailed subspace method calculation method is omitted. Since there is feature information suitable for each attribute, more suitable feature information extraction processing may be used.

  As for the calculation method of attribute features related to age and gender, by applying pre-processing suitable for each age and gender for higher accuracy, instead of calculating the similarity in the personal features of the face, it is age and gender It is possible to calculate feature information for determining suitable similarity. For example, as age increases, heels increase as age increases, so a line segment enhancement filter that emphasizes heels may be applied in advance, and for gender, a filter or skeleton that emphasizes frequency components that emphasize heels It is conceivable to apply a filter that emphasizes information as a preprocessing.

  For feature information about glasses, a mask, and a hat, similar feature information can be obtained by narrowing down an area from the position information of the facial part obtained by the face detection unit 120 and calculating a partial space for the area. For example, if the positions of the eyes, the eyes, and the eyes are known, the feature information for the glasses can be obtained by cutting out the image only in the vicinity of both eyes and using it as the calculation target of the partial space.

  Similarly, feature information can be calculated for a mask by narrowing the region based on the coordinates of the mouth and nose. For hats, if the position of the eyes and eyebrows is known, the upper edge of the skin area of the face is estimated, so the head area of the face is narrowed down and the hat is put on, so the area where the hat exists should be estimated Can do.

  As described above, glasses, a mask, a hat, and the like can be realized by estimating the position from the information obtained by the face detection unit 120, and therefore the same processing can be performed for other attributes. Is clear. There is also an algorithm for directly detecting the worn items, and such a method may be used.

  As for glasses, masks, and hats, if you don't wear them, you may explicitly distinguish the features. Since only features different from feature information such as masks, sunglasses, etc. appear, classification is not necessary.

  The person information management unit 160 is a database to be used when searching by the face search unit 171 and the attribute search unit 172, and includes facial feature information extracted by the face feature extraction unit 151 for each individual to be searched. The attribute information extracted by the attribute feature extraction unit 152 is managed in association with each same person. FIG. 3 shows a configuration example of the management information.

  The content actually managed as face feature information (face feature information) and attribute feature information may be the data itself extracted by the face feature extraction unit 151, and may include m × n feature vectors, subspaces, and KL expansion. The correlation matrix immediately before the execution may be used. Furthermore, by managing the facial feature information extracted by the facial feature extraction unit 151 together with the facial image input at the time of registration by the image input unit 110, it can be used for personal search and search display. In addition, the person ID (identification information) assigned to each person for management, or other essential information or attribute information is associated with text information in advance, although it is not an essential requirement for realizing the present embodiment. It is also possible to manage.

  The face search unit 171 indicates the degree of similarity between the face feature information for the input image calculated by the face feature extraction unit 151 and the face feature information in the person information management unit 160 registered together with a plurality of face images in advance. A calculation is performed to return the search results of face images in descending order of similarity. At this time, as a result of the search processing, a person ID managed in order to identify an individual in the person information management unit 160 and an index indicating the similarity as a calculation result are returned in descending order of the similarity. In addition, information managed for each individual in the person information management unit 160 may be returned together. However, since the association is basically possible by the person ID, the search process itself includes the attached information. It is possible to achieve this without exchanging.

  The index indicating similarity is the similarity between partial spaces managed as face feature information. As a calculation method, a method such as a subspace method or a composite similarity method may be used. In this method, the recognition data in the registration information stored in advance and the input data are expressed as subspaces calculated from a plurality of images, and the “angle” formed by the two subspaces is defined as the similarity. . The partial space input here is referred to as an input means space.

Similarly, a correlation matrix Cin is obtained for the input data string,
Cin = Φin Λin ΦinT
And the eigenvector Φin is obtained. The similarity between subspaces (0.0 to 1.0) of the subspaces represented by two Φin and Φd is obtained and used as the similarity for recognizing this. A specific calculation method can be realized by Document 3 (Erki Oya) introduced in the face feature extraction unit 151. In addition, it is possible to improve accuracy by collecting a plurality of facial images that can be identified as the same person in advance and identifying whether or not the person is the person by projecting it onto a partial space. Can also be performed.

  The attribute search unit 172 has a similarity between the attribute feature information of the attribute for the input image calculated by the attribute feature extraction unit 142 and the attribute feature information in the person information management unit 160 registered together with a plurality of face images in advance. Is performed to return the search results of face images in order from the more similar ones.

  At this time, as a result of the search processing, calculation is performed in accordance with the conditions set in the search condition setting unit 130, and the “order” is “highest” for the attribute that is checked as “search”. If so, the person ID managed in order to identify the individual in the person information management unit 160 and the index indicating the similarity that is the calculation result are returned in descending order of the similarity, and the order is “low to high”. Returns the results in the opposite order.

  If “automatic” is set in the search condition setting unit 130, search results are obtained for all searchable attribute information so that an optimum search result can be selected later, and the search results are input at the same time. A reliability index indicating whether or not an object close to the image has been hit is calculated.

The reliability can be obtained by the following method.
First, as a first method, the similarity up to the top N ranks of search results (N is 1 or more and less than or equal to the number registered in the person information management unit 160) or the average value thereof is the highest. This is a method of selecting a search result and using the absolute value of the similarity of the top candidates as an index.

  Basically, N = 1 and the attribute with the highest similarity of the first search result may be selected. Since the tendency of the similarity distribution may vary depending on the attribute characteristics, the similarity can be normalized using the average value or standard deviation of the entire similarity. For example, it is possible to normalize the absolute value of the similarity and the variance of the similarity by subtracting the average value of the similarity from the similarity and dividing by the standard deviation.

  Next, as the second method, the search results up to the top N (N is the same as the first method) of the search results, and from the N + 1th to Mth (M is N + 1 or more and registered in the person information management unit 160) By calculating the difference in average similarity between the two similarity groups, or the ratio of average similarity, or the degree of separation, the attribute with the largest index is more “other registration information. It indicates that there is registration information that is much more detailed than that, and that it is valid as a search result. This is a method of using the relative relationship between the similarity of the upper candidate and the similarity of the lower candidate as an index.

  The search result management unit 180 manages the search results from the search results obtained by the attribute search unit 172 and the search results obtained by the face search unit 171 according to the contents set by the search condition setting unit 130, and the search conditions If automatic setting is performed by the setting unit 130, one or a plurality of search results with the highest reliability are selected according to a predetermined evaluation value. If automatic setting is not performed, the search condition setting unit 130 selects Select search results using pre-set attribute information.

  First, when “automatic” is not set in the search condition setting unit 130, the search result obtained by the face search unit 171 and the search result obtained by the attribute search unit 172 are used as the search condition setting unit 130. The search result is output so that it can be displayed according to the contents set by. However, if “similarity integration” is checked in the search condition setting unit 130, the result calculated by the face search unit 171 and the similarity value are integrated and the result is returned. In this case, it can be realized by calculating with the following equation (1).

Sim (i) = α * SimFace (i) + β * Sim (i) (1)
Here, i is the index of the person ID managed by the person information management unit 160, SimFace (i) is the similarity to the face feature information corresponding to the i-th person ID, and Sim (i) is the i-th Similarity with the feature information (in this case, the mask) of the corresponding attribute corresponding to the person ID, α and β are coefficients determined by the weight of the face feature information and the attribute feature information, and the search condition setting unit 130 Since it can be set individually, this numerical value may be quoted or may be set in advance as a device.

  As a result, not only in the order of similarity considering only the attribute features, but also from the registered face images, those that are similar as facial features and those that have similar feature information of the attribute features (here, masks) are candidates. It can be displayed as an image.

  Further, here, the similarity between the face feature information and one attribute information is integrated, but the similarity of a plurality of attribute information may be integrated. This case can be dealt with by modifying the right side of the above equation (1) so that a value obtained by multiplying the similarity of the attribute information by a coefficient is added.

  When “automatic” is selected in the search condition setting unit 130, a search similar to the input image from the registered face image is selected from the search results obtained by the face search unit 171 and the attribute search unit 172. Those having high reliability indicating the degree of the result are automatically selected in order and output as search results.

  The output unit 190 displays and outputs the search result selected by the search result management unit 180. In the present embodiment, as in the example shown in FIG. 2, an example is described in which search results using facial features, age, and mask attributes are displayed, but the contents set by the search condition setting unit 130 It is assumed that display output is performed at the same time.

With the configuration as described above, the overall processing flow according to the first embodiment will be described with reference to the flowchart shown in FIG.
In the following description, it is assumed that the setting content in the search condition setting unit 130 is as in the example shown in FIG. Therefore, in this case, a search is performed using facial features, age, and mask attributes, and the search results are displayed.

  In FIG. 4, when an image is input by the image input unit 110 (step S1), the face detection unit 120 detects a face region from the image input by the image input unit 110 (step S2). Thereafter, three search results are calculated.

  First, the search result processing procedure using facial features will be described. The face feature extraction unit 151 extracts face feature information from the face area detected by the face detection unit 120 (step S3). Next, the face search unit 171 compares the face feature information extracted by the face feature extraction unit 151 with the face feature information registered in the person information management unit 160, thereby making the similarity of both face feature information. A plurality of corresponding face images predetermined in order from the one with the highest similarity are searched from the person information management unit 160 (step S4), and the search result is output via the search result management unit 180 to the output unit. For example, as shown in FIG. 5, the search result of the face image using the face feature information is displayed (step S5).

  Next, the processing procedure of the search result using the mask attribute will be described. The attribute feature extraction unit 152 extracts attribute feature information as a mask feature from the face area detected by the face detection unit 120 (step S6). Next, the attribute search unit 172 compares the attribute feature information extracted by the attribute feature extraction unit 152 with the attribute feature information registered in the person information management unit 160, thereby making the similarity between the two attribute feature information. And a plurality of corresponding face images (face images of a person in a similar mask wearing state) determined in advance in descending order of similarity are searched from the person information management unit 160 (step S7). The search result is sent to the output unit 190 via the search result management unit 180, and the search result of the face image using the mask feature information is displayed as shown in FIG. 5, for example (step S8).

  In this example, as shown in FIG. 2, “similarity integration” is “Yes” for the mask feature. Therefore, the search result using the facial feature information in step S4 and the mask feature information in step S7 are used. The search results are displayed by integrating the ranks with the weights set in advance by the search condition setting unit 130.

  Finally, a processing procedure for search results using the age attribute will be described. The attribute feature extraction unit 152 extracts age attribute feature information from the face area detected by the face detection unit 120 (step S9). Next, the attribute search unit 172 compares the attribute feature information extracted by the attribute feature extraction unit 152 with the attribute feature information registered in the person information management unit 160, thereby making the similarity between the two attribute feature information. Is calculated, and a plurality of corresponding face images (face images of persons belonging to similar ages) determined in advance in descending order of similarity are searched from the person information management unit 160 (step S10). The result is sent to the output unit 190 via the search result management unit 180, and the search result of the face image using age attribute feature information is displayed as shown in FIG. 5, for example (step S11).

  The processes in steps S3, S6, and S9 may be performed sequentially or in parallel.

In the screen display example of FIG. 5, the input image 1 input by the image input unit 110 is displayed as it is, and a rectangle is displayed for the face area detected by the face detection unit 120. In this example, since there are a plurality of persons in the input image 1, the face image to be searched is a solid rectangle 2 for easy understanding of which face image is the search target, and the broken line is used for the face image not to be searched. The rectangle 3 is displayed.
If the face image to be searched is to be switched, for example, the search target may be switched by clicking each face area with a mouse (not shown).

  The image group 4 arranged in the upper right of the screen of FIG. 5 is compared with the face image (face feature information) managed in the person information management unit 160 based on the feature information (face feature information) of the face image. The face images are displayed in the order in which the similarity is determined to be high, and a person ID corresponding to the displayed face image and the similarity calculation result 5 are displayed below the face images.

  In the example of FIG. 5, the top five names are displayed as search results. However, it may be possible to change the setting in advance for how many of the top numbers are displayed. For example, images of the next order may be sequentially displayed by a button operation, or a face image may be displayed by scrolling with a scroll bar.

In the image groups 6 and 8 arranged in two rows horizontally at the bottom of the image group 4, the upper image group 6 is a search result using age attribute information, and the lower image group 8 is a search result using mask attribute information. These image groups 6 and 8 are also displayed side by side in the order in which the similarity is determined to be high, and a person ID corresponding to the displayed face image and similarity calculation results 7 and 9 are displayed below the image groups 6 and 8. The
Corresponding setting information 10 and 11 is displayed on the left side of each image group 6 and 8 so that the contents set by the search condition setting unit 130 can be understood.

  From the display of such search results, the search results using the feature information of the face image that is considered to be close to the face of the input image are lined up, and the input face image and `` person who seems to be closest to age '' As a result of arranging the items in order, the results of the input face images and the “similar faces and similar masks worn” arranged in descending order are displayed at the same time.

  As a result, if there is a face concealment (for example, a mask or sunglasses) whose search accuracy in the feature information of the face image is likely to decrease, a search can be performed using information close to the concealment itself, or the facial features And the search result considering both the concealment can be displayed at the same time.

  Also, by setting the order to “lowest”, if the input face image has a mask, you can search for “similarity of facial features” and “face without mask”. It becomes possible.

In the first embodiment, three types of search results are displayed side by side. However, since the number of items varies depending on the search condition setting unit 130, the display area is expanded, reduced, or scrolled. A plurality of search results may be switched.
In addition, face images may be displayed in a cross shape by changing the vertical axis and horizontal axis for each attribute, or similar images can be displayed in order while displaying similar images in three dimensions and operating with a mouse. May be.

  As described above, according to the first embodiment, when searching for a face image similar to the input face image from a plurality of face images registered in advance, the search is similar to the input face image. In addition to displaying a list of face image candidates, it is also possible to visually confirm similar face image candidates from a viewpoint based on attribute information such as age, gender, wear, etc., instead of specifying an individual. By doing so, a list of similar face image candidates from a plurality of viewpoints can be displayed simultaneously for visual confirmation.

Next, a second embodiment will be described.
FIG. 6 schematically shows a configuration of a face image search apparatus according to the second embodiment. The face image search device according to the second embodiment includes an image input unit (image input unit) 210 including an ITV camera 209 that inputs an image including at least the face of the person M to be imaged, and an input input by the image input unit 210. A face detection unit (face detection means) 220 for detecting at least one face area from an image, a search condition setting unit (search for setting search attribute type, search result output method, etc. for setting search conditions) Condition setting means) 230, a composite image management section (composite image management means) 245 for managing a partial area image for synthesizing a predetermined partial area image with the input image input by the image input section 210, search condition setting Based on the conditions set by the unit 230, an image is input to the input image (actually the face area image detected by the face detection unit 220) input by the image input unit 210. Obtained from the face area detected by the image composition unit (image composition unit) 240 and the face detection unit 220 that create an image obtained by compositing the partial area images managed by the composite image management unit 245 for the attribute information selected for composition. Attribute information (age, sex, presence / absence of glasses) of a person obtained from a face feature extraction unit (face feature extraction means) 251 that extracts face feature information indicating individual features of a face image and a face region detected by the face detection unit 220 Attribute feature extraction unit (attribute feature extraction means) 252 for extracting attribute feature information for identifying the presence / absence of a mask, the presence / absence of a hat, race, etc., a facial image registered in advance, and facial feature information corresponding thereto Face feature information for an input image extracted by a person information management unit (person information management unit) 260 that manages attribute feature information and the like in association with each other, and a face feature extraction unit 251 By comparing the face feature information managed by the person information management unit 260 with the face feature information in the face feature information managed by the person information management unit 260, the face feature information for the input image is identical to the face feature information managed by the person information management unit 260. A face search unit (face search means) 271 that searches the person information management unit 260 for a plurality of corresponding face images determined in advance in descending order of similarity. By comparing the attribute feature information for the input image extracted by the extraction unit 252 or the attribute feature information for the image synthesized by the image synthesis unit 240 with the attribute feature information managed by the person information management unit 260, both attribute features are compared. The similarity of information is obtained, and a plurality of corresponding face images determined in advance in descending order of similarity are stored in the person information management unit 260. Attribute search unit (attribute search means) 272 for searching, a plurality of search results obtained by the attribute search unit 272, and an optimum search result automatically selected from the plurality of search results as necessary And a search result management unit (search result management means) 280 for selecting a search result that matches the content set by the search condition setting unit 230 from the search results obtained by the face search unit 271, a search result management unit An output unit (output unit) 290 that outputs the search result selected by H.280 is configured.

  The image input unit 210, the face detection unit 220, the face feature extraction unit 251, the attribute feature extraction unit 252, the person information management unit 260, the face search unit 271, and the search result management unit 280 are described in the first embodiment. Therefore, the description is omitted here, and only different portions from the first embodiment will be described.

  As shown in FIG. 7, the search condition setting unit 230 sets the selection of the attribute information used for the search and the output method of the search result as in the first embodiment. The second embodiment Then, an item “image composition” is added. Here, when “with composition” is selected, the image composition unit 240 sets an image to be composed. As a configuration, the image composition may be set directly on this setting screen. Since other processes and setting screens are the same as those in the first embodiment described above, description thereof will be omitted.

  For example, the image composition unit 240 selects what image is to be combined with the attribute information that has been “combined” in the search condition setting unit 230 on the composite image creation screen as illustrated in FIG. 8. Then, an image composition process is performed on the actual input image (the face area image detected by the face detection unit 220).

  The composite image creation screen of FIG. 8 includes an input image display unit 21 and a composite item display unit 22. In this example, “search mask setting” is set to “mask” in the search condition setting unit 230. In the category 23 of the composite item display unit 22, “mask” is displayed. If a plurality of “synthesizing” are set in the search condition setting unit 230, it is possible to synthesize under a plurality of conditions if the attribute type can be switched from the list.

  Furthermore, the composite item display unit 22 displays a partial area image of an image to be combined for the selected category in the form of an icon 24. By selecting an image to be synthesized from the icon 24 by operating a mouse (not shown) and dragging it to the input image (face area image detected by the face detection unit 220) 25 of the input image display unit 21, a mask image is obtained. Can be synthesized. Obviously, it is also possible to perform operations such as enlargement, reduction, and rotation by operating a mouse (not shown).

  The composite item display unit 22 is provided with a new item take-in button 26 in case the desired partial area does not exist or a predetermined image is input from the outside. When the button 26 is pressed, an item can be added by reading an image file separately. At this time, the image may be displayed so that a partial area of the image can be cut out, and the partial area cut out by the mouse operation may be taken in as an item.

  In FIG. 8, 27 is a delete button, 28 is a confirm button, 29 is a cancel button, and these are displayed on the composite item display unit 22. Reference numeral 30 denotes a mask image to be combined with the input image 25, which is combined by dragging with a mouse operation (not shown).

The composite image management unit 245 manages the partial area image combined by the image composition unit 240 as a database. Each attribute is classified as a “category”, and images to be combined are managed for each category.
Note that, on the composite image creation screen of the image composition unit 240, items can be deleted or newly added by operating the delete button 27 or the new item take-in button 26.

  Since the attribute search unit 272 performs the setting as shown in FIG. 7 by the search condition setting unit 230, the search result (face feature information) using the attribute feature information of the mask other than the search using the face feature information. And a search result for an image synthesized by the image synthesis unit 240 is obtained.

  In this embodiment, when the search result in the composite image is obtained in the attribute search unit 272, the search is performed using the attribute feature information of the mask extracted from the composite image. Switching to the feature information may be performed to search the entire face image.

  The output unit 280 is different from the first embodiment described above in that a composite image is displayed when there is an image composition, and the rest is the same as in the first embodiment, and thus the description thereof is omitted here.

With the configuration as described above, the overall processing flow according to the second embodiment will be described with reference to the flowchart shown in FIG.
In the following description, it is assumed that the setting contents in the search condition setting unit 230 are as in the example shown in FIG. Therefore, in this case, in addition to the search using the face feature information, a search using the mask feature information is performed, and a search is also performed on the synthesized image, and the search result is displayed.

  In FIG. 9, when an image is input by the image input unit 210 (step S21), the face detection unit 220 detects a face area from the image input by the image input unit 210 (step S22). Thereafter, three search results are calculated.

  First, the search result processing procedure using facial features will be described. The face feature extraction unit 251 extracts face feature information from the face area detected by the face detection unit 220 (step S23). Next, the face search unit 271 compares the face feature information extracted by the face feature extraction unit 251 with the face feature information registered in the person information management unit 260, thereby making the similarity of both face feature information. A plurality of corresponding face images determined in advance from the one with the highest similarity are searched from the person information management unit 260 (step S24), and the search result is output via the search result management unit 280. For example, as shown in FIG. 10, the search result of the face image using the face feature information is displayed (step S25).

  Next, the processing procedure of the search result using the mask attribute will be described. The attribute feature extraction unit 252 extracts attribute feature information as a mask feature from the face area detected by the face detection unit 220 (step S26). Next, the attribute search unit 272 compares the attribute feature information extracted by the attribute feature extraction unit 252 with the attribute feature information registered in the person information management unit 260, thereby making the similarity between the two attribute feature information. And a plurality of corresponding face images (face images of a person in a similar mask wearing state) determined in advance in descending order of similarity are searched from the person information management unit 260 (step S27). The search result is sent to the output unit 290 via the search result management unit 280, and the search result of the face image using the mask feature information is displayed as shown in FIG. 10, for example (step S28).

  In the case of this example, as shown in FIG. 7, “similarity integration” is “Yes” for the mask feature, so the search result using the face feature information in step S24 and the mask feature information in step S27 are used. The search results are displayed by integrating the ranks with the weights set in advance by the search condition setting unit 230.

  Finally, the processing procedure of the search result using the mask attribute by the composite image will be described. When the search condition setting unit 230 selects “with mask image synthesis” (in this example, “image synthesis” is “with synthesis” for the mask feature as shown in FIG. 7), the image synthesis unit 240 A mask image is synthesized with the input image (the face area image detected by the face detection unit 220) (step S29). Next, the attribute feature extraction unit 252 extracts the attribute feature information in the mask feature from the synthesized image synthesized by the image synthesis unit 240 (step S30). Next, the attribute search unit 272 compares the attribute feature information extracted by the attribute feature extraction unit 252 with the attribute feature information registered in the person information management unit 260, thereby making the similarity between the two attribute feature information. Is calculated, and a plurality of corresponding face images (face images of a person in a similar mask wearing state) predetermined in order from the highest similarity are searched from the person information management unit 260 (step S31), The search result is sent to the output unit 290 via the search result management unit 280, and for example, as shown in FIG. 10, the search result of the face image using the mask feature information based on the composite image is displayed (step S32).

  Note that the processes of steps S23, S26, and S29 (S30) may be performed sequentially or in parallel.

  The screen display example of FIG. 10 is basically the same as that of FIG. 5 in the first embodiment, but the image group 6 based on the search result of the age attribute information arranged below the image group 4 is synthesized. The difference is that the image group 12 is replaced by the search result of the mask attribute information with respect to the image, and that the composite image 13 is displayed between the image groups 6 and 12 and the setting information 10 and 11, and the others are as shown in FIG. It is the same.

  As described above, according to the second embodiment, the same effect as that of the first embodiment can be obtained, and the state of the input image and the attribute can be changed by synthesizing a predetermined partial region image. Even in this case, similar face images can be searched with high accuracy.

Next, a third embodiment will be described.
FIG. 11 schematically illustrates the configuration of a face image search apparatus according to the third embodiment. The face image search device according to the third embodiment includes an image input unit (image input unit) 310 including an ITV camera 309 that inputs an image including at least the face of the person M to be imaged, and an input input by the image input unit 310. A face detection unit (face detection means) 320 for detecting at least one face area from an image, a search condition setting unit (search for setting the type of attribute to be searched for setting search conditions, a search result output method, etc. (Condition setting means) 330, face feature extraction section (face feature extraction means) 351 for extracting facial feature information indicating personal features of the face obtained from the face area detected by the face detection section 320, and detected by the face detection section 320. Attribute feature extraction that extracts attribute feature information for identifying person attribute information (age, gender, presence / absence of glasses, presence / absence of mask, presence / absence of hat, race, etc.) obtained from the face area Unit (attribute feature extraction unit) 352, a person information management unit (person information management unit) 361 that manages a face image registered in advance, face feature information, attribute feature information, and the like corresponding thereto, one or more in advance Attribute-specific feature information management unit (attribute-specific feature information management means) 362 for managing attribute feature information of each attribute in order to determine the attribute of the attribute, attribute feature information extracted by the attribute feature extraction unit 352 and attribute-specific feature information management The attribute feature information managed by the unit 362 is compared to determine which attribute the face image input by the image input unit 310 corresponds to, and the determination result is recorded in the person information management unit 361. By the face feature information and person information management unit 361 for the input image extracted by the attribute discrimination unit (attribute discrimination unit) 353 and the face feature extraction unit 351 By comparing with the face feature information that is managed, the similarity between the face feature information in the face feature information that is managed by the person information management unit 361 and the face feature information for the input image is most similar The person information management unit 361 searches for a plurality of corresponding face images determined in advance in descending order of similarity, and is managed by a person information management unit 361. The search condition set by the search condition setting unit 330 is compared with the attribute feature information automatically determined using the image by the attribute determination unit 352 and the search condition set by the search condition setting unit 330. Determines which attribute feature information in the attribute feature information managed by the person information management unit 361 corresponds, and a face image corresponding to the attribute feature information The first attribute search unit (first attribute search means) 372 for searching the personal information management unit 361, the attribute feature information as a manually provided character string managed by the personal information management unit 361, and the search condition By comparing with the search condition set by the setting unit 330, the search condition set by the search condition setting unit 330 corresponds to which attribute feature information in the attribute feature information managed by the person information management unit 361. A second attribute search unit (second attribute search unit) 373 that searches the person information management unit 361 for a face image corresponding to the corresponding attribute feature information, and first and second attribute search units The output unit (output) that outputs each search result obtained by 372 and 373 and the search result obtained by the face search unit 371 in accordance with the contents set by the search condition setting unit 330 Configured with a means) 380.

  The image input unit 310, the face detection unit 320, the face feature extraction unit 351, the attribute feature extraction unit 352, and the face search unit 371 are the same as those in the first and second embodiments described above. Then, description is abbreviate | omitted and only a different part from 1st, 2nd embodiment is demonstrated.

  As shown in FIG. 12, the search condition setting unit 330 determines whether to search for a plurality of attribute information regarding the search condition, whether to display the search result, the class of the attribute to be searched, the search target, Use a search (first attribute search unit 372) that uses attribute information automatically given using attribute feature information of an image, whether to select an attribute class that matches or does not match It is set whether to use the search using the attribute information manually given (second attribute search unit 373).

  In this embodiment, for the sake of simplicity, an example using age and gender is used. However, even if an attribute other than the attributes described in FIG. 12 is selected, the target attribute can be selected in the same manner. . For example, “No mask”, “White solid mask”, “White plane mask”, “Blue mask” for masks, “No glasses”, “Glasses with transparent lenses”, “Sunglasses”, and hats for glasses Search by classifying each attribute in advance, such as “no hat”, “baseball cap”, “top hat”, “hunting hat”, “helmet”, “beret”, etc. Selection may be made by the condition setting unit 330.

  In this embodiment, when there are a plurality of attribute information, the search is performed individually. However, the result of logical product (AND) of the search with a plurality of attribute information and the logical sum (OR) are obtained. A setting screen may be prepared for narrowing down the result of the determination or a combination of logical product and logical sum.

  The person information management unit 361 is a database to be used when searching by the face search unit 371 and the first and second attribute search units 372 and 373, and assigns a person ID to each individual to be searched. The face feature information extracted by the face feature extraction unit 351, the attribute feature information automatically obtained by the image feature used by the first attribute search unit 372, and used by the second attribute search unit 373 A character string (class name) indicating attribute feature information given manually is managed in association with each person ID. FIG. 13 shows a configuration example of the management information.

  The contents of the face feature information are the same as those in the first embodiment, and are therefore omitted. Here, the class name is a character string, but an identification index (Index) indicating the class instead of the character string may be provided as a numerical value.

  As for the attribute feature information using the feature information of the image, the attribute discrimination result is automatically recorded at the stage of the registration operation in the person information management unit 361 before the stage of performing the search. However, regarding the manual attribute feature information, it is assumed that the attribute feature information is not given unless a value is explicitly input by the user of this apparatus, and the item described as “N / A” in FIG. As described above, it is possible to manage without “attribute information” and add it later. As a method of adding later, although it is possible to add on a GUI basis based on the search result displayed on the output unit 380, it is possible to separately import an electronic file in which the person ID and attribute information are associated with each other. Alternatively, it may be manually given on the registered information list display screen.

  The attribute-specific feature information management unit 362 manages attribute feature information representing each of the attributes to be determined. For example, in the case of gender, each piece of attribute feature information is managed in order to identify male and female. Each attribute feature information is acquired using the same method as the attribute feature extraction process, but in order to suppress variation among individuals, as many face images as possible are collected for each attribute, and each attribute attribute information is collected for each attribute. It can be created by obtaining average attribute feature information.

  A specific method may be a method of creating a partial space using a plurality of images described in the face feature extraction unit 151 and the attribute feature extraction unit 152 of the first embodiment. Similarly, the age is in units of 10 years, the mask is “no mask”, “white solid mask”, “white flat mask”, “blue mask”, the glasses are “no glasses”, and “transparent lens glasses”. , "Sunglasses" and hats are also classified into multiple categories such as "no hat", "baseball cap", "top hat", "hunting hat", "helmet", and "beret". It is only necessary to collect a large number of images and create attribute feature information representing them.

  The attribute determination unit 353 compares the attribute feature information extracted by the attribute feature extraction unit 352 with the attribute feature information managed by the attribute-specific feature information management unit 362, so that the face input by the image input unit 310 It is determined which attribute the image corresponds to, and the determination result is registered (recorded) in the person information management unit 361.

  Specifically, the process is the same until the attribute feature information is obtained by the method of extracting the attribute feature information described in the first embodiment and the second embodiment, but the processing based on the flowchart shown in FIG. 14 is performed. Thus, the similarity between the attribute feature information (age and sex) extracted from the input image by the attribute feature extraction unit 352 and the representative feature information (dictionary) by attribute managed by the attribute feature information management unit 362 is obtained. Thus, it is determined that the attribute belongs to the attribute having the closest similarity.

  For example, in the case of gender, a similarity between attribute feature information (gender feature information) obtained from an input image is obtained for male feature information (male dictionary) and female feature information (female dictionary), The higher similarity may be output as the attribute discrimination result for the input image.

  That is, for example, as in the technique described in Japanese Patent Application Laid-Open No. 2010-44439, a feature amount holding the frequency of occurrence of local gradient features of the face as statistical information is used, and the statistical information is the most A technique for discriminating two classes such as men and women by selecting a gradient feature to be identified, obtaining a discriminator for identifying the feature by learning, and the like can be applied.

  If the attribute is not 2 classes as in gender discrimination but more than 3 classes as in age estimation, create an average facial feature dictionary for each class (here age), It can be easily realized by finding the similarity and selecting the class with the maximum value, but as a technique for estimating the age with higher accuracy, the age is estimated by the following method using the above-mentioned 2-class classifier Is possible.

  First, in order to estimate the age, a face image for each age to be distinguished is prepared by being divided into classes to be identified in advance. For example, when it is desired to discriminate between the ages of 10 and around 60, facial images from under 10 to over 60 are prepared in advance. The more you collect the number of preparations, the better the accuracy, and the wider the age you can distinguish by collecting a wide range of ages.

  Next, a discriminator for discriminating whether “above or below the reference age” is prepared. As a method used for two-class discrimination, a standard method such as linear discriminant analysis can be realized. However, in this embodiment, Support Vector Machine (hereinafter abbreviated as “SVM”), Reference 6 (Takio Kurita: “Support”). Use the “Introduction to Vector Machines”).

  In SVM, if two classes to be discriminated are given, a boundary condition for discriminating the two classes can be set and an index of how far away from the boundary can be obtained. By classifying face images that belong to an age older than N years old and face images that belong to a lower age, it is possible to determine whether the age is higher than or lower than a reference age.

  For example, when 30 years old is set as the reference age, an image group is prepared for determining whether it is above or below 30 years old, and an image including 30 years or older is input as an image of the positive class “30 years old or older”. Then, an image of a negative class “under 30 years old” is prepared and learning is performed with SVM. Similarly, by classifying images prepared while shifting the reference age from 10 to 60 according to the reference age, SVM learners can prepare for the number of reference ages (in this embodiment, 10). The number is 6 from age to 60, but the age range and the unit for shifting the reference age may be changed freely).

  By always learning a class that is “above the reference age” as a “positive” class, the index returns a positive value when an image that is older than the reference age is input. become. By executing this discrimination process while shifting the reference age from 10 years old to 60 years old, it is possible to obtain an index that is higher or lower than the reference age, and among these output indices, The place where the index is close to zero is close to the age to be output. Here, an example of the age estimation method is shown in FIG.

  After the output value of SVM for each reference age is obtained, the output value is plotted assuming a graph with the output value on the vertical axis and the horizontal axis on the reference age. In order to obtain the age at which the output value is considered to be close to zero for this plot, the following method can be used.

  As a first method, a plot closest to zero is selected. In the case of FIG. 15, since the reference age of 30 years is the closest to zero, it can be returned as “30s”. When the plot is unstable and fluctuates up and down, the age can be returned stably by taking a moving average with the adjacent reference age.

  As a second method, an approximate function is obtained based on the output plot, and the value on the horizontal axis when the function intersects the vertical axis is used as the estimated age. As shown in FIG. 15, by approximating with a straight line of a linear function, the age of about 33 years of intersection can be output. Rather than obtaining one approximation function for all plots when obtaining an approximation function, multiple approximation functions for a subset (for example, plots for three adjacent reference ages) are obtained, and this corresponds to the function with the smallest approximation error. A method of outputting the intersection with the horizontal axis is also possible.

FIG. 16 shows a flowchart summarizing the age estimation method described above.
In addition to the above, a method of discriminating a class with a value obtained through a predetermined conversion function may be used.

  The results obtained here are recorded in the person information management unit 361, so that attribute determination processing for the information in the person information management unit 361 will not be performed again in the future when performing attribute search using image information. Can also be good.

  In this embodiment, when the search condition is set, it is checked whether or not there is information in the person information management unit 361. However, when a face image is registered in advance, all attributes are automatically set. On the other hand, if an attribute discrimination result using an image feature is obtained and recorded, such a process does not have to be performed each time a search is performed.

  The first attribute search unit 372 performs a search condition on the attribute information automatically recorded using the attribute feature information of the image managed in association with each person ID by the person information management unit 361. The search is performed according to the search condition set in the setting unit 330.

  In this embodiment, “age” that matches “30's” is searched based on the result of automatic attribute information discrimination by image processing. Therefore, as shown in FIG. A face image matching “30's” recorded in “age” of “attribute information (automatic)” managed in 361 is output as a search result. In this example, face images (persons) with person IDs “0002” and “0003” are output as search results.

  The second attribute search unit 373 sets the search condition set in the search condition setting unit 330 for the manually provided attribute feature information managed in association with each person ID in the person information management unit 361. Also search.

  In this embodiment, “gender” that matches “male” is searched for “manually assigned”, and thus is managed by the person information management unit 361 as shown in FIG. A face image matching “male” recorded in “sex” of “attribute information (manual)” is output as a search result. In this example, face images (persons) with person IDs “0002”, “0003”, and “0004” are output as search results.

In the attribute information (manual) of the person information management unit 361 to be searched by the second attribute search unit 373, the attribute feature information is not recorded unless the user of the apparatus manually assigns it. Since it is “N / A”, it is excluded from the search target here.
When the search condition setting unit 330 sets the search target to “N / A”, only “N / A” can be output as the search result.

  The output unit 380 displays and outputs the search results obtained by the face search unit 371 and the search results of the first and second attribute search units 372 and 373 according to the contents set by the search condition setting unit 330. .

With the configuration as described above, an overall processing flow according to the third embodiment will be described.
First, the flow of processing when a person (face image, attribute information) is registered in the person information management unit 361 will be described with reference to the flowchart shown in FIG.

  When an image to be registered is input by the image input unit 310 (step S41), the face detection unit 320 detects a face area from the image input by the image input unit 310 (step S42). Next, the face feature extraction unit 351 extracts face feature information from the face area detected by the face detection unit 320, and performs a search using the feature of the entire face together with the face image in the extracted face area. It is registered in the person information management unit 361 as face feature information (step S43).

  At the same time or at a predetermined timing, the attribute feature extraction unit 352 extracts attribute feature information that can be extracted from the face area detected by the face detection unit 320 (step S44). Next, by comparing the attribute feature information extracted by the attribute feature extraction unit 352 with the attribute feature information managed by the attribute-specific feature information management unit 362, which face image is input by the image input unit 310 It is discriminated whether it corresponds to the attribute, and the discrimination result is registered in the attribute feature information automatically given by the person information management unit 361 (step S45).

  By such registration processing, it is possible to reduce the work of inputting manually when performing facial image registration processing, and it becomes possible to register personal information efficiently.

Next, the flow of processing when searching for a registered person (face image) will be described with reference to the flowchart shown in FIG.
In the following description, it is assumed that the setting content in the search condition setting unit 330 is as in the example shown in FIG.

  When an image to be searched is input by the image input unit 310 (step S51), the face detection unit 320 detects a face area from the image input by the image input unit 310 (step S52). Next, the face feature extraction unit 351 extracts face feature information for search using the feature of the entire face from the face area detected by the face detection unit 320 (step S53).

  Next, the face search unit 371 compares the facial feature information extracted by the facial feature extraction unit 351 with the facial feature information registered in the person information management unit 361, thereby making the similarity of both facial feature information. A plurality of corresponding face images predetermined in order from the one with the highest similarity are searched from the person information management unit 361 (step S54), and the search result is sent to the output unit 380. For example, FIG. As shown, the search result of the face image using the face feature information is displayed (step S55).

  On the other hand, if there is attribute information set by the search condition setting unit 330, the search is narrowed down using the attribute information. In this case, the search condition setting unit 330 can select “manual” or “automatic” as a target attribute search.

  First, when “automatic” is selected as the target attribute search, the first attribute search unit 372 searches the attribute feature information automatically determined from the input image managed by the person information management unit 361. By comparing with the search condition set by the condition setting unit 330, which attribute feature information in the attribute feature information managed by the person information management unit 361 is the search condition set by the search condition setting unit 330. The face information corresponding to the corresponding attribute feature information is searched from the person information management unit 361 (step S56), and the search result is sent to the output unit 380. For example, as shown in FIG. The search result of the face image using the attribute feature information given automatically is displayed (step S57).

  As for the result of searching using the feature information of the face image and the first attribute search result using the automatically assigned attribute feature information, manual attribute feature information may not be assigned. For example, the attribute registration / edit button 14 shown in FIG. 19 is validated, an image as a search result is selected, and the attribute registration / edit button 14 is pressed, so that “attribute information (manual)” of the person information management unit 361 is displayed. Attribute feature information may be added to or modified by editing.

  Next, when “manual” is selected as the target attribute search, the second attribute search unit 373 includes attribute feature information as a manually provided character string managed by the person information management unit 361 and By comparing the search condition set by the search condition setting unit 330 with which attribute information in the attribute feature information managed by the person information management unit 361 the search condition set by the search condition setting unit 330 is. A face image corresponding to the attribute feature information is searched from the person information management unit 361 (step S58), and the search result is sent to the output unit 380. For example, as shown in FIG. The search result of the face image using the attribute feature information given in is displayed (step S59).

  As described above, in the case of attribute feature information for which “automatic” is selected, the search can be performed using the attribute feature information automatically determined at the time of registration as described above. It is possible to narrow down the search without doing so. By using the first attribute search result output as the search result, it is possible to manually register the attribute feature information.

  For example, looking at the image displayed by searching under the condition that the age is 30 years old, if it is certainly 30 years old, based on the result, it is newly added to the personal information management unit 361 as “attribute feature information manually assigned”. You can register. Since the attribute feature information is information input through visual confirmation, it can be considered that the probability that the attribute feature information is correct is high. As such operations are repeated, the information manually added to the attribute feature information gradually increases, and the search accuracy can be gradually increased.

  The screen display example in FIG. 19 is basically the same as that in FIG. 5 in the first embodiment, but the image group 6 based on the search result of the age attribute information arranged in the lower part of the image group 4 is The image group 8 based on the search result of the mask attribute information arranged in the lower part of the image group 4 is replaced with the image group 15 based on the search result of the first attribute search unit 372, and the image based on the search result of the second attribute search unit 373 The point that the group 16 is replaced with the attribute registration / edit button 14 is the same as FIG.

  In the third embodiment, an example in which age and gender are separated is shown. However, a plurality of pieces of attribute information may be combined with logical product (AND) or logical sum (OR). . Also, only the search results that use the face feature information are extracted by the first attribute search unit 372 and the second attribute search unit 373 that match the search conditions, and the order of the search is arranged in the order of the face search results. However, the arrangement order may be displayed independently of the result of the face search unit 371 as in the first embodiment.

  As described above, according to the third embodiment, a search using attribute feature information can be performed even if attribute feature information for a person is not given in advance, and the search result is used. It is possible to easily add attribute information to person information.

  In the above embodiment, the attribute information of a person is age, sex, presence / absence of glasses, presence / absence of a mask, presence / absence of a hat, race, etc., but besides these, the type of glasses, the type of mask, the type of hat , Beard, mole, wrinkle, injury, hairstyle, hair color, clothing color, clothing shape, hat, ornaments, face wear, facial expression, wealth, at least one of these One attribute information may be used.

  As described above, according to the above-described embodiment, when searching for a face image similar to the input face image from a plurality of face images registered in advance, a face similar to the input face image. In addition to displaying a list of image candidates, it is possible to visually confirm similar candidates from the viewpoint of attribute information such as age, gender, and wearing items instead of identifying individuals. From this viewpoint, a list of similar face images can be displayed for visual confirmation.

  In addition, if there is an accessory such as a mask or sunglasses on the face, there may be a decrease in accuracy when searching for facial features for the purpose of personal identification, but the ability to synthesize attribute information using images By providing the function, it is possible to provide a function for preventing a decrease in accuracy.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  M ... person, 109, 209, 309 ... ITV camera, 110, 210, 310 ... image input unit (image input means), 120, 220, 320 ... face detection unit (face detection means), 130, 230, 330 ... search Condition setting unit (search condition setting means), 151, 251, 351... Face feature extraction unit (face feature extraction means), 152, 252, 352... Attribute feature extraction unit (attribute feature extraction means), 171, 271, 371. Face search section (face search means), 172, 272 ... Attribute search section (attribute search means), 160, 260, 361 ... Person information management section (person information management means), 180, 280 ... Search result management section (search results) Management means), 190, 290, 380... Output section (output means), 240... Image composition section (image composition means), 245... Composite image management section (composite image management means), 353. Part (attribute discriminating means), 362 ... attribute-specific feature information management part (attribute-specific feature information management means), 372 ... first attribute search part (first attribute search means), 373 ... second attribute search part ( Second attribute search means).

Claims (13)

An image input means for inputting an image including at least a person's face image;
Face detection means for detecting at least one face region from an input image input by the image input means;
Search condition setting means for setting the type of attribute to be searched for setting search conditions and the output method of search results;
Facial feature extraction means for extracting facial feature information indicating personal features of a face obtained from the face area detected by the face detection means;
Attribute feature extraction means for extracting at least one attribute feature information of a person obtained from the face area detected by the face detection means;
Personal information management means for managing the face image registered in advance and the face feature information and attribute feature information corresponding to the face image;
By comparing the facial feature information for the input image extracted by the facial feature extraction unit with the facial feature information managed by the personal information management unit, the facial feature information for the input image is obtained by the personal information management unit. The similarity of which facial feature information in the managed facial feature information is similar is obtained, and a plurality of corresponding face images determined in advance in descending order of similarity are obtained from the person information management means. A face search means for searching;
The similarity between both attribute feature information is obtained by comparing the attribute feature information for the input image extracted by the attribute feature extraction means with the attribute feature information managed by the person information management means, and the similarity is high. Attribute search means for searching a plurality of corresponding face images predetermined in order from the person information management means,
A plurality of search results obtained by the attribute search means, an optimum search result automatically selected from the plurality of search results as necessary, and a search result obtained by the face search means Search result management means for selecting a search result that matches the content set by the search condition setting means from among,
Output means for outputting the search results selected by the search result management means;
A face image retrieval apparatus comprising:   The search condition setting means can select the type of attribute information to be searched, and each attribute information is in descending order of similarity to the attribute feature information obtained from the input face image. Whether it is in order of low or high, attribute feature information obtained from the input face image or not, or the similarity of the face image based on personal face feature information and one or more attribute feature information Either the search results are output by integrating the similarities, the search results based on the attribute feature information are output independently, or more similar to the face image input from the information managed by the person information management means 2. The face image retrieval apparatus according to claim 1, wherein whether or not to automatically select and output a search result said to be output can be switched. An image input means for inputting an image including at least a person's face image;
Face detection means for detecting at least one face region from an input image input by the image input means;
Search condition setting means for setting the type of attribute to be searched for setting search conditions and the output method of search results;
Combined image management means for managing a partial area image for combining a predetermined partial area image with the input image input by the image input means;
An image obtained by synthesizing the partial area image managed by the synthesized image management unit with respect to the attribute information in which the image synthesis is selected for the input image input by the image input unit based on the condition set by the search condition setting unit Image composition means for creating
Facial feature extraction means for extracting facial feature information indicating personal features of a face obtained from the face area detected by the face detection means;
Attribute feature extraction means for extracting at least one attribute feature information of a person obtained from the face area detected by the face detection means;
Personal information management means for managing the face image registered in advance and the face feature information and attribute feature information corresponding to the face image;
By comparing the facial feature information for the input image extracted by the facial feature extraction unit with the facial feature information managed by the personal information management unit, the facial feature information for the input image is obtained by the personal information management unit. The similarity of which facial feature information in the managed facial feature information is similar is obtained, and a plurality of corresponding face images determined in advance in descending order of similarity are obtained from the person information management means. A face search means for searching;
By comparing the attribute feature information for the input image extracted by the attribute feature extracting means or the attribute feature information for the image synthesized by the image synthesizing means with the attribute feature information managed by the person information managing means. Attribute search means for obtaining similarity of attribute feature information and searching for a plurality of corresponding face images determined in advance in descending order of similarity from the person information management means;
A plurality of search results obtained by the attribute search means, an optimum search result automatically selected from the plurality of search results as necessary, and a search result obtained by the face search means Search result management means for selecting a search result that matches the content set by the search condition setting means from among,
Output means for outputting the search results selected by the search result management means;
A face image retrieval apparatus comprising:   The search condition setting means can select the type of attribute information to be searched, and each attribute information is in descending order of similarity to the attribute feature information obtained from the input face image. Whether it is in order of low or high, attribute feature information obtained from the input face image or not, or the similarity of the face image based on personal face feature information and one or more attribute feature information Information managed by the person information management means whether to output the search result by integrating the similarities, to output the search result by the attribute information independently, to synthesize a predetermined image with the input image or not to synthesize 4. It is possible to switch whether to automatically select and output a search result that is more similar to a face image input from among the images. face Image retrieval apparatus.   The image synthesizing means is configured to select either a partial region image selected from partial region images prepared in advance according to a plurality of attribute information or a partial region image designated from a separately prepared image. 4. The face image search apparatus according to claim 3, wherein the face image search apparatus can synthesize the image input by the image input means.   The search result management means obtains a search result for at least one of the attribute information managed by the person information management means, and for a search result whose similarity to the high-order candidate of the obtained search result is low 2. An optimum search result selecting unit that obtains an index as to whether the search result is a superior search result and outputs a number of search results set in advance by the search condition setting unit in descending order of the index. The face image search device according to claim 1 or 3. An image input means for inputting an image including at least a person's face image;
Face detection means for detecting at least one face region from an input image input by the image input means;
Search condition setting means for setting the type of attribute to be searched for setting search conditions and the output method of search results;
Facial feature extraction means for extracting facial feature information indicating personal features of a face obtained from the face area detected by the face detection means;
Corresponds to pre-registered face image and face feature information for it, first attribute feature information automatically given using image feature, and second attribute feature information given manually Personal information management means to manage,
By comparing the facial feature information for the input image extracted by the facial feature extraction unit with the facial feature information managed by the personal information management unit, the facial feature information for the input image is obtained by the personal information management unit. The similarity of which facial feature information in the managed facial feature information is similar is obtained, and a plurality of corresponding face images determined in advance in descending order of similarity are obtained from the person information management means. A face search means for searching;
By comparing the first attribute feature information managed by the person information management means with the search condition set by the search condition setting means, the search condition set by the search condition setting means is It is determined which first attribute feature information among the first attribute feature information managed by the information management means corresponds to, and the face information corresponding to the corresponding first attribute feature information is the person information management First attribute search means for searching from the means;
By comparing the second attribute feature information managed by the person information management means with the search condition set by the search condition setting means, the search condition set by the search condition setting means is It is determined which second attribute feature information in the second attribute feature information managed by the information management means corresponds to the face information corresponding to the corresponding second attribute feature information, and the person information management Second attribute search means for searching from the means;
An output means for outputting the search results obtained by the face search means and the search results obtained by the first and second attribute search means in accordance with the contents set by the search condition setting means;
A face image retrieval apparatus comprising: Attribute feature extraction means for extracting at least one attribute feature information of a person obtained from the face area detected by the face detection means;
Attribute-specific feature information management means for managing attribute feature information of each attribute in order to discriminate one or more attributes in advance;
By comparing the attribute feature information extracted by the attribute feature extraction unit with the attribute feature information managed by the attribute-specific feature information management unit, which attribute the face image input by the image input unit corresponds to Attribute determination means for determining whether or not to register the result of the determination as the first attribute feature information in the person information management means;
The face image search device according to claim 7, further comprising:   The second attribute feature information managed by the person information management means is registered by manual operation based on the search result of the first attribute search means output by the output means. The face image search device according to claim 7 or 8.   The search condition setting means can select the type of attribute information to be searched, and for each attribute information, select attribute information to be used as a search target, and match the selected attribute information. It is possible to select whether to search for what is to be searched or for what does not match, and whether to use the first attribute search means or the second attribute search means for the selected attribute information. The face image search device according to claim 7. An image input step of inputting an image including at least a human face image by an image input means;
A face detection step of detecting at least one face region by the face detection means from the input image input in the image input step;
A search condition setting step for setting the type of attribute to be searched for setting the search condition and the output method of the search result by the search condition setting means,
A facial feature extraction step of extracting facial feature information indicating personal features of the face obtained from the facial region detected by the facial detection step by facial feature extraction means;
An attribute feature extraction step of extracting at least one attribute information of a person obtained from the face area detected by the face detection step by an attribute feature extraction unit;
The face managed by the personal information managing means for managing the face feature information for the input image extracted in the face feature extraction step, the face image registered in advance, and the face feature information and attribute feature information for the face image. By comparing the feature information with the face search means, the similarity of which face feature information in the face feature information managed by the person information management means is similar to the face feature information for the input image is determined. A face search step of searching for a plurality of corresponding face images determined in advance in descending order of similarity from the person information management means;
The attribute feature information for the input image extracted in the attribute feature extraction step and the attribute feature information managed by the person information management means are compared by the attribute search means to determine the similarity between the two attribute feature information, and the similarity An attribute search step of searching the person information management means for a plurality of corresponding face images predetermined in order from the highest degree;
A plurality of search results obtained by the attribute search step, an optimum search result automatically selected from the plurality of search results as necessary, and a search result obtained by the face search step A search result management step for selecting a search result that matches the content set by the search condition setting step from among the search result management means;
An output step for outputting the search result selected in the search result management step by an output means;
A face image search method characterized by comprising: An image input step of inputting an image including at least a human face image by an image input means;
A face detection step of detecting at least one face region by the face detection means from the input image input in the image input step;
A search condition setting step for setting the type of attribute to be searched for setting the search condition and the output method of the search result by the search condition setting means,
With respect to the attribute information in which image composition is selected for the input image input by the image input step based on the conditions set by the search condition setting step, a predetermined value is set for the input image input by the image input step. An image composition step for creating an image by the image composition means by combining the partial area images managed by the composite image management means for managing the partial area images for composing the partial area images;
A facial feature extraction step of extracting facial feature information indicating personal features of the face obtained from the facial region detected by the facial detection step by facial feature extraction means;
An attribute feature extraction step of extracting at least one attribute feature information of a person obtained from the face area detected by the face detection step by an attribute feature extraction unit;
The face managed by the personal information managing means for managing the face feature information for the input image extracted in the face feature extraction step, the face image registered in advance, and the face feature information and attribute feature information for the face image. By comparing the feature information with the face search means, the similarity of which face feature information in the face feature information managed by the person information management means is similar to the face feature information for the input image is determined. A face search step of searching for a plurality of corresponding face images determined in advance in descending order of similarity from the person information management means;
The attribute feature information for the input image extracted by the attribute feature extraction step or the attribute feature information for the image synthesized by the image synthesis step and the attribute feature information managed by the person information management means are compared by the attribute search means. An attribute search step of obtaining a similarity between both attribute feature information by searching the plurality of corresponding face images determined in advance in descending order of similarity from the person information management means;
A plurality of search results obtained by the attribute search step, an optimum search result automatically selected from the plurality of search results as necessary, and a search result obtained by the face search step A search result management step for selecting a search result that matches the content set by the search condition setting step from among the search result management means;
An output step for outputting the search result selected in the search result management step by an output means;
A face image search method characterized by comprising: An image input step of inputting an image including at least a human face image by an image input means;
A face detection step of detecting at least one face region by the face detection means from the input image input in the image input step;
A search condition setting step for setting the type of attribute to be searched for setting the search condition and the output method of the search result by the search condition setting means,
A facial feature extraction step of extracting facial feature information indicating personal features of the face obtained from the facial region detected by the facial detection step by facial feature extraction means;
Face feature information for the input image extracted in the face feature extraction step, face image registered in advance, face feature information for the face image, and first attribute feature information automatically given using the image feature; By comparing the face feature information managed by the person information management means that manages the second attribute feature information given manually by the face search means, the face feature information for the input image becomes the person. The similarity of which facial feature information is similar to the facial feature information managed by the information management means is obtained, and a plurality of corresponding facial images determined in advance in descending order of similarity are said person. A face search step for searching from information management means;
Setting by the search condition setting step by comparing the first attribute feature information managed by the person information management means with the search condition set by the search condition setting step by the first attribute search means. It is determined which first attribute feature information in the first attribute feature information managed by the person information management means corresponds to the corresponding first attribute feature information. A first attribute search step of searching for a face image from the person information management means;
Setting by the search condition setting step by comparing the second attribute feature information managed by the person information management means with the search condition set by the search condition setting step by the second attribute search means It is determined which second attribute feature information in the second attribute feature information managed by the person information management means corresponds to the corresponding second attribute feature information. A second attribute search step of searching for a face image from the person information management means;
An output step for outputting the search results obtained by the face search step and the search results obtained by the first and second attribute search steps by output means in accordance with the contents set by the search condition setting step; ,
A face image search method characterized by comprising:

Images