CN106909870A - The search method and device of facial image - Google Patents

Abstract

The present invention provides a face image retrieval method and device, wherein the method includes: training the semantic features of the face images in the gallery to obtain the attribute features of the first number of face images; Refer to the specified semantic features of the face image for training to obtain the similarity features of the specified semantic features of the second quantity relative to the attribute features in the gallery; use the first quantity of attribute features and the second quantity of similarity features as each image in the gallery The feature vector of the face image; calculate the feature vector of the image to be retrieved and the feature vector of each image in the gallery through preset rules, and retrieve one or more images matching the image to be detected according to the calculation results. The present invention solves the problem of poor face retrieval effect caused by using low-level features for face recognition in the related art, and improves the efficiency and matching degree of face retrieval.

Description

Face Image Retrieval Method and Device

technical field

The invention relates to the field of face image recognition, in particular to a method and device for retrieving a face image.

Background technique

With the continuous development of society and the urgent requirements for fast and effective automatic identity verification, biometric technology has developed rapidly in recent decades. Among them, the research on face recognition technology has attracted a large number of researchers. Face recognition technology is widely used, such as assisting the public security department in criminal investigation and solving cases, automatic identity verification by machines, video surveillance tracking and recognition, facial expression analysis and so on. At present, many countries have carried out research on face recognition. The methods mainly focus on template matching, example learning, neural networks, methods based on hidden Markov models, and methods based on support vector machines.

In computer face recognition, the features obtained through simple processing of a large amount of image data can be defined as low-level features, while the description features such as lines, faces, and patterns can be defined as high-level features. Image Principal Component Analysis (PCA) features, wavelet transform features and some statistical features belong to the category of low-level features, while the results of face part shape analysis are high-level features. Using attributes such as male, female, smiling, black hair, and glasses for face recognition can achieve good results. In addition, face recognition can also be performed using similarity data with a certain face. The Outdoor Face Detection Library (Labeled Faces in the Wild, referred to as LFW) and the Public Figures Face Database of Columbia University (Pubfig for short) are two independent public data sets, the pictures of which are all in Africa. Obtained in a controlled environment. The difference in pose, expression, lighting, etc. in these two data sets will have a great impact on face recognition. Traditional methods in related technologies only use low-level features for face recognition, resulting in poor face retrieval results.

For the above-mentioned problems in the related art, there is no effective solution yet.

Contents of the invention

The present invention provides a face image retrieval method and device to at least solve the problem in the related art that low-level features are used for face recognition, resulting in poor face retrieval results.

The present invention provides a method for retrieving human face images, comprising: training the semantic features of human face images in a gallery to obtain attribute features of a first number of said human face images; The specified semantic features of the second quantity are trained to obtain the similarity features of the specified semantic features with respect to the attribute features in the gallery; the attribute features of the first quantity and the similarity features of the second quantity are used as The feature vector of each face image in the gallery; the matching degree between the feature vector of the image to be retrieved and the feature vector of each image in the gallery is calculated by preset rules, and the matching degree is retrieved according to the matching degree. One or more images that detect image matches.

Further, the training the semantic features of the face images in the gallery to obtain the attribute features of the first quantity of the face images includes: detecting key points in each face image in the gallery, wherein , the key points include: the four corners of the eyes, the tip of the nose, and the two ends of the mouth; according to the key points, the face image is divided into regions, and the underlying features of the face corresponding to different regions are extracted; according to the attributes The classifier classifies and learns the multiple underlying features of the human face in different regions to obtain different types of the first quantity of attribute features.

Further, the training of the specified semantic features of the first predetermined number of reference face images to obtain the similarity features of the specified semantic features of the second number with respect to the attribute features in the gallery includes: Refer to the key points of the face image for detection, wherein the key points include: the four corners of the eyes, the tip of the nose, and the two ends of the mouth; the specified semantic features are extracted according to the key points to obtain the corresponding The data set is extracted to obtain the underlying features of the face corresponding to different regions; the data set is classified and learned according to the similarity classifier to obtain the second number of similarity features.

Further, the attribute classifier and the similarity classifier include: a support vector machine (SVM) classifier.

Further, calculating the eigenvector of the image to be retrieved and the eigenvector of each image in the gallery through preset rules, and retrieving one or more images matching the image to be detected according to the calculation result includes: obtaining the The feature vector of the image to be retrieved and the feature vector of each image in the gallery; the distance calculation is performed between the feature vector of the image to be retrieved and the feature vector of each image in the gallery, and the distance calculation method includes: Cosine distance method, Euclidean distance method; multiple calculation results are sorted according to the rule from large to small, and the face images corresponding to the second predetermined number of calculation results with the highest value are selected from the sorted calculation results as The matching image of the image to be retrieved.

According to another aspect of the present invention, there is provided a human face image retrieval device, including: a first semantic feature extraction module, used to train the semantic features of the human face images in the gallery to obtain the first number of said people The attribute feature of the face image; the second semantic feature extraction module is used to train the specified semantic features of the first predetermined number of reference face images to obtain the second quantity of the specified semantic features relative to the attribute features in the gallery Similarity feature; processing module, for using the attribute feature of the first quantity and the similarity feature of the second quantity as the feature vector of each face image in the gallery; retrieval module, for by preset The rule calculates the matching degree between the feature vector of the image to be retrieved and the feature vector of each image in the gallery, and retrieves one or more images matching the image to be detected according to the matching degree.

Further, the first semantic feature extraction module includes: a first detection unit for detecting key points in each face image in the gallery, wherein the key points include: four eye corners, nose tip and both ends of the mouth; the first processing unit is used to divide the face image into regions according to the key points, and extract the underlying features of the face corresponding to different regions; the second semantic feature extraction unit is used to According to the attribute classifier, classify and learn the plurality of human face underlying features in different regions to obtain the first number of attribute features of different types.

Further, the second semantic feature extraction module includes: a second detection unit, configured to detect key points of the first predetermined number of reference face images, wherein the key points include: four corners of both eyes, The tip of the nose and both ends of the mouth; the second processing unit is used to extract the specified semantic features according to the key points to obtain a data set corresponding to the specified semantic features, and to extract the underlying features of the face corresponding to different regions and a second semantic feature extraction unit, configured to classify and learn the data set according to a similarity classifier to obtain a second number of similarity features.

Further, the attribute classifier and the similarity classifier include: a support vector machine (SVM) classifier.

Further, the retrieval module includes: an acquisition unit for obtaining the feature vector of the image to be retrieved and the feature vector of each image in the gallery; a calculation unit for calculating the feature vector and the feature vector of the image to be retrieved The eigenvectors of each image in the gallery are subjected to distance calculation, and the distance calculation methods include: cosine distance method and Euclidean distance method; a retrieval unit is used to sort a plurality of calculation results according to the rules from large to small, And from the sorted calculation results, select the face images corresponding to the second predetermined number of calculation results with higher values as the matching images of the image to be retrieved.

In the present invention, the semantic features of the face image are trained to obtain the first number of attribute features, and in addition, the specified semantic features of the first predetermined number of reference face images are trained to obtain the second number of specified semantic features With respect to the similarity features of the attribute features in the gallery, the feature vectors of each image are obtained by obtaining the first number of attribute features and the second number of similarity features, and comparing the image to be detected with the feature vectors of the images in the gallery, To retrieve one or more images that match the image to be detected is to compare the feature vector of the image to be detected with the image in the gallery. The feature vector is composed of attribute features and similarity features, and attribute features and similarity The features are all high-level features, so the matching result has a high matching degree with the image to be detected, thus solving the problem of using low-level features for face recognition in related technologies, which leads to poor face retrieval results, and improving the face detection efficiency. Retrieval efficiency and matching degree.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

Fig. 1 is the flow chart of the retrieval method of face image according to the embodiment of the present invention;

Fig. 2 is the structural block diagram of the retrieval device of face image according to the embodiment of the present invention;

Fig. 3 is an optional structural block diagram 1 of a retrieval device for a face image according to an embodiment of the present invention;

Fig. 4 is an optional structural block diagram 2 of a retrieval device for a face image according to an embodiment of the present invention;

FIG. 5 is an optional structural block diagram three of a face image retrieval device according to an embodiment of the present invention;

Fig. 6 is a schematic diagram of face key point detection according to an optional embodiment of the present invention;

Fig. 7 is a schematic diagram of a coordinate system according to an optional embodiment of the present invention;

8a-8b are schematic diagrams of comparisons before and after rotational alignment of face images according to an optional embodiment of the present invention;

Fig. 9 is a schematic diagram of similarity image region segmentation according to an optional embodiment of the present invention;

Fig. 10 is a schematic diagram of attribute/similarity feature classifier learning and feature extraction process according to an optional embodiment of the present invention;

Fig. 11 is a schematic diagram of image storage and retrieval process according to an optional embodiment of the present invention.

detailed description

Hereinafter, the present invention will be described in detail with reference to the drawings and examples. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence.

A kind of retrieval method of human face image is provided in the present embodiment, and Fig. 1 is the flow chart of the retrieval method of human face image according to the embodiment of the present invention, as shown in Fig. 1, this process comprises the following steps:

Step S102: training the semantic features of the face images in the gallery to obtain the attribute features of the first number of face images;

Step S104: Train the specified semantic features of the first predetermined number of reference face images to obtain the second number of specified semantic features relative to the similarity features in the gallery;

Step S106: using the first number of attribute features and the second number of similarity features as feature vectors of each face image in the gallery;

Step S108: Calculate the matching degree between the feature vector of the image to be retrieved and the feature vector of each image in the gallery according to preset rules, and retrieve one or more images matching the image to be detected according to the matching degree.

From step S102 to step S108, it can be known that in this embodiment, at first, the semantic features of the face image are trained to obtain the first number of attribute features; in addition, the first predetermined number of specified semantic features of the reference face image Train to obtain the similarity features of the specified semantic features of the second number relative to the attribute features in the gallery, obtain the feature vector of each image through the obtained first number of attribute features and the second number of similarity features, and pass the image to be detected Compared with the feature vector of the image in the gallery, one or more images matching the image to be detected are retrieved, that is to say, in this embodiment, the feature vector of the image to be detected is compared with the image in the gallery, and the feature vector It is due to the composition of attribute features and similarity features, and both attribute features and similarity features belong to high-level features, so the matching result has a high matching degree with the image to be detected, thus solving the problem of using low-level features in related technologies to detect faces. Recognition, which leads to the problem of poor face retrieval effect, improves the efficiency and matching degree of face retrieval.

Regarding the method of training the semantic features of the face images in the gallery to obtain the attribute features of the first number of face images involved in the above step S102, in an optional implementation of this embodiment, it can be implemented in the following manner accomplish:

Step S102-1: Detect the key points in each face image in the gallery, where the key points include: the four corners of the eyes, the tip of the nose and the two ends of the mouth;

It should be noted that the key points mentioned above are only preferred key points of this optional embodiment and do not constitute a limitation to the present invention. For example, other key points: hair, chin, ears, etc. are all possible. In other words, as long as it is the features of the human face, it is acceptable.

Step S102-2: Divide the face image into regions according to the key points, and extract the underlying features of the face corresponding to different regions;

Step S102-3: According to the attribute classifier, classify and learn the multiple underlying features of human faces in different regions to obtain a first number of attribute features of different types.

In the above step S102-1 to step S102-3, in the specific application scenario of this embodiment, the first number can be selected as a value of 69, because the face attribute features include: male, female, smiling, black hair, wearing glasses Wait, these all describe the semantic features of the face, and the goal of the face attribute classifier is to classify the face image and judge whether the face image has a certain attribute, that is, it can be trained based on the above steps in specific application scenarios A classifier with 69 attribute features such as smiling, black hair, and wearing glasses is used to represent the face features, and the attribute feature extraction is to extract the attribute features of the face through the attribute feature classifier obtained through training, that is, through 69 trained attributes The classifier calculates 69 attribute values for the image, and concatenates them to form the attribute characteristics of the image.

Based on the above description, in an optional implementation manner of this embodiment, the methods involved in step S102-1 to step S103-3 may be:

First of all, the underlying features of the extracted attributes: perform face detection on each face image in the gallery, locate key points, obtain key point information of the face image, and rotate and align the images. The image is segmented according to the attribute requirements (for example, the region corresponding to the glasses attribute is the eye region, and the region corresponding to the white hair attribute is the hair region), and different attributes may need to be divided into different numbers of regions. For the segmented area, extract the underlying features that are valid for this attribute.

Then, divide the extracted underlying attribute features into two equal parts, half for training and half for testing (of course, this is just an example, other ratios are also possible, and can be divided according to the actual situation). If a certain attribute uses multiple segmentation regions, feature stitching needs to be performed first, such as whether to wear earrings or not, this attribute needs to use the left and right ear regions. Learn the corresponding SVM attribute classifiers for the underlying features, such as smiley face attribute classifiers, black hair attribute classifiers, glasses attribute classifiers, and a total of 69 attribute classifiers.

Finally, the attribute classification effect is verified against the classification value of the attribute classifier.

The manner involved in step S104 to train the specified semantic features of the first predetermined number of reference face images to obtain the similarity features of the second number of specified semantic features with respect to the attribute features in the gallery is optional in this embodiment. In the implementation mode, it can be realized in the following ways:

Step S104-1: Detect key points of the first predetermined number of reference face images, wherein the key points include: the four corners of both eyes, the tip of the nose, and both ends of the mouth;

Like the key points involved in the above step S102-1, the key points involved in the above are only the preferred key points of this optional embodiment, and do not constitute a limitation of the present invention, such as other key points: hair, chin, Ears etc. are fine. In other words, as long as it is the features of the human face, it is acceptable.

Step S104-2: Extract the specified semantic features according to the key points to obtain a data set corresponding to the specified semantic features, and extract the underlying features of the face corresponding to different regions;

Step S104-3: Perform classification learning on the data set according to the similarity classifier to obtain a second number of similarity features.

Wherein, the first predetermined number involved in the above may take a value of 10. Based on this, the above step S104-1 to step S104-4, in a specific application scenario, the training process of the similarity classifier is as follows:

First, select, for example, 10 reference persons, and process each reference person separately, take all the pictures of each reference person as positive samples, and select the same number of other face pictures as negative samples, and form a data set with reference persons as units set.

Then, process the extracted features for each data set as follows: first perform face detection and key point positioning, and rotate and align the pictures. Then, four sub-blocks of eyes, eyebrows, nose and mouth are segmented on each face picture, and the underlying features are extracted from the four sub-blocks. Transform this dataset into 4 new sub-datasets, namely eye dataset, eyebrow dataset, mouth dataset and nose dataset.

Finally, each sub-dataset of each reference person is divided, half of the data is used as training, and the other half is used as testing. The SVM model is learned on the training set, and the classification effect of the similarity classifier is verified on the test set. Save the model file and feature normalization file generated by training for subsequent similarity feature extraction.

It should be noted that the attribute classifier and the similarity classifier involved in this embodiment may be a support vector machine (Support Vector Machine, SVM for short) attribute classifier.

In addition, in another optional implementation of this embodiment, the feature vector of the image to be retrieved and the feature vector of each image in the gallery involved in step S108 are calculated according to the preset rules, and one or more images are retrieved according to the calculation result. The mode of multiple images matching the image to be detected can be realized in the following way:

Step S108-1: Obtain the feature vector of the image to be retrieved and the feature vector of each image in the gallery;

Step S108-2: Calculate the distance between the feature vector of the image to be retrieved and the feature vector of each image in the gallery;

Wherein, the distance calculation method may be a cosine distance method or a Euclidean distance method in this optional implementation manner.

Step S108-3: Sort the multiple calculation results according to the rule from large to small, and select the face images corresponding to the second predetermined number of calculation results with the highest values as the images to be retrieved from the sorted calculation results matching image.

In a specific application scenario, the method of the above step S108-1 to step S108-3 can be: after obtaining the feature vector composed of the attribute value and similarity value of the face image, it can be based on the combination of face attribute features and similarity Feature face retrieval, the obtained 69 attribute values and 40 similarity values are concatenated as the feature vector of each image, and then the Large Margin Nearest Neighbors (LMNN) algorithm is used to optimize each feature vector. dimension weight. The similarity between two faces can be calculated using feature vectors and weights. In this embodiment, the cosine (Cosine) of the angle between two vectors is used as the similarity value between the vectors. The value range of cosθ is [-1,+1], and the closer to +1, it represents the faces in the two pictures more similar.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to enable a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

In this embodiment, a device for retrieving human face images is also provided, and the device is used to implement the above embodiments and preferred implementation modes, and what has been explained will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

Fig. 2 is the structural block diagram of the retrieval device of face image according to the embodiment of the present invention, as shown in Fig. 2, this device comprises: the first semantic feature extraction module 22, is used for carrying out the semantic feature of the face image in the gallery Training obtains the attribute features of the first quantity of face images; the second semantic feature extraction module 24 is coupled and connected with the first semantic feature extraction module 22, and is used to train the specified semantic features of the first predetermined number of reference face images to obtain The similarity feature of the specified semantic features of the second quantity with respect to the attribute features in the gallery; the processing module 26, coupled with the second semantic feature extraction module 24, is used to combine the attribute features of the first quantity with the similarity of the second quantity Feature is used as the feature vector of each face image in the gallery; the retrieval module 28 is coupled with the processing module 26, and is used to calculate the matching of the feature vector of the image to be retrieved and the feature vector of each image in the gallery by preset rules degree, and according to the matching degree, one or more images matching the image to be detected are retrieved.

Fig. 3 is an optional structural block diagram one of a retrieval device of a human face image according to an embodiment of the present invention. As shown in Fig. 3 , the first semantic feature extraction module 22 includes: a first detection unit 32, which is used to search each image in the gallery The key points in the face image are detected, wherein the key points are the four corners of the eyes, the tip of the nose, and the two ends of the mouth; the first processing unit 34 is coupled with the first detection unit 32, and is used to detect the human face according to the key points. The image is divided into regions, and the bottom layer features of people's faces corresponding to different regions are extracted; the second semantic feature extraction unit 36 is coupled with the first processing unit 34, and is used to classify multiple people's faces in different regions according to the attribute classifier. The features are classified and learned to obtain different types of attribute features.

Fig. 4 is the optional structural block diagram two of the retrieval device of human face image according to the embodiment of the present invention, as shown in Fig. Detect with reference to the key points of the face image, wherein the key points include: the four corners of the eyes, the tip of the nose, and the two ends of the mouth; the second processing unit 44 is coupled to the second detection unit 42 for specifying semantics based on the key points The feature is extracted to obtain a data set corresponding to the specified semantic feature, and the underlying features of the face corresponding to different regions are extracted; the second semantic feature extraction unit 46 is coupled to the second processing unit 44, and is used to classify according to the similarity. Classification learning is performed on the dataset to obtain a second quantity of similarity features.

Optionally, the attribute classifier and the similarity classifier involved in this embodiment are support vector machine (SVM) classifiers.

Fig. 5 is the optional structural block diagram three of the retrieval device of face image according to the embodiment of the present invention, as shown in Fig. 5, retrieval module 28 comprises: acquisition unit 52, is used to obtain the feature vector of image to be retrieved and each in the gallery The feature vector of the image; the calculation unit 54 is coupled with the acquisition unit 52, and is used to calculate the distance between the feature vector of the image to be retrieved and the feature vector of each image in the gallery; the retrieval unit 56 is coupled with the calculation unit 54. The plurality of calculation results are sorted according to the rule from large to small, and the face images corresponding to the second predetermined number of calculation results with the highest value are selected from the sorted calculation results as matching images of the image to be retrieved.

It should be noted that each of the above-mentioned modules can be implemented by software or hardware. For the latter, it can be implemented in the following manner, but not limited to this: the above-mentioned modules are all located in the same processor; or, the above-mentioned modules are respectively located in multiple in the processor.

The present invention is illustrated below in conjunction with optional embodiments of the present invention;

This optional embodiment provides a method for extracting high-level semantic features of human faces. The method extracts high-level semantic features of human faces through attribute classifiers and similarity classifiers, and extracts high-level semantic features of human faces by combining human face attribute features and similarity features. Similarity measurement to achieve similar face retrieval, including face attribute classifier learning and face attribute feature acquisition methods, face similarity classifier learning and similarity feature acquisition methods, based on face attribute features and similarity features There are three parts of face retrieval, and the three parts are described in detail below:

1. Face attribute classifier learning and face attribute feature acquisition method;

(1) The process of learning the face attribute classifier: face attributes include: male, female, smiling, black hair, wearing glasses, etc., which describe the semantic features of the face. The goal of the face attribute classifier is to classify the face image and judge whether the face image has a certain attribute. In the patent of the present invention, a classifier with 69 attribute features such as smile, black hair, and glasses is trained for the representation of human face features. The attribute feature extraction is to extract the attribute feature of the face through the attribute feature classifier obtained through training. In this embodiment, 69 attribute values are calculated for the image through 69 trained attribute classifiers, and the image attribute feature is formed by splicing. .

(2) The training process of the attribute classifier is as follows:

First of all, it is necessary to obtain the labeled image of this attribute, and select a certain scale of positive samples and negative sample face pictures for each attribute (the sample images are obvious for this attribute) as the label set of this attribute.

Secondly, the underlying features of the attribute are extracted according to the following process: perform face detection on each face image in the annotation set, locate key points, obtain key point information of the face image, and rotate and align the images. The image is segmented according to the attribute requirements (for example, the region corresponding to the glasses attribute is the eye region, and the region corresponding to the white hair attribute is the hair region), and different attributes may need to be divided into different numbers of regions. For the segmented area, the underlying features (such as LBP, Gabor, etc.) that are valid for the area of the attribute are extracted.

Then, the underlying attribute features extracted from the positive and negative images in the previous annotation set are divided into two equal parts, one half is used for training and the other half is used for testing. If a certain attribute uses multiple segmentation regions, feature stitching needs to be performed first, such as whether to wear earrings or not, this attribute needs to use the left and right ear regions. Learn corresponding SVM attribute classifiers for the underlying features, such as smiley face attribute classifiers, black hair attribute classifiers, glasses attribute classifiers, etc., a total of 69 attribute classifiers, and generate feature normalization files on the training set. Save the model file and feature normalization file generated by training.

Finally, after the same normalization processing and feature splicing are performed on the test set, the attribute classification effect is verified according to the classification value of the attribute classifier.

(3) The face similarity classifier learning process includes:

The training goal of the similarity classifier is to train the facial features similarity classifier of the reference person, classify the new face image according to the facial features similarity classifier, and judge whether the facial features of the face are similar to those of the reference person.

(4) The training process of the similarity classifier is as follows:

First, select several (for example, 10) reference persons, and process each reference person separately, take all the pictures of each reference person as positive samples, and select the same number of other face pictures as negative samples, and use the reference persons as negative samples. Units make up a dataset.

Then, process the extracted features for each data set as follows: first perform face detection, locate key points, and rotate and align the images. Next, four sub-blocks of eyes, eyebrows, nose and mouth are segmented on each face picture, and the underlying features (such as LBP, Gabor, etc.) are extracted from the four sub-blocks respectively. Transform this dataset into 4 new sub-datasets, namely eye dataset, eyebrow dataset, mouth dataset and nose dataset.

Finally, each sub-dataset of each reference person is divided, half of the data is used as training, and the other half is used as testing. The SVM model is learned on the training set, and the classification effect of the similarity classifier is verified on the test set. Save the model file and feature normalization file generated by training for subsequent similarity feature extraction.

2. Extraction of attribute features and similarity features;

(1) The feature extraction process of attribute classifier and similarity classifier is similar, as follows:

First of all, for an input image, the process of extracting attribute features is as follows. First, the face image is detected according to the same process as in the attribute classifier training, and the key points are located to obtain the key point information of the face image. The image is rotated and aligned, and the image is segmented according to the requirements of each attribute, and the attribute classifier obtained by training is called to classify to obtain the value of the attribute classifier, and all the attribute classification values are spliced to obtain the face attribute characteristics of the input image.

Then, for an input image to extract similarity features, the specific process is similar to extracting attribute features, including: first, face detection is performed on the face image, key points are located, and the image is rotated and aligned. Then, four sub-blocks of eyes, eyebrows, nose and mouth are segmented on each face picture, and the underlying features are extracted from the four sub-blocks. Then call the trained similarity classifier to calculate the similarity values for the four sub-blocks, and splicing all the similarity values to obtain the face similarity features of the input image.

Finally, the attribute features and similarity features of the input image are concatenated, so that a feature vector composed of 69 attribute values and 40 similarity values of this image is obtained.

3. Face retrieval based on the combination of attribute features and similarity features;

After obtaining the feature vector composed of the attribute value and similarity value of the face image, face retrieval based on the combination of face attribute features and similarity features can be performed, as follows:

The obtained 69 attribute values and 40 similarity values are concatenated as the feature vector of each image, and then the weight of each dimension of the feature vector is optimized using the LMNN (Large Margin Nearest Neighbors) algorithm. The similarity between two faces can be calculated using feature vectors and weights. This patent uses the cosine (Cosine) of the angle between two vectors as the similarity value between the vectors. The value range of cosθ is [-1,+1]. The closer to +1, the more the faces in the two pictures are. resemblance.

Optional embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings;

Firstly, the technical solution adopted by each component of the algorithm in this optional embodiment is introduced. The technical solution includes: face key point detection, image preprocessing, image region segmentation, feature extraction, and classifier training.

(1) key point detection;

Fig. 6 is a schematic diagram of face key point detection according to an optional embodiment of the present invention. As shown in Fig. 6, in this optional embodiment, flandmark is used for fast face key point detection, and the detection points are the corners of the eyes, the tip of the nose and the two corners of the mouth. End these 7 key points.

(2) Image preprocessing;

Image preprocessing is the rotation and alignment of the original face image. According to the obtained data of the key points, the position information of the pupils of both eyes can be obtained by positioning. Since the pupils of the rotated face should be on a straight line, that is, the X values of the pupil coordinates should be equal, and then the angle of rotation can be calculated. The rotated image is saved with a size of 250 pixels*250 pixels, and the insufficient part is filled with black.

It should be noted that the coordinate system used in this optional embodiment is different from the usual coordinate system, the y-axis is from left to right in the horizontal direction, and the x-axis is from top to bottom in the vertical direction.

Assuming an image, the coordinates of the left and right eyes are (plx,ply) and (prx,pry) respectively, the coordinates of the midpoint of the line between the two eyes are (mx,my), and the distance between the pupils of the two eyes is d. At this time, the image scaling ratio ratio=d/dd (dd is 75 by default). The angle between the line connecting the two eyes and the y-axis is θ, and the slope of the line connecting the two eyes is k. Fig. 7 is a schematic diagram of a coordinate system according to an alternative embodiment of the present invention. As shown in Fig. 7, an ellipse represents the position of an eye.

To segment an image conforming to the human face standard from the original image, the image needs to be processed in the following steps: rotate θ degrees so that the line connecting the two eyes coincides with the y-axis; zoom the image so that the distance between the two eyes is dd; move the image so that Move the midpoint of the two eyes to (mx,my).

Figures 8a-8b are schematic diagrams showing comparisons of face images before and after rotation alignment according to an optional embodiment of the present invention, wherein Figure 8a is before rotation and Figure 8b is after rotation;

(3) Regional segmentation;

Image area segmentation, taking the segmentation of eyes as an example, the specific method is: find the key point coordinates pLeftIndex of the left eye corner and the key point coordinates pRightIndex of the right eye corner, and calculate their midpoint according to the coordinates of these two points Take this midpoint as the center point of the rectangle, define the distance from the center point to the left and right borders of the rectangle as centerToLeft, and the distance from the upper and lower borders of the rectangle as centerToUp, and define the width of the image as width and height as height, according to the position of the center point and centerToUp and centerToLeft can get the coordinates of the upper left corner of the segmented area. According to the coordinates of the upper left corner and the information of width and height, an image containing information of the segmented eye area is obtained.

(4) feature extraction;

Feature extraction mainly uses Gabor wavelet transform to extract the features of image blocks. In terms of feature extraction, Gabor wavelet transform is compared with other methods: on the one hand, it processes less data and can meet the real-time requirements of the system; on the other hand, wavelet transform is not sensitive to illumination changes and can tolerate a certain degree of For image rotation and deformation, when the recognition is based on the cosine distance of the included angle, the feature pattern and the feature to be tested do not need to be strictly corresponding, so the robustness of the system can be improved. Therefore, in the process of face recognition, we use the Gabor wavelet transform method to extract the features of the image.

(5) SVM classifier;

The SVM classifier was trained using LIBSVM. LIBSVM is a library that implements the SVM algorithm for support vector machines. Using LIBSVM requires two steps: first, train a data set to obtain a classification model, and then use the model to predict the class label of the test data set. The learning goal of the SVM classifier is to find a separating hyperplane that maximizes the margin in the feature space, separating different classes.

2. The specific implementation of the attribute classifier similarity classifier;

(1) The implementation of the attribute classifier. The attribute classifier uses the LFW data set. For each attribute in the 69 attributes in Table 1, select 1000 pictures that meet this attribute and do not meet this attribute from the LFW data set. (It is recommended not to be less than this number), mark each picture, if it meets this attribute, mark it as +1, if it does not meet it, mark it as -1. Segment the area corresponding to the attribute, see Table 2 for the specific area. Because some attributes can use the same segmentation region, 69 attributes can be reduced to 19 regions. The Gabor wavelet transform feature is extracted for the pictures used. Each attribute is then trained using LIBSVM.

Table 1

Table 2

(2) The specific implementation of the similarity classifier. The similarity classifier adopts the PubFig data set, selects 10 representative people from the PubFig image library, and each person has no less than 150 pictures. The four regions of eyes, eyebrows, nose and mouth required by the similarity classifier, Fig. 9 is a schematic diagram of similarity image region segmentation according to an optional embodiment of the present invention, as shown in Fig. 9, so that a total of 40 regions can be obtained, Extract features separately. Taking the eye area of the first reference person as an example, take his own eye area as a positive example and mark it as +1. In addition, select pictures that are not this person from the gallery (it is recommended to be multiple different people), the number of which is roughly equal to the number of all pictures of this reference person, segment the eye area, extract features, and mark it as -1 as a negative example. Similarly, Gabor wavelet transform features are extracted from the segmented regions. 40 similarity classifiers were trained using LIBSVM.

3. Calculate the attribute features and similarity features of the input image and feature weight learning;

(1) Perform face key point detection, image preprocessing, image region segmentation, and feature extraction operations on the input image, and 19 regional features are obtained from 19 regions.

(2) For 69 attributes and 40 similarity classifier models, the corresponding values can be obtained through the svmpredict() function of LIBSVM. In this way, 69 attribute values and 40 similarity values of this image are obtained. Combine the attribute value and similarity value to get the feature vector of this picture

(3) On the input image set of marked persons (the LFW dataset used), use the LMNN algorithm to optimize the weight of each dimension of the feature vector for the attribute features and similarity features extracted from the image set, and obtain

4. Implementation process of face retrieval;

(1) Process all the images in the retrieval gallery to obtain the 109-dimensional feature vector of each image.

(2) The newly input image to be retrieved is also processed to obtain a 109-dimensional feature vector.

(3) Calculate the similarity of face pictures, here the angle cosine is used, for two vectors with

Among them, in the above formula, <x, y> means to find the inner product between two vectors x and y, ||·|| means to find the modulus of the vector, and the value range of sim(v _i , v _j ) is [- 1,+1], the closer to +1, the more similar the faces in the two pictures. Calculate the cosine of the angle between the eigenvector of the image to be retrieved and the eigenvectors of all images in the retrieval library, and take the first N (default is 1000) images with the largest cosine value. The person in the selected top image is considered most likely to be the same person in the input image.

The process of face attribute classifier learning and attribute feature acquisition in this optional embodiment, face similarity classifier learning and similarity feature acquisition, FIG. 10 is an attribute/similarity feature classification according to an optional embodiment of the present invention A schematic diagram of machine learning and feature extraction process, as shown in Figure 10, the extraction process includes: attribute/similarity classifier training process and attribute/similarity classifier extraction process;

Among them, the steps of the attribute/similarity classifier training process include:

Step S1002: a gallery with markers;

Step S1004: fast face key point detection;

Step S1006: image preprocessing;

Step S1008: image region segmentation;

Step S1010: feature extraction;

Step S1012: SVM classification training and testing.

The steps of the attribute/similarity classifier extraction process include:

Step S1014: test or retrieve pictures;

Step S1016: fast face key point detection;

Step S1018: image preprocessing;

Step S1020: image region segmentation;

Step S1022: feature extraction;

Step S1024: Calculate the attribute/similarity value.

It should be noted that the difference between these two classifiers lies in the training and testing part of the SSVM classifier. In the attribute classifier, the attribute is marked as +1, and if not, it is marked as -1; the current reference person in the similarity classifier +1 for flagging, -1 for others. The face retrieval process of face attribute features and similarity features,

Fig. 11 is a schematic diagram of a picture storage and retrieval process according to an optional embodiment of the present invention. As shown in Fig. 11, the process includes two processes: a process of storing pictures in a database and a process of retrieval;

Among them, the steps of storing pictures in the database process include:

Step S1102: Retrieve the gallery;

Step S1104: calculating attribute and similarity feature values;

Step S1106: attribute/similarity feature combination;

Step S1108: portrait attribute/similarity feature database; after this step is executed, step S1016 is executed;

Steps in the retrieval process:

Step S1110: the picture to be retrieved;

Step S1112: calculating attribute and similarity feature values;

Step S1114: attribute/similarity feature combination;

Step S1116: feature comparison;

Step S1118: Retrieve result/most similar picture.

It should be noted that the final feature vector is a 109-dimensional vector containing 69 attribute values and 40 similarity values.

The embodiment of the invention also provides a storage medium. Optionally, in this embodiment, the above-mentioned storage medium may be configured to store program codes for performing the following steps:

S1: Train the semantic features of the face images in the gallery to obtain the attribute features of the first number of face images;

S2: Train the specified semantic features of the first predetermined number of reference face images to obtain the similarity features of the second number of specified semantic features with respect to the attribute features in the gallery;

S3: using the first number of attribute features and the second number of similarity features as feature vectors of each face image in the gallery;

S4: Calculate the feature vector of the image to be retrieved and the feature vector of each image in the gallery by preset rules, and retrieve one or more images matching the image to be detected according to the calculation result. Optionally, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementation manners, and details are not repeated in this embodiment.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned present invention can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network formed by multiple computing devices Alternatively, they may be implemented in program code executable by a computing device so that they may be stored in a storage device to be executed by a computing device, and in some cases in an order different from that shown here The steps shown or described are carried out, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present invention is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A retrieval method for face images, characterized in that, comprising: The semantic features of the face images in the gallery are trained to obtain the attribute features of the first quantity of the face images; Train the specified semantic features of the first predetermined number of reference face images to obtain the similarity features of the specified semantic features with respect to the attribute features in the gallery; Using the attribute feature of the first quantity and the similarity feature of the second quantity as the feature vector of each face image in the gallery; Computing the matching degree between the feature vector of the image to be retrieved and the feature vector of each image in the gallery through preset rules, and retrieving one or more images matching the image to be detected according to the matching degree. 2. method according to claim 1, is characterized in that, described the semantic feature of the face image in the gallery is trained and obtains the attribute feature of the described face image of the first quantity and comprises: Detect key points in each face image in the gallery, wherein the key points include: the four corners of the eyes, the tip of the nose and the two ends of the mouth; Carrying out regional division of the human face image according to the key points, and extracting the underlying features of the human face corresponding to different regions; According to the attribute classifier, classify and learn the plurality of human face underlying features in different regions to obtain the first number of attribute features of different types. 3. The method according to claim 1, wherein the specified semantic features of the first predetermined number of reference face images are trained to obtain the second quantity of the specified semantic features relative to the attribute features in the gallery. Similarity features include: Detecting the key points of the first predetermined number of reference face images, wherein the key points include: the four corners of both eyes, the tip of the nose, and both ends of the mouth; Extracting the specified semantic features according to the key points to obtain a data set corresponding to the specified semantic features, and extracting the underlying features of the face corresponding to different regions; Classifying and learning the data set according to a similarity classifier to obtain the second number of similarity features. 4. The method according to claim 2 or 3, wherein the attribute classifier and the similarity classifier comprise: a support vector machine (SVM) classifier. 5. The method according to claim 1, wherein the matching degree of the feature vector of the image to be retrieved is calculated by preset rules and the feature vector of each image in the gallery, and the matching degree is retrieved according to the matching degree. The one or more images that match the image to be detected include: Obtaining the feature vector of the image to be retrieved and the feature vector of each image in the gallery; Carrying out distance calculation between the feature vector of the image to be retrieved and the feature vector of each image in the gallery, the distance calculation method includes: cosine distance method, Euclidean distance method; Sorting a plurality of calculation results according to the rule from large to small, and selecting the face images corresponding to the second predetermined number of calculation results with higher values from the sorted calculation results as the matching images of the image to be retrieved . 6. A retrieval device for human face images, comprising: The first semantic feature extraction module is used to train the semantic features of the face images in the gallery to obtain the first number of attribute features of the face images; The second semantic feature extraction module is used to train the specified semantic features of the first predetermined number of reference face images to obtain a second number of similarity features of the specified semantic features with respect to the attribute features in the gallery; A processing module, configured to use the first number of attribute features and the second number of similarity features as feature vectors of each face image in the gallery; The retrieval module is used to calculate the matching degree between the feature vector of the image to be retrieved and the feature vector of each image in the gallery through preset rules, and retrieve one or more images matching the image to be detected according to the matching degree. image. 7. The device according to claim 6, wherein the first semantic feature extraction module comprises: The first detection unit is configured to detect the key points in each face image in the gallery, wherein the key points include: four corners of the eyes, the tip of the nose, and both ends of the mouth; A first processing unit, configured to divide the face image into regions according to the key points, and extract the underlying features of the face corresponding to different regions; The second semantic feature extraction unit is configured to classify and learn a plurality of the underlying features of the human face in different regions according to the attribute classifier to obtain the first number of attribute features of different types. 8. The device according to claim 7, wherein the second semantic feature extraction module comprises: The second detection unit is configured to detect the key points of the first predetermined number of reference face images, wherein the key points include: the four corners of both eyes, the tip of the nose, and both ends of the mouth; The second processing unit is configured to extract the specified semantic feature according to the key point to obtain a data set corresponding to the specified semantic feature, and extract the underlying features of the face corresponding to different regions; The second semantic feature extraction unit is configured to classify and learn the data set according to a similarity classifier to obtain a second number of similarity features. 9. The device according to claim 7 or 8, wherein the attribute classifier and the similarity classifier comprise: a support vector machine (SVM) classifier. 10. The device according to claim 6, wherein the retrieval module comprises: an acquisition unit, configured to acquire the feature vector of the image to be retrieved and the feature vector of each image in the gallery; A calculation unit, configured to calculate the distance between the feature vector of the image to be retrieved and the feature vector of each image in the gallery, the distance calculation method includes: cosine distance method, Euclidean distance method; A retrieval unit, configured to sort a plurality of calculation results according to the rule from large to small, and select, from the sorted calculation results, the face images corresponding to the second predetermined number of calculation results with the highest values as the to-be-waited Retrieves matching images for an image.