CN108009528A - Face authentication method, device, computer equipment and storage medium based on Triplet Loss - Google Patents

Abstract

The present invention relates to a kind of face authentication method based on Triplet Loss, device, computer equipment and storage medium, this method includes：Asked based on face authentication, obtain certificate photograph and the scene photo of personage；Carry out Face datection, crucial point location and image preprocessing respectively to scene photo and certificate photograph, obtain the corresponding scene facial image of scene photo, and the corresponding certificate facial image of certificate photograph；Scene facial image and certificate facial image are input to the advance trained convolutional neural networks model for face authentication, and the corresponding first eigenvector of scene facial image of convolutional neural networks model output is obtained, and the corresponding second feature vector of certificate facial image；Calculate the COS distance of first eigenvector and second feature vector；Compare COS distance and predetermined threshold value, and face authentication result is determined according to comparative result.It the method increase the reliability of face authentication.

Description

Face authentication method, device, computer equipment and storage based on Triplet Loss medium

technical field

The present invention relates to the technical field of image processing, in particular to a face authentication method, device, computer equipment and storage medium based on Triplet Loss.

Background technique

Face authentication refers to comparing the photos of people and scenes collected on the spot with the ID photos in the identity information to determine whether they are the same person. The key technology of face authentication is face recognition.

With the rise of deep learning technology, the related issues of face recognition continue to break through the traditional technical bottleneck, and the performance level has been greatly improved. In the research work of using deep learning to solve the problem of face recognition, there are two mainstream methods: the method based on classification learning and the method based on metric learning. Among them, the method based on classification learning mainly calculates the classification loss function (such as softmax loss, center loss and related variants) of the sample after the deep convolutional network extracts the features to optimize the network. The last layer of the network is used for classification. In the fully connected layer, the number of output nodes is often consistent with the total number of categories in the training data set. This type of method is suitable for a large number of training samples, especially when the training samples of the same category are abundant, and the network can get better results. Training effect and generalization ability. However, when the number of categories reaches hundreds of thousands or more, the parameters of the final classification layer (fully connected layer) of the network will increase linearly and become quite large, making it difficult to train the network.

Another type of method is a method based on metric learning, which organizes training samples in tuples (such as binary pairs or triplets), and does not need to pass through the classification layer after the deep convolutional network, but directly based on the volume The product feature vector is used to calculate the metric loss between samples (such as contrastive loss, triplet loss, etc.) to optimize the network. This method does not need to train the classification layer, so the number of network parameters is not affected by the increase in the number of categories. The category of the training data set The number is unlimited, and it is only necessary to select the same or different samples according to the corresponding strategy to construct the appropriate tuple. Compared with the classification learning method, the metric learning method is more suitable for the situation where the training data has a large breadth but insufficient depth (there are many sample categories, but few similar samples). Through different combinations of samples, quite rich tuple data can be constructed It is used for training, and the metric learning method pays more attention to the internal relationship of tuples. It has inherent advantages for judging yes or no such as 1:1 face verification.

In practical applications, many institutions require real-name registration, for example, bank account opening, mobile phone number registration, financial account opening and so on. The real-name system registration requires the user to bring the ID card to the designated place, and the account can only be successfully opened after the staff verifies that the person and the photo of the ID card correspond. With the development of Internet technology, more and more institutions have launched convenience services, and no longer require customers to go to designated outlets. The geographical location of the user is not limited, upload the ID card, and use the image acquisition device of the mobile terminal to collect photos of people and scenes on the scene, and the system will perform face authentication, and after the face authentication is passed, the account can be successfully opened. The traditional measurement-based learning method uses Euclidean distance to measure the similarity between samples, and Euclidean distance measures the absolute distance of each point in space, which is directly related to the position coordinates of each point, which is not in line with human The distribution properties of the face feature space lead to low reliability of face recognition.

Contents of the invention

Based on this, it is necessary to provide a face authentication method, device, computer equipment and storage medium based on TripletLoss to solve the problem of low reliability of traditional face authentication methods.

A face authentication method based on Triplet Loss, comprising:

Based on the face authentication request, obtain the photo of the ID card and the scene photo of the person;

Carrying out face detection, key point positioning and image preprocessing on the scene photo and the certificate photo respectively, to obtain a scene face image corresponding to the scene photo and a certificate face image corresponding to the certificate photo;

The scene face image and the certificate face image are input to the pre-trained convolutional neural network model for face authentication, and the first corresponding to the scene face image output by the convolutional neural network model is obtained. A feature vector, and a second feature vector corresponding to the ID face image; wherein, the convolutional neural network model is obtained based on the supervised training of the triple loss function;

calculating the cosine distance of the first eigenvector and the second eigenvector;

Comparing the cosine distance with a preset threshold, and determining a face authentication result according to the comparison result.

In one embodiment, the method also includes:

Acquiring marked training samples, the training samples include a document face image and at least one scene face image marked with each marked object;

According to the training sample training convolutional neural network module, the triplet element corresponding to each training sample is generated by OHEM; the triplet element includes a reference sample, a positive sample and a negative sample;

According to the triplet elements of each training sample, based on the supervision of the triplet loss function, train the convolutional neural network model; the triplet loss function uses the cosine distance as a measurement method to optimize the model through a stochastic gradient descent algorithm parameter;

The verification set data is input into the convolutional neural network model, and when the training end condition is reached, the trained convolutional neural network model for face authentication is obtained.

In another embodiment, according to the training sample training convolutional neural network model, the step of generating triplet elements corresponding to each training sample by OHEM includes:

Randomly select an image as a reference sample, and select an image that belongs to the same label object and is different from the reference sample category as a positive sample;

According to the OHEM strategy, the currently trained convolutional neural network model is used to extract the cosine distance between features. For each reference sample, from other images that do not belong to the label object, select the image with the smallest distance that is different from the reference sample. The image of the category is used as a negative sample for this reference sample.

In another embodiment, the triplet loss function includes a limitation on the cosine distance of samples of the same type, and a limitation on the cosine distance of samples of different types.

In another embodiment, the triplet loss function is:

Among them, cos(·) represents the cosine distance, and its calculation method is N is the number of triplets, Represents the feature vector of the reference sample, Represents the feature vector of the same positive sample, Represents the eigenvector of heterogeneous negative samples, and the meaning of [ ] ₊ is as follows: α ₁ is the inter-class interval parameter, and α ₂ is the intra-class interval parameter.

In another embodiment, the method further includes: initializing the basic model parameters based on massive open source face data training, adding a normalization layer and a triplet loss function layer after the feature output layer to obtain the convolutional neural network model.

A face authentication device based on Triplet Loss, comprising: an image acquisition module, an image preprocessing module, a feature acquisition module, a calculation module and an authentication module;

The image acquisition module is used to acquire a photo of a certificate and a scene photo of a person based on a face authentication request;

The image preprocessing module is used to perform face detection, key point positioning and image preprocessing on the scene photo and the ID photo, respectively, to obtain the scene face image corresponding to the scene photo, and the ID photo The corresponding ID face image;

The feature acquisition module is used to input the scene face image and the certificate face image into the pre-trained convolutional neural network model for face authentication, and obtain all the output of the convolutional neural network model. The first feature vector corresponding to the face image of the scene, and the second feature vector corresponding to the face image of the certificate; wherein, the convolutional neural network model is obtained based on the supervised training of the triple loss function;

The calculation module is used to calculate the cosine distance between the first eigenvector and the second eigenvector;

The authentication module is configured to compare the cosine distance with a preset threshold, and determine a face authentication result according to the comparison result.

In another embodiment, the device further includes: a sample acquisition module, a triplet acquisition module, a training module and a verification module;

The sample acquisition module is used to acquire marked training samples, the training samples include a certificate face image and at least one scene face image marked with each marked object;

The triplet acquisition module is used to train the convolutional neural network model according to the training samples, and generate triplet elements corresponding to each training sample through OHEM; the triplet elements include reference samples, positive samples and negative samples ;

The training module is used to train the convolutional neural network model according to the triplet elements of each training sample and the supervision of the basic triplet loss function; the triplet loss function uses cosine distance as a measurement method, through Stochastic gradient descent algorithm to optimize model parameters;

The verification module is used to input the verification set data into the convolutional neural network model, and when the training end condition is reached, the trained convolutional neural network model for face authentication is obtained.

A computer device includes a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the computer program, the steps of the above-mentioned face authentication method based on Triplet Loss are realized.

A storage medium, on which a computer program is stored, is characterized in that, when the computer program is executed by a processor, the steps of the above-mentioned face authentication method based on Triplet Loss are realized.

The face authentication method, device, computer equipment and storage medium based on Triplet Loss of the present invention utilizes the pre-trained convolutional neural network for face authentication, since the convolutional neural network model is based on the supervised training of the triplet loss function obtained, and the similarity between the scene face image and the certificate face image is calculated according to the cosine distance between the first feature vector corresponding to the scene face image and the second feature vector corresponding to the certificate face image, and the cosine distance measures the space vector The included angle is more reflected in the difference in direction, which is more in line with the distribution attribute of the face feature space, and improves the reliability of face authentication.

Description of drawings

Fig. 1 is the structural representation of the face authentication system based on Triplet Loss of an embodiment;

Fig. 2 is the flowchart of the face authentication method based on Triplet Loss in an embodiment;

Fig. 3 is a flowchart of the steps of training the convolutional neural network model for face authentication in one embodiment;

Figure 4 is a schematic diagram of the probability of sample misclassification when the inter-class interval is consistent and the intra-class variance is large;

Figure 5 is a schematic diagram of the probability of sample misclassification when the inter-class interval is consistent and the intra-class variance is small;

Fig. 6 is a schematic diagram of the migration learning process of face authentication based on Triplet Loss in an embodiment;

Fig. 7 is a schematic structural diagram of a convolutional neural network model for face authentication in an embodiment;

Fig. 8 is a schematic flow chart of a face authentication method based on Triplet Loss in an embodiment;

Fig. 9 is a structural block diagram of a face authentication device based on Triplet Loss in an embodiment;

Fig. 10 is a structural block diagram of a face authentication device based on Triplet Loss in another embodiment.

Detailed ways

FIG. 1 is a schematic structural diagram of a face authentication system based on Triplet Loss in an embodiment. As shown in FIG. 1 , the face authentication system includes a server 101 and an image acquisition device 102 . Wherein, the server 101 is connected to the image acquisition device 102 through a network. The image collection device 102 collects real-time scene photos and ID photos of the user to be authenticated, and sends the collected real-time scene photos and ID photos to the server 101 . The server 101 judges whether the person in the scene photo and the person in the ID photo are the same person, and authenticates the identity of the user to be authenticated. Based on specific application scenarios, the image acquisition device 102 may be a camera, or a user terminal with a camera function. Taking the account opening site as an example, the image collection device 102 may be a camera; taking financial account opening via the Internet as an example, the image collection device 102 may be a mobile terminal with a camera function.

In other embodiments, the face authentication system may also include a card reader, which is used to read the ID photo in the chip of the ID card (eg, ID card, etc.).

Fig. 2 is a flowchart of a face authentication method based on Triplet Loss in an embodiment. As shown in Figure 2, the method includes:

S202. Based on the face authentication request, acquire a photo of a certificate and a scene photo of a person.

Wherein, the certificate photo refers to the photo corresponding to the certificate that can prove the identity of the person, such as the certificate photo printed on the ID card or the certificate photo in the chip. The certificate photo can be obtained by taking a photo of the certificate, or reading the certificate photo stored in the certificate chip through a card reader. The certificate in this embodiment can be an ID card, a driver's license or a social security card, etc.

The scene photo of the person refers to the photo of the user to be authenticated in the on-site environment collected when the user to be authenticated is authenticated. The on-site environment refers to the environment in which the user takes a photo, and the on-site environment is not limited. The way to acquire the scene photo may be to use a mobile terminal with a camera function to collect the scene photo and send it to the server.

Face authentication refers to comparing the photos of people and scenes collected on the spot with the ID photos in the identity information to determine whether they are the same person. The face authentication request is triggered based on the actual application operation, for example, based on the user's account opening request, the face authentication request is triggered. The application program prompts the user to collect photos on the display interface of the user terminal, and after the photo collection is completed, the collected photos are sent to the server for face authentication.

S204. Perform face detection, key point location, and image preprocessing on the scene photo and the ID photo, respectively, to obtain a scene face image corresponding to the scene photo and a ID face image corresponding to the ID photo.

Face detection refers to identifying photos and obtaining face areas in photos.

Key point positioning refers to obtaining the position of the key points of the face in each photo for the detected face area in the photo. The key points of the human face include the eyes, the tip of the nose, the corner of the mouth, the eyebrows and the contour points of each part of the face.

In this embodiment, the cascaded convolutional neural network MTCNN method based on multi-task joint learning can be used to complete face detection and face key point detection at the same time, and the face detection method based on LBP features and the human face detection method based on shape regression can also be used. Face keypoint detection method.

Image preprocessing means that according to the position of the detected face key points in each picture, portrait alignment and cropping are performed, so as to obtain the normalized size of the scene face image and the ID face image. Among them, the scene face image refers to the face image obtained after performing face detection, key point positioning and image preprocessing on the scene photo, and the certificate face image refers to the face detection, key point positioning and image preprocessing of the certificate photo. The face image obtained after processing.

S206, input the scene face image and the certificate face image to the pre-trained convolutional neural network model for face authentication, and obtain the first feature vector corresponding to the scene face image output by the convolutional neural network model, And the second feature vector corresponding to the face image of the certificate.

Among them, the convolutional neural network model is pre-trained based on the supervision of the triple loss function based on the training samples. The convolutional neural network includes a convolutional layer, a pooling layer, an activation function layer, and a fully connected layer. The parameters of each neuron in each layer are determined through training. Using the trained convolutional neural network, through the forward propagation of the network, the first feature vector of the scene face image output by the fully connected layer of the convolutional neural network model and the second feature vector corresponding to the ID face image are obtained.

A triplet is to randomly select a sample from the training data set, which is called a reference sample, and then randomly select a sample that belongs to the same person as the reference sample as a positive sample, and select a sample that does not belong to the same person as a negative sample. samples, thus forming a (reference sample, positive sample, negative sample) triplet. Since the witness comparison is mainly based on the comparison between the ID photo and the scene photo, rather than the comparison between the ID photo and the ID photo, or the scene photo and the scene photo, there are mainly two combinations of triplet modes: When the image is used as a reference sample, both the positive sample and the negative sample are scene photos; when the scene photo image is used as the reference sample, both the positive sample and the negative sample are ID photos.

For each sample in the triplet, a parameter-sharing network is trained to obtain the feature representation of the three elements. The purpose of improving the triplet loss (triplet loss) is to make the distance between the feature expression of the reference sample and the positive sample as small as possible through learning, and the distance between the feature expression of the reference sample and the negative sample is as large as possible, and to Let there be a minimum interval between the distance between the reference sample and the feature expression of the positive sample and the distance between the reference sample and the feature expression of the negative sample.

S208. Calculate a cosine distance between the first eigenvector and the second eigenvector.

Cosine distance, also known as cosine similarity, uses the cosine value of the angle between two vectors in a vector space as a measure of the difference between two individuals. The larger the cosine distance between the first eigenvector and the second eigenvector, the greater the similarity between the scene face image and the document face image, and the smaller the cosine distance between the first eigenvector and the second eigenvector, it represents the scene face The similarity between the image and the ID face image is smaller. When the cosine distance between the scene face image and the certificate face image is closer to 1, the probability that the two images belong to the same person is greater; when the cosine distance between the scene face image and the certificate face image is smaller, the two images belong to The less likely it is the same person.

In the traditional triplet loss method, Euclidean distance is used to measure the similarity between samples. The Euclidean distance measures the absolute distance of each point in the space, which is directly related to the position coordinates of each point, which does not conform to the distribution attribute of the face feature space. In this embodiment, the cosine distance is used to measure the similarity between samples in consideration of the distribution attribute of the face feature space and the actual application scenario. The cosine distance measures the included angle of the space vector, which is more reflected in the difference in direction rather than position, so it is more in line with the distribution attribute of the face feature space.

Specifically, the formula for calculating the cosine distance is:

Wherein, x represents the first eigenvector, and y represents the second eigenvector.

S210. Compare the cosine distance with a preset threshold, and determine a face authentication result according to the comparison result.

The authentication result includes the authentication pass, that is, the ID photo and the scene photo belong to the same person. The authentication result also includes authentication failure, that is, the ID photo and the scene photo do not belong to the same person.

Specifically, the cosine distance is compared with the preset threshold. When the cosine distance is greater than the preset threshold, it means that the similarity between the ID photo and the scene photo is greater than the preset threshold, and the authentication is successful. When the cosine distance is smaller than the preset threshold, it means That is, if the similarity between the ID photo and the scene photo is less than the preset threshold, the authentication fails.

The above-mentioned face authentication method based on Triplet Loss uses the pre-trained convolutional neural network for face authentication. Since the convolutional neural network model is obtained based on the supervised training of the triplet loss function, the scene face image and the ID face The similarity of the image is calculated based on the cosine distance between the first feature vector corresponding to the scene face image and the second feature vector corresponding to the ID face image. The cosine distance measures the included angle of the space vector, which is more reflected in the difference in direction , which is more in line with the distribution attribute of the face feature space, and improves the reliability of face authentication.

In another embodiment, the face authentication method further includes the step of training a convolutional neural network model for face authentication. Fig. 3 is a flow chart of the steps of training a convolutional neural network model for face authentication in one embodiment. As shown in Figure 3, this step includes:

S302. Obtain marked training samples, where the training samples include a certificate face image and at least one scene face image marked with each marked object.

In this embodiment, the marking object is a person, and the training sample is based on a person, and the scene face image and the certificate face image belonging to the same person are marked. Specifically, the scene face image and the ID face image can be obtained by performing face detection, key point location and image preprocessing on the marked scene photo and ID photo.

Face detection refers to identifying photos and obtaining face areas in photos.

Key point positioning refers to obtaining the position of the key points of the face in each photo for the detected face area in the photo. The key points of the human face include the eyes, the tip of the nose, the corner of the mouth, the eyebrows and the contour points of each part of the face.

In this embodiment, the cascaded convolutional neural network MTCNN method based on multi-task joint learning can be used to complete face detection and face keying at the same time, and the face detection method based on LBP features and the face keying based on shape regression can also be used. point detection method.

Image preprocessing means that according to the position of the detected face key points in each picture, portrait alignment and cropping are performed to obtain a size-normalized scene face image and a certificate face image. Among them, the scene face image refers to the face image obtained after performing face detection, key point positioning and image preprocessing on the scene photo, and the certificate face image refers to the face detection, key point positioning and image preprocessing of the certificate photo. The face image obtained after processing.

S304. Train the convolutional neural network model according to the training samples, and generate triplet elements corresponding to each training sample through OHEM; the triplet elements include reference samples, positive samples, and negative samples.

There are two combinations of triplets: when the ID photo image is used as the reference sample, both the positive sample and the negative sample are scene photos; when the scene photo image is used as the reference sample, both the positive sample and the negative sample are ID photo images.

Specifically, taking the ID photo as the reference image as an example, a sample of a person’s ID photo is randomly selected from the training data set, which is called a reference sample, and then a scene photo sample belonging to the same person as the reference sample is randomly selected as a positive sample , select scene photos that do not belong to the same person as negative samples, thus forming a (reference sample, positive sample, negative sample) triplet.

That is, the positive sample and the reference sample are the same type of sample, that is, they belong to the same person image. Negative samples are heterogeneous samples from the reference sample, i.e. images that do not belong to the same person. Among them, the reference samples and positive samples in the triplet elements are marked in the training samples. During the training process of the convolutional neural network, the negative samples are constructed online using the OHEM (Online Hard Example Mining) strategy, that is, in the network In the process of each iterative optimization, the current network is used to perform forward calculations on the candidate triplets, and the images in the training samples that do not belong to the same user as the reference samples and have the closest cosine distance are selected as negative samples, so as to obtain the corresponding training samples. triplet element.

In one embodiment, the step of training the convolutional neural network according to the training samples and generating the triplet elements corresponding to each training sample includes the following steps S1 and S2:

S1: Randomly select an image as a reference sample, and select an image that belongs to the same label object and is different from the reference sample category as a positive sample.

The category refers to the image type to which it belongs. In this embodiment, the categories of the training samples include scene face images and certificate face images. Because face authentication is mainly a comparison between the ID photo and the scene photo, the reference sample and the positive sample should belong to different categories. If the reference sample is a scene face image, the positive sample is a ID face image; if the reference sample is a ID face image, the positive sample is the scene face image.

S2: According to the OHEM strategy, use the currently trained convolutional neural network model to extract the cosine distance between features. For each reference sample, select the smallest distance from other images that do not belong to the same label object and belong to a different category from the reference sample. The image of , as the negative sample of the reference sample.

Negative samples are selected from face images with labels that do not belong to the same person as the reference sample. Specifically, during the training process of the convolutional neural network, the negative samples are constructed online using the OHEM strategy, that is, each iteration of the network is optimized In the process of , the current network is used to perform forward calculation on the candidate triplets, and the images in the training samples that do not belong to the same user as the reference samples, have the closest cosine distance, and do not belong to the same category as the reference samples are selected as negative samples. That is, a negative sample is of a different class than the reference sample. It can be considered that if the ID photo is used as the reference sample in the triplet, both the positive sample and the negative sample are scene photos; otherwise, if the scene photo is used as the reference sample, then both the positive sample and the negative sample are ID photos.

S306, according to the triplet elements of each training sample, and based on the supervision of the triplet loss function, train the convolutional neural network model. The triplet loss function uses the cosine distance as the measurement method to optimize the model through the stochastic gradient descent algorithm. parameter.

The witness verification terminal verifies the identity of the user by comparing whether the chip photo of the user’s ID card is consistent with the scene photo. The data collected in the background is often a sample of a single person with only two pictures, that is, the ID photo and the scene captured at the moment of comparison. photos, but the number of different individuals can be tens of thousands. If this kind of data with a large number of categories and few similar samples is trained by a classification-based method, the parameters of the classification layer will be too large and the network will be very difficult to learn. Therefore, the method of metric learning is considered to solve it. Among them, the typical metric learning is to use triplet loss (triplet loss) method to learn an effective feature map by constructing image triplets. Under this map, the feature distance of similar samples is smaller than that of heterogeneous samples. In order to achieve the purpose of correct comparison.

The purpose of triplet loss (triplet loss) is to make the distance between the feature expression of the reference sample and the positive sample as small as possible through learning, and the distance between the feature expression of the reference sample and the negative sample is as large as possible, and let There is a minimum gap between the distance between the feature representation of the reference sample and the positive sample and the distance between the feature representation of the reference sample and the negative sample.

In another embodiment, the triplet loss function includes a restriction on the cosine distance of samples of the same kind, and a restriction on the cosine distance of samples of the different class.

Among them, similar samples refer to reference samples and positive samples, and heterogeneous samples refer to reference samples and negative samples. The cosine distance of the same sample refers to the cosine distance between the reference sample and the positive sample, and the cosine distance of the heterogeneous sample refers to the cosine distance between the reference sample and the negative sample.

On the one hand, the original triplet loss method only considers the inter-class gap but not the intra-class gap. If the intra-class distribution is not convergent enough, the generalization ability of the network will be weakened, and the adaptability to the scene will also be reduced. On the other hand, the original triplet loss method uses the Euclidean distance to measure the similarity between samples. In fact, after the face model is deployed, the cosine distance is often used to measure the feature comparison. Euclidean distance measures the absolute distance of each point in space, which is directly related to the position coordinates of each point; while cosine distance measures the angle between space vectors, which is more reflected in the difference in direction rather than position, so it is more in line with Distribution properties of face feature space.

The triplet loss method is adopted to perform iterative optimization by constructing triplet data input network online, and then backpropagating the triplet metric loss. Each triplet contains three images, which are a reference sample, a positive sample of the same type as the reference sample, and a negative sample of the same type as the reference sample, marked as (anchor, positive, negative). The basic idea of the original triplet loss is that the distance between the reference sample and the positive sample is smaller than the distance between the reference sample and the negative sample through metric learning, and the distance difference is greater than a minimum interval parameter α. So the original triplet loss loss function is as follows:

where N is the number of triplets, Represents the feature vector of the reference sample (anchor), Represents the eigenvector of the same positive sample (positive), A feature vector representing a heterogeneous negative sample (negative). Represents the L2 paradigm, that is, the Euclidean distance. [ ] ₊ has the following meanings:

It can be seen from the above formula that the original triplet loss function only limits the distance between similar samples (anchor, positive) and heterogeneous samples (anchor, negative), that is, the inter-class distance can be increased as much as possible through the interval parameter α, while for The intra-class distance is not limited, that is, there is no constraint on the distance between samples of the same class. If the intra-class distance is scattered and the variance is too large, the generalization ability of the network will be weakened, and the probability of samples being misclassified will be greater. Figure 4 is a schematic diagram of the probability of sample misclassification when the inter-class interval is consistent and the intra-class variance is large. Figure 5 is a schematic diagram of the probability of sample misclassification when the inter-class interval is consistent and the intra-class variance is small. 4 and Figure 5, the shaded part represents the probability of sample misclassification. In the case of consistent inter-class intervals and large intra-class variance, the probability of sample misclassification is significantly greater than that of consistent inter-class intervals and small intra-class variance. The probability of sample misclassification.

In view of the above problems, the present invention proposes an improved triplet loss method. On the one hand, it retains the limitation on the distance between classes in the original method, and at the same time adds constraints on the distance within a class, so that the distance within a class can converge as much as possible. Its loss function expression is:

Among them, cos(·) represents the cosine distance, and its calculation method is N is the number of triplets, Represents the feature vector of the reference sample, Represents the feature vector of the same positive sample, Represents the eigenvector of heterogeneous negative samples, and the meaning of [ ] ₊ is as follows: α ₁ is the inter-class interval parameter, and α ₂ is the intra-class interval parameter.

Compared with the original triplet loss function, the measurement method of the improved triplet loss function is changed from Euclidean distance to cosine distance, which can maintain the consistency of the measurement method between the training phase and the deployment phase, and improve the continuity of feature learning. At the same time, the first item of the new triplet loss function has the same effect as the original triplet loss, which is used to increase the inter-class gap, and the second item adds a distance constraint on the same sample pair (positive tuple), which is used to narrow the intra-class gap. α ₁ is the inter-class interval parameter, the value range is 0-0.2, α ₂ is the intra-class interval parameter, the value range is 0.8-1.0. It is worth noting that, since it is measured by the cosine method, the obtained measurement value corresponds to the similarity between two samples, so in In the expression, only the samples whose negative tuple cosine similarity is greater than the positive tuple cosine similarity in the range of α ₁ will really participate in the training.

The model is trained based on the improved triplet loss function, and the model is optimized for backpropagation through the joint constraints of inter-class loss and intra-class loss, so that similar samples are as close as possible in the feature space and heterogeneous samples are in the feature space. Keep it as far away as possible to improve the recognition of the model, thereby improving the reliability of face authentication.

S308. Input the verification set data into the convolutional neural network, and obtain a trained convolutional neural network for face authentication when the training end condition is reached.

Specifically, take 90% of the data from the witness image data pool as the training set, and the remaining 10% as the verification set. Based on the above formula, the improved triplet loss value is calculated and fed back to the convolutional neural network for iterative optimization. At the same time, observe the performance of the model in the verification set. When the verification performance no longer improves, the model reaches the convergence state and the training phase ends.

The above-mentioned face authentication method, on the one hand, adds a constraint on the distance of samples within a class to the loss function of the original triplet loss, thereby increasing the gap between classes while reducing the gap within a class, and improving the generalization ability of the model; On the one hand, the measurement method of the original triplet loss is changed from Euclidean distance to cosine distance, so as to maintain the measurement consistency between training and deployment, and improve the continuity of feature learning.

In another embodiment, the step of training the convolutional neural network further includes: initializing the basic model parameters trained based on massive open source face data, adding a normalization layer and an improved triplet after the feature output layer The loss function layer is used to obtain the convolutional neural network to be trained.

Specifically, when deep learning is used to solve the problem of the combination of human and evidence, the performance of the conventional deep face recognition model trained based on massive Internet face data will be greatly reduced in the application of human-evidence comparison in specific scenarios, while the specific application The source of witness data in the scene is relatively limited, and direct learning often leads to unsatisfactory training results due to insufficient samples. Therefore, it is extremely necessary to develop an effective method of expanding training for small data sets of scene data to improve face recognition. The accuracy of the model in a specific application scenario meets market application requirements.

Deep learning algorithms often rely on the training of massive data. In the application of human-certificate integration, the comparison of ID photos and scene photos is a heterogeneous sample comparison problem. The conventional deep face recognition model based on massive Internet face data training The performance of the application of witness comparison will drop significantly. However, the source of witness data is limited (need to have the ID card image of the same person and the corresponding scene image at the same time), the amount of data available for training is small, and direct training will lead to unsatisfactory training results due to insufficient samples. When training the model of the combination of witnesses and witnesses, the idea of transfer learning is often used to train a basic model with reliable performance on the open source test set based on massive Internet face data, and then perform a secondary test on the limited witness data. Extended training enables the model to automatically learn the feature representation of a specific modality and improve model performance. This process is shown in Figure 6.

In the process of secondary training, the entire network is initialized with pre-trained basic model parameters, and then an L2 normalization layer and an improved triplet loss layer are added after the feature output layer of the network. The convolutional neural network to be trained The network structure diagram is shown in Figure 7.

In one embodiment, a schematic flowchart of a face authentication method is shown in FIG. 8 , which includes three stages, namely, a data collection and preprocessing stage, a training stage, and a deployment stage.

In the data collection and preprocessing stage, the card reader module of the personal identification verification terminal device reads the chip photo of the certificate, and the front camera captures the on-site photo, and passes through the face detector, key point detector, face alignment and cutting After the module, the size-normalized ID face image and scene face image are obtained.

In the training phase, take 90% of the data from the witness image data pool as the training set, and the remaining 10% as the verification set. Since the witness comparison is mainly the comparison between the ID photo and the scene photo, because if the ID photo is used as the reference image (anchor) in the triplet, the other two images are both scene photos; otherwise, if the scene photo is used as a reference picture, the other two pictures are ID photos. Adopt the strategy of OHEM to construct triplets online, that is, in the process of each iterative optimization of the network, use the current network to perform forward calculation on the candidate triplets, filter the effective triplets that meet the conditions, and calculate the improved triplets according to the above formula The triplet loss value is fed back to the network for iterative optimization. At the same time, observe the performance of the model in the verification set. When the verification performance no longer improves, the model reaches the convergence state and the training phase ends.

In the deployment stage, when the trained model is deployed to the witness verification terminal for use, the images collected by the device undergo the same preprocessing procedure as in the training stage, and then the feature vector of each face image is obtained through the forward calculation of the network. Calculate the cosine distance to obtain the similarity of the two images, and then make a judgment according to the preset threshold. If it is greater than the preset threshold, it is the same person, otherwise it is a different person.

In the above-mentioned face authentication method, the original triplet loss function only limits the learning relationship of the inter-class distance. In the above-mentioned face authentication method, by improving the original triplet loss loss function and increasing the constraint item of the intra-class distance, the network can be trained In this method, the inter-class gap is increased while the intra-class gap is reduced as much as possible, so as to improve the generalization ability of the network, and then improve the scene adaptability of the model. In addition, the Euclidean distance measurement method in the original triplet loss is replaced by the cosine distance, which is more in line with the distribution attribute of the face feature space, and the consistency of the measurement method between the training phase and the deployment phase is maintained, making the comparison result more reliable.

In one embodiment, a face authentication device is provided, as shown in FIG. 9 , including: an image acquisition module 902 , an image preprocessing module 904 , a feature acquisition module 906 , a calculation module 908 and an authentication module 910 .

The image acquisition module 902 is configured to acquire the certificate photo and the person's scene photo based on the face authentication request.

The image preprocessing module 904 is configured to perform face detection, key point location, and image preprocessing on the scene photo and the ID photo, respectively, to obtain the scene face image corresponding to the scene photo, and the ID face image corresponding to the ID photo.

The feature acquisition module 906 is used to input the scene face image and the certificate face image into the pre-trained convolutional neural network model for face authentication, and obtain the scene face image corresponding to the convolutional neural network model output. The first feature vector, and the second feature vector corresponding to the ID face image; wherein, the convolutional neural network model is obtained based on the supervised training of the triplet loss function.

A calculation module 908, configured to calculate a cosine distance between the first eigenvector and the second eigenvector.

The authentication module 910 is configured to compare the cosine distance with a preset threshold, and determine a face authentication result according to the comparison result.

The above-mentioned face authentication device uses a pre-trained convolutional neural network for face authentication. Since the convolutional neural network model is obtained based on the supervised training of the improved triple loss function, the scene face image and the certificate face image The similarity of is calculated based on the cosine distance between the first eigenvector corresponding to the scene face image and the second eigenvector corresponding to the ID face image. The cosine distance measures the angle between space vectors, which is more reflected in the difference in direction. Instead of the position, it is more in line with the distribution attribute of the face feature space and improves the reliability of face authentication.

As shown in FIG. 9 , in another embodiment, the face authentication device further includes: a sample acquisition module 912 , a triplet acquisition module 914 , a training module 916 and a verification module 918 .

The sample obtaining module 912 is configured to obtain marked training samples, where the training samples include a certificate face image and at least one scene face image marked with each marked object.

The triplet acquisition module 914 is used to train the convolutional neural network model according to the training samples, and generate triplet elements corresponding to each training sample through OHEM; the triplet elements include reference samples, positive samples and negative samples.

Specifically, the triplet acquisition module 914 is used to randomly select an image as a reference sample, select an image that belongs to the same label object and is different from the reference sample category as a positive sample, and is also used to use the currently trained convolution The neural network model extracts the cosine distance between features. For each reference sample, from other face images that do not belong to the same label object, select the image with the smallest distance and belonging to a different category from the reference sample as the negative of the reference sample. sample.

Specifically, when ID photos are used as reference samples, both positive and negative samples are scene photos; when scene photos are used as reference samples, both positive samples and negative samples are ID photos.

The training module 916 is used to train the convolutional neural network model based on the triplet elements of each training sample, based on the supervision of the triplet loss function, and the triplet loss function uses the cosine distance as a measurement method to descend through stochastic gradient Algorithms to optimize model parameters.

Specifically, the improved triplet loss function includes the limitation of the cosine distance of the same kind of samples, and the limitation of the cosine distance of the heterogeneous samples.

The improved triplet loss function is:

Among them, cos(·) represents the cosine distance, and its calculation method is N is the number of triplets, Represents the feature vector of the reference sample, Represents the feature vector of the same positive sample, Represents the eigenvector of heterogeneous negative samples, and the meaning of [ ] ₊ is as follows: α ₁ is the inter-class interval parameter, and α ₂ is the intra-class interval parameter.

The verification module 918 is used to input the verification set data into the convolutional neural network model, and obtain the trained convolutional neural network model for face authentication when the training end condition is reached.

In another embodiment, the face authentication device also includes a model initialization module 920, which is used to initialize the basic model parameters trained based on massive open source face data, and add a normalization layer and a triplet after the feature output layer The loss function layer is used to obtain the convolutional neural network to be trained. The above-mentioned face authentication device, on the one hand, adds a constraint on the intra-class sample distance in the loss function of the original triplet loss, thereby increasing the inter-class gap while reducing the intra-class gap, and improving the generalization ability of the model; On the one hand, the measurement method of the original triplet loss is changed from Euclidean distance to cosine distance, so as to maintain the measurement consistency between training and deployment, and improve the continuity of feature learning.

A computer device includes a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the computer program, the steps of the face authentication method in the above-mentioned embodiments are realized.

A storage medium, on which a computer program is stored, is characterized in that, when the computer program is executed by a processor, the steps of the face authentication method in the above-mentioned embodiments are realized.

The technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, should be considered as within the scope of this specification.

The above-mentioned embodiments only express several implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but should not be construed as limiting the patent scope of the invention. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (10)

1. A face authentication method based on Triplet Loss, comprising: Based on the face authentication request, obtain the photo of the ID card and the scene photo of the person; Carrying out face detection, key point positioning and image preprocessing on the scene photo and the certificate photo respectively, to obtain a scene face image corresponding to the scene photo and a certificate face image corresponding to the certificate photo; The scene face image and the certificate face image are input to the pre-trained convolutional neural network model for face authentication, and the first corresponding to the scene face image output by the convolutional neural network model is obtained. A feature vector, and a second feature vector corresponding to the ID face image; wherein, the convolutional neural network model is obtained based on the supervised training of the triple loss function; calculating the cosine distance of the first eigenvector and the second eigenvector; Comparing the cosine distance with a preset threshold, and determining a face authentication result according to the comparison result. 2. The method according to claim 1, characterized in that the method further comprises: Acquiring marked training samples, the training samples include a document face image and at least one scene face image marked with each marked object; According to the training sample training convolutional neural network model, the triplet element corresponding to each training sample is generated by OHEM; the triplet element includes a reference sample, a positive sample and a negative sample; According to the triplet elements of each training sample, based on the supervision of the triplet loss function, train the convolutional neural network model; the triplet loss function uses the cosine distance as a measurement method to optimize the model through a stochastic gradient descent algorithm parameter; The verification set data is input into the convolutional neural network model, and when the training end condition is reached, the trained convolutional neural network model for face authentication is obtained. 3. method according to claim 2, is characterized in that, according to described training sample training convolutional neural network model, the step of triple group element that produces each training sample correspondence by OHEM, comprises: Randomly select an image as a reference sample, and select an image that belongs to the same label object and is different from the reference sample category as a positive sample; According to the OHEM strategy, using the currently trained convolutional neural network model to extract the cosine distance between features, for each reference sample, from other images that do not belong to the same label object, select the smallest distance and belong to a different category from the reference sample The image of , as the negative sample of the reference sample. 4. The method according to claim 2, wherein the triplet loss function includes a limitation on the cosine distance of samples of the same type, and a limitation on the cosine distance of samples of different types. 5. The method according to claim 4, wherein the triplet loss function is: <mrow><mover><mi>L</mi><mo>~</mo></mover><mo>=</mo><munderover><mo>&amp;Sigma;</mo><mi>i</mi><mi>N</mi></munderover><msub><mrow><mo>&amp;lsqb;</mo><mi>c</mi><mi>o</mi><mi>s</mi><mrow><mo>(</mo><mi>f</mi><mo>(</mo><msubsup><mi>x</mi><mi>i</mi><mi>a</mi></msubsup><mo>)</mo><mo>,</mo><mi>f</mi><mo>(</mo><msubsup><mi>x</mi><mi>i</mi><mi>n</mi></msubsup><mo>)</mo><mo>)</mo></mrow><mo>-</mo><mi>c</mi><mi>o</mi><mi>s</mi><mrow><mo>(</mo><mi>f</mi><mo>(</mo><msubsup><mi>x</mi><mi>i</mi><mi>a</mi></msubsup><mo>)</mo><mo>,</mo><mi>f</mi><mo>(</mo><msubsup><mi>x</mi><mi>i</mi><mi>p</mi></msubsup><mo>)</mo><mo>)</mo></mrow><mo>+</mo><msub><mi>&amp;alpha;</mi><mn>1</mn></msub><mo>&amp;rsqb;</mo></mrow><mo>+</mo></msub><mo>+</mo><munderover><mo>&amp;Sigma;</mo><mi>i</mi><mi>N</mi></munderover><msub><mrow><mo>&amp;lsqb;</mo><msub><mi>&amp;;alpha;</mi><mn>2</mn></msub><mo>-</mo><mi>c</mi><mi>o</mi><mi>s</mi><mrow><mo>(</mo><mi>f</mi><mo>(</mo><msubsup><mi>x</mi><mi>i</mi><mi>a</mi></msubsup><mo>)</mo><mo>,</mo><mi>f</mi><mo>(</mo><msubsup><mi>x</mi><mi>i</mi><mi>p</mi></msubsup><mo>)</mo><mo>)</mo></mrow><mo>&amp;rsqb;</mo></mrow><mo>+</mo></msub></mrow> Among them, cos(·) represents the cosine distance, and its calculation method is N is the number of triplets, Represents the feature vector of the reference sample, Represents the feature vector of the same positive sample, Represents the eigenvector of heterogeneous negative samples, and the meaning of [ ] ₊ is as follows: α ₁ is the inter-class interval parameter, and α ₂ is the intra-class interval parameter. 6. The method according to claim 2, characterized in that, the method also includes: initializing the basic model parameters based on massive open source face data training, adding a normalization layer and a ternary after the feature output layer The group loss function layer is used to obtain the convolutional neural network model to be trained. 7. A face authentication device based on Triplet Loss, comprising: an image acquisition module, an image preprocessing module, a feature acquisition module, a calculation module and an authentication module; The image acquisition module is used to acquire a photo of a certificate and a scene photo of a person based on a face authentication request; The image preprocessing module is used to perform face detection, key point positioning and image preprocessing on the scene photo and the ID photo, respectively, to obtain the scene face image corresponding to the scene photo, and the ID photo The corresponding ID face image; The feature acquisition module is used to input the scene face image and the certificate face image into the pre-trained convolutional neural network model for face authentication, and obtain all the output of the convolutional neural network model. The first feature vector corresponding to the face image of the scene, and the second feature vector corresponding to the face image of the certificate; wherein, the convolutional neural network model is obtained based on the supervised training of the triple loss function; The calculation module is used to calculate the cosine distance between the first eigenvector and the second eigenvector; The authentication module is configured to compare the cosine distance with a preset threshold, and determine a face authentication result according to the comparison result. 8. The device according to claim 7, further comprising: a sample acquisition module, a triplet acquisition module, a training module and a verification module; The sample acquisition module is used to acquire marked training samples, the training samples include a certificate face image and at least one scene face image marked with each marked object; The triplet acquisition module is used to train the convolutional neural network model according to the training samples, and generate triplet elements corresponding to each training sample through OHEM; the triplet elements include reference samples, positive samples and negative samples ; The training module is used to train the convolutional neural network model based on the triplet elements of each training sample, based on the supervision of the triplet loss function; the triplet loss function uses cosine distance as a measurement method, through Stochastic gradient descent algorithm to optimize model parameters; The verification module is used to input the verification set data into the convolutional neural network model, and when the training end condition is reached, the trained convolutional neural network model for face authentication is obtained. 9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, characterized in that, when the processor executes the computer program, any one of claims 1 to 6 is realized. The steps of the face authentication method based on Triplet Loss described in item. 10. A storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the face authentication method based on Triplet Loss described in any one of claims 1 to 6 are realized.