CN113076927B - Method and system for finger vein recognition based on multi-source domain migration - Google Patents

Abstract

The method for recognizing finger veins based on multi-source domain migration provided by the present invention includes the following steps: first, obtain the ROI corresponding to the finger vein picture in the target domain; then input the ROI into the basic feature extraction network to extract basic features; and then Input the basic features into CFTN and DFTN respectively to obtain general features and domain-specific features; after splicing the general features and domain-specific features, the final aggregated features are obtained; the obtained aggregated features are searched in the existing finger vein feature database , get the matching score of the input finger vein picture and the existing finger vein in the database; finally output the matching result of the input finger vein according to the matching score. By using multiple source domains to migrate to the target domain, the sample requirements of the target domain are reduced, thereby reducing the cost of collection and labeling; the general knowledge of multiple source domain datasets is transferred to the target domain, and domain-specified features are retained, thereby maximizing the improvement Performance of transfer learning.

Description

Method and system for finger vein recognition based on multi-source domain migration

technical field

The invention belongs to the field of biological feature recognition, in particular to a finger vein recognition method and system based on multi-source domain migration.

Background technique

Finger vein recognition is a newly proposed modality for biometric recognition that is user-friendly, highly secure, and possesses natural anti-counterfeiting properties. More and more researchers and engineers have begun to devote themselves to research in the field of finger vein recognition. However, in the practice of finger vein engineering, finger vein recognition is still affected by the following two problems:

Image difference: Due to the difference between the optical imaging sensor and the near-infrared light, the finger vein images collected by different devices in different scenarios are different, as shown in Figure 1. In addition, their grayscale histograms and local binary coded images are also listed below to illustrate their differences. In addition, different irradiation intensities such as near-infrared light will also cause nonlinear degradation to the finger vein image. Since this image degradation is highly nonlinear, the nonlinear degradation caused by illumination intensity cannot be directly modeled, thereby eliminating the influence of illumination intensity. In addition, different image sensors have different spectral response curves, which will also lead to different imaging results for the same finger under the same light. Although the above two reasons have been proposed, there is still no suitable method to solve them due to the limitation of the precision of electronic equipment, the limitation of optical equipment, and the limitation of manufacturing cost. Therefore, in the actual application of finger veins, there are inevitable differences in the finger vein images collected by different devices, which will cause the deep learning model trained on a data set to be unable to be well applied to the newly designed finger vein device .

Missing data: Finger veins are a new biometric modality compared to fingerprints, faces, and gestures. At present, the amount of training data in the finger vein database is not enough, and the variation within the class is not large enough, which will affect the performance of the finger vein recognition algorithm. However, most of the current pattern recognition and deep learning methods require a large amount of data sets to obtain an effective and robust recognition model. To alleviate the data shortage problem, one solution is to pre-train on related datasets with sufficient data and then fine-tune or jointly train on the target dataset. But deep learning models suffer from severe forgetting problems when fine-tuned from multiple source domains, and joint training discards common features across multiple datasets.

Due to its extremely high security and unique liveness detection, finger vein recognition is gaining more and more influence in academia and industry. Most methods, especially those based on deep learning, tend to use a large amount of training data to obtain an effective and robust recognition model. However, in practical applications, collecting sufficient data for each newly designed finger vein recognition device is time-consuming, labor-intensive and expensive. Therefore, how to obtain an optimal model under the experimental setting of small samples has become a recent research hotspot. A common approach to this type of few-shot learning problem is to use fine-tuning or joint training. But fine-tuning from multiple source domains suffers from severe forgetting problems, while joint training on multiple source domains discards common features.

Kyoung Jun Noh et al. (Noh K J, Choi J, Hong J S, et al.Finger-VeinRecognition Using Heterogeneous Databases by Domain Adaption Based on aCycle-Consistent Adversarial Network[J].Sensors,2021,21(2):524) for To solve the problem of multi-source domains in finger vein recognition, the test effect of deep learning models on multiple data sets is relatively poor, and a method of using CycleGAN to improve the recognition performance on heterogeneous data sets is proposed. That is to use CycleGAN to migrate at the image layer to achieve the purpose of the model being able to adapt to multiple different finger vein fields. The overall process of the algorithm is as follows: a. Preprocess the collected pictures; b. Send the preprocessed ROI pictures to CycleGAN to generate domain-adaptive pictures; c. Send the domain-adaptive pictures to the deep neural network Perform feature extraction; d. Use the features extracted in the previous step to perform feature matching and retrieval to obtain the final recognition result. But CycleGAN is an unsupervised generative adversarial network, and its main idea is to train two pairs of generator-discriminator models to convert images from one domain to another, requiring cycle consistency in the process. That is, after sequentially applying the generator, you should get an image similar to the original L1 loss. Therefore, a cyclic loss function (cyclic loss) is needed, which can ensure that the generator does not transfer images from one domain to another domain that is completely unrelated to the original image. However, such conversion is usually unstable, and it is usually required that the distribution of input images during testing should be completely consistent with the distribution of images during training, but this requirement is difficult to meet in actual use. Therefore, using CycleGAN for domain adaptation of the image layer will affect the quality of the original image.

Guoqing Wang et al. (Wang G, Sun C, Sowmya A. Learning a compact vein discrimination model with GANerated samples[J]. IEEE Transactions on Information Forensics and Security, 2019, 15:635-650) in order to solve small samples in finger vein recognition problem, a cascaded Generative Adversarial Network (GAN) is proposed for data augmentation. However, GANs consist of a generator and a discriminator. The discriminator makes D(G(Z)) close to 0 as much as possible, and the generator tries to generate high-quality samples with the same distribution as much as possible to make D(G(Z)) close to 1. When the performance of the generator and the discriminator are trained well enough, a Nash equilibrium is reached (G(Z) generated by the generator has the same distribution as the training data, and for each input x of the discriminator, D(x)=0.5) . There are three reasons why the training of the generative confrontational neural network is unstable: a. It is difficult to make a pair of models (G and D at the same time) converge. Most deep model training uses optimization algorithms to find lower values of the loss function. Optimization algorithms are usually a reliable "downhill" process. Generating an adversarial neural network requires both parties to reach an even match (equilibrium) in the process of the game. Each model successfully "goes downhill" during the update process (such as the generator), but the same update may cause another model of the game (such as the discriminator) to "go uphill". Even sometimes, although the two sides of the game finally reach an equilibrium, the continuous offsetting of the progress of the other side does not make both sides reach a useful place at the same time. Simultaneous gradient descent for all models leads some models to converge but not all models to converge optimally. b. Mode collapse occurs in the generator G: similar samples are generated for different inputs, and only a single sample is generated in the worst case, and the learning of the discriminator will reject these similar or even the same single samples. In practical applications, complete mode collapse is rare, and partial mode collapse is common. Partial mode crashes are when the generator makes different images contain the same color or texture theme, or different images contain different parts of the same dog. MinBatch GAN alleviates the problem of mode collapse but also causes problems such as counting, perspective and global structure. c. The problem of generator gradient disappearance: When the discriminator is very accurate, the loss of the discriminator quickly converges to 0, which cannot provide a reliable path to continue updating the generator gradient, causing the generator gradient to disappear. The training of GAN is due to the random noise distribution at the beginning, which is too far away from the real data distribution, and there is almost no overlap between the two distributions. At this time, the discriminator can quickly learn to distinguish the real data from the generated fake data. To achieve the optimum of the discriminator, the gradient of the generator cannot continue to be updated or even the gradient disappears. That is, the domain adaptive method and data amplification method based on GAN are relatively unstable.

Contents of the invention

In order to solve the problems of small sample identification and multi-source domain migration in the prior art, the present invention provides a finger vein recognition method based on multi-source domain migration, which reduces the sample requirements of the target domain by using multiple source domains to migrate to the target domain , so as to reduce the cost of collection and labeling; transfer the general knowledge of multiple source domain data sets to the target domain, and retain the domain-specific features, so as to maximize the performance of transfer learning.

In order to achieve the purpose of the invention, the finger vein recognition method based on multi-source domain migration provided by the present invention includes the following steps:

Construct and train a decoupled transfer learning network, which includes an embedding network, a general feature transformation network and a target domain-specific feature transformation network, the embedding network is used to convert finger vein images into basic features, and the general feature transformation network It is used to decouple the basic features to obtain general features, and the target domain domain-specified feature transformation network is used to decouple the basic features to obtain domain-specified features. When training the decoupled transfer learning network, the target domain and multiple source domains are used for Training, the domain-specific features and general knowledge of multiple source domains are transferred to the target domain, and the target domain domain-specific feature transformation network is obtained;

Input the region of interest of the finger vein picture to be recognized into the embedding network to obtain the basic features;

The basic features are input into the general feature transformation network and the target domain domain-specific feature transformation network respectively, and the general features and domain-specific features are obtained respectively;

Concatenate the obtained general features and domain-specific features to obtain aggregated features;

Calculate the cosine distance between the aggregated features and all registered sample features in the registered sample database, and obtain the matching score of the input finger vein picture and the existing finger veins in the database;

Output the matching result of the input finger vein according to the matching score.

Further, the method steps of the training decoupling transfer learning network include:

First, for multiple source domains, randomly sample mini-batches in any two source domains, and perform basic feature extraction, general feature extraction, and domain-specific feature extraction on the mini-batches of the two source domains, and then use the overall The loss function calculates the loss between two mini-batches and uses the RMSprop algorithm to update the DTL network;

A mini-batch is randomly sampled in any source domain and target domain, and the basic feature extraction, general feature extraction, and domain-specific feature extraction are performed on the mini-batch of the source domain and the target domain, and then the overall loss function is used to calculate The loss between two mini-batches and update the DTL network using the RMSprop algorithm;

Randomly sample the mini-batch of the target domain, perform basic feature extraction, general feature extraction, and domain-specific feature extraction on the mini-batch, and then use the target domain loss function to calculate the loss and use the RMSprop algorithm to update the DTL network.

Further, the calculation formula of the target domain loss function is as follows:

表示DFTN的交叉熵损失，β是中心损失的系数，L_center表示中心损失。In the formula,

Represents the cross-entropy loss of DFTN, β is the coefficient of the center loss, and L _center represents the center loss.

Further, the calculation formula of the overall loss function is as follows:

in,

表示通用特征变换网络CFTN的交叉熵损失；

表示DFTN的交叉熵损失； L_mmmd表示多源域最大均值差异损失；L_center表示中心损失，α和β分别是MMMD损失和中心损失的系数，N是整个指静脉混合数据集的类别数目，y表示该样本类别标签的one-hot编码， y_a表示类别标签one-hot编码的第a个类别的指示值；l_a(g)是通用特征g属于第a个类别的分类分数，Z表示源域的样本数量，g^m表示第m个源域的通用特征，g^t表示目标域上的通用特征，mmd(g^m，gⁿ)表示第m个源域和第n个源域之间通用特征的两个小批量数据之间的MMD距离。In the formula,

Represents the cross-entropy loss of the general feature transformation network CFTN;

Represents the cross-entropy loss of DFTN; L _mmmd represents the maximum mean difference loss in multi-source domain; L _center represents the center loss, α and β are the coefficients of MMMD loss and center loss respectively, N is the category number of the entire finger vein mixed data set, y Indicates the one-hot encoding of the sample category label, y _a indicates the indicator value of the a-th category of the category label one-hot encoding; l _a (g) is the classification score of the general feature g belonging to the a-th category, and Z indicates the source The number of samples in the domain, g ^m represents the general feature of the mth source domain, g ^t represents the general feature on the target domain, mmd(g ^m , g ⁿ ) represents the common feature between the mth source domain and the nth source domain The MMD distance between two mini-batches of a feature.

Further, the classification score l _a (g) in the cross-entropy loss of the general feature transformation network CFTN is calculated as follows:

之间的角度值；θ_b是特征g和对应超平面

之间的角度值，M为加性角度间隔惩罚项。In the formula, s is a hyperparameter, which represents the scale factor of scaling; θ _a is the feature g and the corresponding hyperplane

The angle value between; θ _b is the feature g and the corresponding hyperplane

The angle value between, M is the additive angle interval penalty term.

Further, the general feature transformation network includes a network structure and a pooling layer, the network structure is used to perform feature transformation on the general features, and the pooling layer is used to pool the mean value of each channel into a value along the channel dimension, so that the final The means of each channel form the feature vector.

Further, the target domain specified feature transformation network includes a network structure and a pooling layer, the network structure is used to perform feature transformation on general features, and the pooling layer is used to pool the mean value of each channel into a value along the channel dimension , so that the final mean of each channel forms a feature vector.

Further, the region of interest of the finger vein picture to be recognized is input into the network, and the finger vein picture to be recognized is processed by grayscale, edge extraction operator processing, finger edge acquisition, and ROI interception to obtain the The region of interest ROI corresponding to the picture.

Further, outputting the matching result of the input finger vein according to the matching score includes: sorting the similarity between the input finger vein image and all registered samples from high to low according to the matching score, using the sorting and adopting The decision algorithm obtains the final decision matching result.

The present invention also provides a finger vein recognition system based on multi-source domain migration, which is used to implement the aforementioned method, including: a network building module, used to build and train a decoupled transfer learning network, the decoupled transfer learning network includes an embedded network, The general feature transformation network and the target domain specified feature transformation network, the embedding network is used to convert finger vein images into basic features, the general feature transformation network is used to decouple the basic features to obtain general features, and the target domain specified feature transformation network is used for Decoupling the basic features to obtain domain-specific features, in which, when training the decoupled transfer learning network, the target domain and multiple source domains are used for training, and the domain-specific features and general knowledge of multiple source domains are transferred to the target domain to obtain The target domain specifies the feature transformation network;

The basic feature extraction module is used to input the region of interest of the finger vein picture to be identified into the embedding network to obtain the basic features;

The general feature extraction module is used to input the basic features into the general feature transformation network to obtain the general features;

The domain specified feature extraction module is used to input the target domain domain specified feature transformation network to obtain the domain specified feature;

The aggregation module is used to splice the obtained general features and domain-specific features to obtain aggregated features;

Calculation module, used to calculate the cosine distance between the aggregation feature and all registered sample features in the registered sample database, and obtain the matching score of the input finger vein picture and the existing finger vein in the database;

The matching module is configured to output the matching result of the input finger vein according to the matching score.

The present invention proposes a brand-new decoupling transfer learning method to learn general knowledge between different finger vein databases, and then transfer the extracted general knowledge to the target database. Experiments show that the decoupling transfer learning method proposed by the present invention can well extract general knowledge in multiple data sets collected by different devices to realize finger vein recognition under the smallest training samples.

1) After analyzing the differences between different finger vein images collected by different devices, the present invention proposes a method to decouple the basic features of finger vein images into general features and domain-specific features. Among them, general features include general information for identification, and domain-specific features include features affected by factors that are easy to collect and external factors.

2) A framework for multi-source domain decoupled transfer recognition is proposed, enabling transfer learning from multiple source domains to a single target domain. Therefore, a finger vein recognition model can be trained more fully and can achieve superior performance under the condition of small samples.

3) The best result under small sample recognition is obtained through experiments, and the experiments show that the present invention can solve the performance of finger vein recognition when training data is insufficient. Especially for the actual engineering application of finger vein recognition, it can prevent a lot of manpower and material resources from collecting data and labeling data centers during the collection process.

Description of drawings

Fig. 1 is a schematic diagram of finger vein images and GLH and LBP of different data sets in the prior art.

Fig. 2 is a schematic diagram of a decoupled transfer learning network in a finger vein recognition method based on multi-source domain transfer provided by an embodiment of the present invention.

Fig. 3 is a schematic diagram of the division of the feature space in the case of different numbers of training set categories in the embodiment of the present invention.

Fig. 4 is a schematic diagram of manifolds in which features of finger vein images in different fields fall in different manifolds according to an embodiment of the present invention.

Fig. 5 is a schematic diagram of the CFTN structure in the embodiment of the present invention.

Fig. 6 is a schematic diagram of normalization and distribution alignment of two different manifolds in an embodiment of the present invention.

Fig. 7 is a schematic diagram of a DFTN structure in an embodiment of the present invention.

Fig. 8 is a flowchart of finger vein recognition in an embodiment of the present invention.

Fig. 9 is a schematic structural diagram of a system provided by an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts are within the protection scope of the present invention.

Please refer to Fig. 8, the multi-source domain migration-based finger vein recognition method provided by the present invention includes the following steps:

Step 1: Construct and train a decoupled transfer learning network, which includes an embedding network, a general feature transformation network, and a target domain-specific feature transformation network. The embedding network is used to convert finger vein images into basic features, and the general The feature transformation network is used to decouple the basic features to obtain general features, and the target domain domain-specific feature transformation network is used to decouple the basic features to obtain domain-specific features.

Step 1.1: The constructed decoupled transfer learning network is as follows:

Deep learning methods have been widely used in the field of biometric recognition due to their powerful feature extraction capabilities. Many outstanding network structures, such as AlexNet, ResNet, MobileNet, SENet, etc. have been proposed and have achieved remarkable results in different fields. Generally speaking, there are three ways to scale a deep neural network to adapt to different tasks: width scale, depth scale, and resolution scale, but modifying a single scale can only modify the network capacity from a single factor. However, the networks that have been proposed above mainly focus on the single-layer or single-block network structure, and its optimal width, depth, and resolution still need to be determined through hyperparameter tuning. In the practical application of finger vein recognition, how to select a suitable network size to improve performance as much as possible under limited computing resources is still an urgent problem to be solved. Especially when the deep learning model is deployed on an embedded platform, the trade-off between computation and performance is particularly important. Therefore, a neural network architecture search technology (Network Architecture Search, NAS) can be used to search for network structures under different parameter quantities. The embedding network can use deep neural networks such as EfficientNet and NasNet. These deep neural networks can have strong feature extraction capabilities under limited computing resources.

In one embodiment of the present invention, the embedding network in the decoupling transfer learning network adopts the EfficientNet network. The EfficientNet network is an existing network, such as in the document "Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks [C]//International Conference on Machine Learning. PMLR, 2019: 6105-6114." The network used in this embodiment has the same composite scale parameters and basic structure as the network used in this example. In EfficienetNet-B1, the width, depth, resolution and dropout probability are 1.0, 1.0, 224 and 0.2, respectively. The main block of the network is the mobile inverted bottleneck (MBConv) combined with squeeze-and-excitation (SE) optimization.

The basic features b extracted by the embedding network are relatively low-level and contain all general and domain-specific features. In order to decouple these two features, the present invention connects a dual-stream structure behind the embedding network to decouple the basic features into domain-independent and domain-related knowledge. One of the branches is the general feature transformation network CFTN, which receives the feature map b from the output of the embedding network, and then outputs the general feature g. It is used to extract general features g=G(b), and the other branch is the target domain domain-specific feature transformation network DFTN, which is used to extract domain-specific features.

In one embodiment of the present invention, please refer to FIG. 5 , the general feature transformation network CFTN includes a two-dimensional convolution and a global average pooling (GAP). Through a single two-dimensional convolution, feature transformation can be performed on general features, so that the transformed features can adapt to different source domains. In order to finally obtain the feature vector, the mean value of each channel is pooled into a value along the channel dimension through global average pooling, and finally the mean value of each channel forms a feature vector. Of course, the general-purpose feature transformation network CFTN is not limited to the above-mentioned structure. In other embodiments, the network structure of the general-purpose feature transformation network CFTN can use a combination of ordinary convolutional networks, SE modules, residual network modules, etc., and the pooling layer Global max pooling or SPP pooling can be used.

The target domain domain-specified feature transformation network receives the feature map b of the finger vein picture to be recognized output from the embedding network, and outputs domain-specified features. In one embodiment of the present invention, please refer to FIG. 7 , the target domain-specific feature transformation network includes a two-dimensional convolution and a global average pooling (GAP), through a single two-dimensional convolution, the general features can be characterized Transformation, so that the transformed features can target different source domains. In order to finally obtain the feature vector, the global average pooling is used to pool the mean value of each channel into a value along the channel dimension, and finally the mean value of each channel forms the feature vector. Of course, the target domain-specified feature transformation network is not limited to the aforementioned structure. In other embodiments, the network structure of the target domain-specified feature transformation network DFTN can adopt ordinary convolutional networks, SE modules, residual network modules, etc. In combination, the pooling layer can adopt global maximum pooling or SPP pooling; the network structure in the example of the present invention adopts simple two-dimensional convolution; the pooling layer adopts global average pooling.

Step 1.2: Train the decoupled transfer learning network. The decoupled transfer learning network is equipped with an embedding network, a general feature transformation network CFTN and multiple domain-specific feature transformation networks DFTN during training. Among them, multiple domain-specific feature transformation networks include A target domain-specified feature transformation network and multiple source domain-specified feature transformation networks, the specific training process is as follows:

In one embodiment of the present invention, in the feature space, the decoupled transfer learning method proposed by the present invention can reduce domain differences between different data sets and decouple basic features into general features and domain-specific features.

的指静脉图像x分别被嵌入网络F转换为相应的基础特征b，这些不同的嵌入网络是共享参数的。然后基础特征b被两个特征变换网络解耦。第一条支路是域指定特征变换网络，这条支路是和特定领域相关的。在这条支路上，基础特征b被变换为领域指定特征h。每个领域都有一个自己独特的DFTN，即每个领域的网络参数不一样，并且这条支路最终得到的领域指定特征只在他们对应的领域上进行分类。另一条支路通用特征变换网络，他在多个领域之间是参数共享的。这条支路上得到的通用特征g是在整个混合数据集上进行分类的。在测试阶段，指静脉图像被转换为领域指定特征和通用特征，这两类特征最终被串联成为最终的混合特征，用于指静脉认证。The entire framework of the decoupled transfer learning network DTL during training is shown in Figure 2: the embedding network F is used to extract the basic feature b: b=F(x), where x is the finger vein image of domain D, from different domains

The finger vein images x of are respectively converted into corresponding basic features b by the embedding network F, and these different embedding networks share parameters. Then the base feature b is decoupled by two feature transformation networks. The first branch is the domain-specific feature transformation network, which is domain-specific. In this branch, the base feature b is transformed into a domain-specific feature h. Each domain has its own unique DFTN, that is, the network parameters of each domain are different, and the domain-specific features finally obtained by this branch are only classified in their corresponding domains. Another branch is the general feature transformation network, which is parameter shared among multiple domains. The general feature g obtained on this branch is classified on the whole mixed data set. In the testing stage, finger vein images are transformed into domain-specific features and general features, and these two types of features are finally concatenated into the final hybrid features for finger vein authentication.

上训练通用特征变换网络CFTN，其中，

表示不同源域的数据集，D^t表示目标域的数据集。The general features extracted by the general feature transformation network CFTN are domain-independent, so the present invention is used in the mixed data set that mixes the target domain and all source domains

On training the general feature transformation network CFTN, where,

Denote the datasets of different source domains, and D ^t denote the datasets of the target domain.

In one embodiment of the present invention, when training on a mixed data set, the cross-entropy loss of the general feature g can be expressed as:

Among them, N is the number of categories of the entire finger vein mixed data set, y represents the one-hot encoding of the sample category label, and _{y a} indicates the 0/1 indicator value of the ath category of the category label one-hot encoding; l _a ( g) is the classification score of the generic feature g belonging to the a-th category.

In one embodiment of the present invention, in classification or agent-based metric learning, a common way to calculate the classification score l _a (g) is to use the softmax equation, the formula is as follows:

表示长度为d的一维实数空间；

表示权重

的第a行，

表示权重

的第b行，

表示长度分别为d和N的二维实数空间；B_a、

表示偏置项，，

为超平面，b表示第b个类别。Among them, d represents the length of the feature vector, and N represents the number of categories of the finger vein mixed data set;

Represents a one-dimensional real number space of length d;

Indicates the weight

row a of

Indicates the weight

line b of

represent the two-dimensional real number space whose lengths are d and N respectively; B _a ,

represents the bias term,

is a hyperplane, and b represents the bth category.

θ_b表示特征g和W_b之间的向量夹角。为了更进一步地优化，加性角度间隔惩罚项M被加在了角度值θ上。因此最终的分类分数l_a(g) 可以表示为：In order to push the entire feature space onto a hypersphere, in one embodiment of the present invention, the general feature is unitized to g: g=g/|g|. At the same time, this embodiment refers to the excellent performance of the additive angular margin loss (ArcFace) in face recognition, sets the bias item B _b to 0, and normalizes each column of the weight matrix W _ya . So the inner product of W _b and g can be expressed as the cosine of an angle

θ _b represents the vector angle between feature g and W _b . For further optimization, an additive angle interval penalty term M is added to the angle value θ. So the final classification score l _a (g) can be expressed as:

之间的角度值；θ_b是特征g和对应超平面

之间的角度值。在测试阶段，本发明计算测试样本对x_i和x_j的通用特征g_i和g_j之间的余弦距离，其中，x_i和x_j为不同领域的指静脉图像，g_i和g_j是嵌入网络分别输入x_i和x_j得到的通用特征：余弦距离distance＝|g_i||g_j|cos(θ)。最终，使用正则化和加性角度间隔，能够使得作用在训练阶段的损失惩罚作用和作用在测试阶段的度量准则保持一致。Among them, s is a hyperparameter, representing the scale factor of scaling; θ _a is the feature g and the corresponding hyperplane

The angle value between; θ _b is the feature g and the corresponding hyperplane

Angle value between. In the test phase, the present invention calculates the cosine distance between the general features g _i and g _j of the test sample pair x _i and x _j , where x _i and x _j are finger vein images in different fields, g _i and g _j are The general features obtained by embedding the network into x _i and x _j respectively: cosine distance distance=|g _i ||g _j |cos(θ). Finally, using regularization and additive angle intervals, it is possible to align the loss penalties applied during training with the metrics applied during testing.

The general features of different fields obtained by the general feature transformation network are distributed and aligned.

The dimensionality of general features is fixed, but generally speaking, the entire feature space will not be covered by all existing feature points. The entire data set or domain actually falls on a subspace or low-dimensional manifold of the feature space. If the decoupled transfer learning network is directly trained on the mixed data set, although better performance may be obtained, the features extracted from different fields may still fall on different manifolds, as shown in Figure 6, the manifold The distribution A and the manifold distribution B are two different fields.

Therefore, in one of the embodiments of the present invention, the whole feature space is firstly compressed to a subspace (ie, a hypersphere) using a normalization method. In this hypersphere, compared with the original manifold in the entire feature space, the distribution differences of different domains will be narrowed due to the smaller feature space. Therefore, the manifold of the target domain is more likely to overlap with other domains, and more samples from the source domain are more likely to be projected onto the target manifold, which is equivalent to directly increasing the training samples in the feature space, and the source domain samples in the The feature space is transferred to the target domain. The distributions of the two manifolds in (b) overlap more easily in a normalized hypersphere than the entire original feature space in (a), on which the transfer between domains Easier and more efficient metric learning.

Normalization reduces the entire feature space to a hypersphere, making transfer easier and more efficient. However, different domains may lie in different regions on the same hypersphere. Therefore, the present invention proposes the Multi-Source Domain Maximum Mean Difference (MMMD) to align the distributions of multiple datasets, as shown in the following equation. MMMD is defined as follows:

和

计算这两个分布之间距离的方法是：In the formula, Z represents the number of samples in the source domain, and g ^m represents the general feature g ^m =G(F(x ^m )) of the mth source domain. g ^t represents the general feature g ^t =G(F(X ^t )) on the target domain. mmd(g ^m , g ⁿ ) represents the MMD distance between two mini-batches of common features between the mth source domain and the nth source domain. MMD is a kernel technique that measures the difference between two distributions based on a reproducible kernel Hilbert space. Suppose X=x ₁ ,...,x _n1 and Y=y ₁ ,...,y _n1 are two arbitrary sets, where x ₁ , x _n1 represent samples in set X, y ₁ ,... , y _n1 represents the samples in the set Y. The two sets have the same dimensionality and their distributions are

and

The way to calculate the distance between these two distributions is:

表示分布

上的数学期望，

表示分布

上的数学期望。其中

表示广义的的RKHS (Reproducing kernel Hilbert space，可再生核希尔伯特空间)。φ(·)表示将特征图从原始的样本空间映射到RKHS的映射函数，在符合

的前提下，选取核函数k(x，y)，使得k与φ(·)满足k(x，y)＝<φ(x)，φ(y))，其中<·，·>表示向量之间的内积。在解耦迁移学习方法中，核k采用一个高斯核函数，

和

两分布之间的距离

始终是非负的，并且当

n1，n2→∞时，其值不断趋向于0。in

Indicates the distribution

The mathematical expectation on

Indicates the distribution

Mathematical expectations on . in

Represents a generalized RKHS (Reproducing kernel Hilbert space, reproducible kernel Hilbert space). φ( ) represents the mapping function that maps the feature map from the original sample space to the RKHS.

Under the premise of , the kernel function k(x, y) is selected so that k and φ(·) satisfy k(x, y)=<φ(x), φ(y)), where <·,·> represent the vector Inner product between. In the decoupled transfer learning method, the kernel k uses a Gaussian kernel function,

and

distance between two distributions

is always non-negative, and when

When n1, n2→∞, its value tends to 0 continuously.

MMD is used to measure the distribution difference between source and target domains. While minimizing the MMD loss, deep neural networks are able to extract domain-independent features. Therefore, one embodiment of the present invention uses MMD to align multiple marginal distributions between source and target domains. As explained above, by minimizing the distribution difference between two manifolds A and B, features that are nearly aligned on the hypersphere can be extracted using deep neural networks. In this way, the finger vein images in different fields have almost the same feature distribution after passing through the deep neural network.

The general features of different fields extracted above are correspondingly input into the domain-specified feature transformation network DFTN, and different domain-specified features are extracted.

From the previous analysis, we can see that the separable features of different domains are very similar, but there are still some domain-related features in the separable features, and these features vary with the imaging characteristics of the finger vein imaging equipment. Therefore, in the method proposed by the present invention, another branch of the two-stream structure is the domain-specific feature transformation network DFTN. The domain-specific feature transformation network DFTN is used to extract specific domain-specific features, such as grayscale information of the background, brightness distribution, finger surface texture, and image gradient.

In one embodiment of the present invention, please refer to FIG. 7 , each domain-specific feature transformation network DFTN includes a two-dimensional convolution and a global average pooling (GAP). DFTN receives feature maps b from each domain output by the embedding network, and correspondingly outputs domain-specific features h for each domain.

In the workflow of the general feature transformation network CFTN, by normalizing the feature space to a sphere, the features of multiple source and target domains are restricted to the same manifold as much as possible. However, directly forcing the feature distributions of multiple source and target domains to become exactly the same may lead deep neural networks to only focus on features that are relatively common in different domains. In this case, some domain-specific features may be discarded in the feature extraction stage. From this point of view, in the decoupled transfer learning proposed by the present invention, since CFTN only focuses on general-purpose features, DFTN plays an important role in preserving domain-specific features.

This dual-stream mechanism can relieve the pressure of deep neural networks in a specific domain while learning general knowledge and domain-specific knowledge.

不再重复列举。In the general feature transformation network CFTN and the domain-specific feature transformation network DFTN, the only difference in cross-entropy loss is the number of categories of samples. In the general feature transformation network CFTN, the features are classified on the mixed dataset, so N is the number of categories in the mixed dataset. In domain-specific feature transformation network (DFTN), N is the number of categories for a single domain-specific. Therefore, to prevent duplication, the cross-entropy loss of DFTN

Do not repeat the enumeration.

In order to obtain the optimal parameters, the present invention uses three losses to train the deep neural network: MMMD loss, ArcFAce loss and center loss. The output features of the general feature transformation network CFTN are used to calculate the MMMD loss, which can reduce domain differences and extract common features between multiple domains. The output of the general feature transformation network CFTN and the domain specific feature transformation network DFTN will be used to calculate the ArcFace loss and center loss. The additional normalized additive angle-spaced cross-entropy loss works well for single-domain metric learning. Regularization operations also remove the impact of absolute values, and an additive angular margin loss provides additional margins for classification boundaries. However, the intra-class distance is still not strongly constrained. Therefore, the present invention uses an additional center loss to extract more effective and robust intra-class features in the Euclidean space.

表示第y_i个类的特征中心，并且g_i也属于类y_i。F表示 mini-batch的大小。与交叉熵损失类似的是，通用特征变换网络CFTN的中心损失的类别数与混合数据集的类别数相同；特征变换网络DFTN的中心损失的类别数与单个特定领域的类别数相同。In this formula, _LC is the center loss,

represents the feature center of the y _i -th class, and g _i also belongs to the class y _i . F represents the size of the mini-batch. Similar to the cross-entropy loss, the number of categories of the central loss of the general feature transformation network CFTN is the same as that of the mixed dataset; the number of categories of the central loss of the feature transformation network DFTN is the same as that of a single domain.

As emphasized in the introduction of MMMD loss, the data in the target domain is insufficient, so it is relatively more prone to statistical errors. Therefore, the design idea of the training strategy is: learn knowledge from a large source domain and update the network at a small scale. In fact, how to use a small number of samples to optimize the feature extractor more effectively is the key to vein small sample learning. In order to obtain more stable knowledge, the present invention first uses the sample pairs of the source domain to train the network, and uses the following loss function to make the general network better adapt to the target domain:

Among them, α and β are the coefficients of MMMD loss and center loss respectively, and the final loss on a separate target domain can be written as:

表示通用特征变换网络CFTN的交叉熵损失；

表示DFTN的交叉熵损失；L_mmmd表示多源域最大均值差异损失；L_C表示中心损失。Among them, L represents the loss that is finally used for backpropagation;

Represents the cross-entropy loss of the general feature transformation network CFTN;

Indicates the cross-entropy loss of DFTN; L _mmmd indicates the multi-source domain maximum mean difference loss; L _C indicates the center loss.

That is, the training process is as follows:

First, for multiple source domains, randomly sample mini-batches in any two source domains, and perform basic feature extraction, general feature extraction, and domain-specific feature extraction on the mini-batches of the two source domains, and then use the formula (8) Calculate the loss between two mini-batches and use the backpropagation algorithm to update the DTL network; secondly, randomly sample the mini-batch in any source domain and target domain, and compare the source domain and target domain The mini-batch of the mini-batch performs basic feature extraction, general feature extraction, and domain-specific feature extraction respectively, and then uses the formula (8) to calculate the loss between the two mini-batches and uses the backpropagation algorithm to update the DTL network; finally, the target domain Randomly sample the mini-batch, perform basic feature extraction, general feature extraction, and domain-specific feature extraction on the mini-batch, then use formula (9) to calculate the loss and use the back propagation algorithm to update the DTL network.

Wherein, in one embodiment of the present invention, the backpropagation algorithm adopts the RMSprop algorithm. Of course, in other embodiments, other algorithms may also be used, such as SGD algorithm (stochastic gradient descent) or Adam algorithm or Adagrad algorithm.

Step 2: Collect finger vein pictures through the finger vein collection device.

Step 3: Input the picture of the finger vein to be recognized into the embedding network to obtain the basic features.

Firstly, the finger vein image to be recognized is processed by gray scale, edge extraction operator, finger edge acquisition, and ROI interception to obtain the ROI corresponding to the image; then input the ROI into the network to extract Obtain the basic features of the finger vein map to be recognized.

Step 4: Input the basic features extracted by the embedding network F into the general feature transformation network CFTN, and extract general features. Input the basic features extracted by the embedded network F into the domain-specified feature transformation network of the target domain, and extract the domain-specified features of the target domain;

Step 5: Concatenate the obtained general features and domain-specific features to obtain aggregated features;

Step 6: Calculate the cosine distance between the aggregated feature and all registered sample features in the registered sample database, and obtain the matching score between the input finger vein picture and the existing finger veins in the database;

Step 7: According to the matching score, sort the similarity between the input finger vein image and all the registered samples from high to low, use this sort and use the decision algorithm to get the final decision matching result. The decision algorithm can use any of the maximum score matching, re-matching and top-k methods. In one embodiment of the present invention, top-k is used as a decision-making algorithm, the largest K samples are taken out, and the registered samples corresponding to the similarity with the highest frequency of occurrence are used as the matching result of input finger veins.

In order to verify the effectiveness of the method proposed in the present invention, the present invention uses four public finger vein databases: FV-USM, MMCBNU-6000, THU-FVFDT, PolyU to conduct experiments, and the experimental results are shown in Table 1. As can be seen from Table 1, in the case of single-source domain migration in the present invention, under the condition of using exactly the same training set, verification set, and test set, the equal error rate obtained by the DTL method (the method provided by the present invention) It is obviously better than fine-tuning, proving that the DTL method is effective on single-source domain transfer. It can also be seen from the table that the DTL method can not only be applied to single-source domain migration, but also has a good effect on multi-source domain migration. When performing multi-source domain migration on the Fusion database (mixed dataset), better results can be obtained than cross-dataset testing and fine-tuning on the Fusion dataset. In multi-source domain transfer, the DTL method can not only transfer the common features in multiple palm vein datasets from multiple source domains, but also preserve the domain-specific features of each target domain. For the target domain, it not only transfers the knowledge of the source domain to the target domain, but also retains the domain-specific features of the target domain, so the DTL method can achieve a lower equal error rate.

Table 1

The present invention also provides a system for implementing the aforementioned method.

In one of the embodiments of the present invention, the finger vein recognition system based on multi-source domain migration is provided, including:

The network building module is used to construct and train a decoupling transfer learning network, the decoupling transfer learning network includes an embedding network, a general feature transformation network and a target domain domain specified feature transformation network, and the embedding network is used to convert the finger vein image into a base Features, the general feature transformation network is used to decouple the basic features to obtain general features, and the target domain domain-specified feature transformation network is used to decouple the basic features to obtain domain-specified features. When training the decoupled transfer learning network, the target domain is used Train with multiple source domains, transfer the domain-specific features and general knowledge of multiple source domains to the target domain, and obtain the target domain-specific feature transformation network;

The basic feature extraction module is used to input the region of interest of the finger vein picture to be identified into the embedding network to obtain the basic features;

The general feature extraction module is used to input the basic features into the general feature transformation network to obtain the general features;

The domain specified feature extraction module is used to input the target domain domain specified feature transformation network to obtain the domain specified feature;

The aggregation module is used to splice the obtained general features and domain-specific features to obtain aggregated features;

Calculation module, used to calculate the cosine distance between the aggregation feature and all registered sample features in the registered sample database, and obtain the matching score of the input finger vein picture and the existing finger vein in the database;

The matching module is configured to output the matching result of the input finger vein according to the matching score.

The system provided by this embodiment has the same beneficial effect as the above method.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. For the finger vein recognition system based on multi-source domain migration disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for relevant details, please refer to the description of the method part.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to these embodiments shown in the present invention, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention.

Claims (7)

1. The finger vein recognition method based on multi-source domain transfer, is characterized in that, comprises the following steps: Construct and train a decoupled transfer learning network, which includes an embedding network, a general feature transformation network and a target domain-specific feature transformation network, the embedding network is used to convert finger vein images into basic features, and the general feature transformation network It is used to decouple the basic features to obtain general features, and the target domain domain-specified feature transformation network is used to decouple the basic features to obtain domain-specified features. When training the decoupled transfer learning network, the target domain and multiple source domains are used for Training, the domain-specific features and general knowledge of multiple source domains are transferred to the target domain, and the target domain domain-specific feature transformation network is obtained; Input the region of interest of the finger vein picture to be recognized into the embedding network to obtain the basic features; The basic features are input into the general feature transformation network and the target domain domain-specific feature transformation network respectively, and the general features and domain-specific features are obtained respectively; Concatenate the obtained general features and domain-specific features to obtain aggregated features; Calculate the cosine distance between the aggregated features and all registered sample features in the registered sample database, and obtain the matching score of the input finger vein picture and the existing finger veins in the database; Outputting the matching result of the input finger vein according to the matching score; The method steps of the training decoupling transfer learning network include: First, for multiple source domains, randomly sample mini-batches in any two source domains, and perform basic feature extraction, general feature extraction, and domain-specific feature extraction on the mini-batches of the two source domains, and then use The overall loss function calculates the loss between two mini-batches and updates the DTL network using the backpropagation algorithm; Then, randomly sample a mini-batch in any source domain and target domain, and perform basic feature extraction, general feature extraction, and domain-specific feature extraction on the mini-batch of the source domain and target domain, and then use the overall loss The function calculates the loss between two mini-batches and updates the DTL network using the backpropagation algorithm; Finally, randomly sample a mini-batch from the target domain, perform basic feature extraction, general feature extraction, and domain-specific feature extraction on the mini-batch, and then use the target domain loss function to calculate the loss and use the back propagation algorithm to update the DTL network; The calculation formula of the target domain loss function is as follows:

(9) In the formula,

Denotes the cross-entropy loss of DFTN,

is the coefficient of the center loss,

Indicates the center loss; The calculation formula of the overall loss function is as follows:

; in,

;

;

In the formula,

Represents the cross-entropy loss of the general feature transformation network CFTN;

Represents the cross-entropy loss of DFTN;

Indicates the multi-source domain maximum mean difference loss;

represents the central loss,

and

are the coefficients of MMMD loss and center loss, respectively, and N is the number of categories in the entire finger vein mixed data set

represents the one-hot encoding of the sample category label,

Indicates the indicator value of the a-th category of the category label one-hot encoding;

is a common feature

the classification score belonging to the a-th class,

represents the sample size of the source domain,

Indicates the first

common features of source domains,

Denotes the general features of the nth source domain,

Represents general features on the target domain,

The MMD distance between two mini-batches representing common features between the mth source domain and the nth source domain,

Indicates the first

source domain and the

The MMD distance between two mini-batches of common features between source domains, F represents the size of the mini-batch,

Indicates the first

common features of the class,

Indicates the first

feature center of a class. 2. The finger vein recognition method based on multi-source domain migration according to claim 1, wherein the classification score in the cross-entropy loss of the general feature transformation network CFTN

The calculation formula is as follows:

In the formula, s is a hyperparameter, representing the scale factor of scaling;

is a feature

and the corresponding hyperplane

Angle value between;

is a feature

and the corresponding hyperplane

the angle value between,

is an additive angular interval penalty term. 3. The finger vein recognition method based on multi-source domain migration according to claim 1, wherein the general feature transformation network comprises a network structure and a pooling layer, and the network structure is used to perform feature transformation to general features, and the pooling The layer is used to pool the mean value of each channel into a value along the channel dimension, so that the final mean value of each channel forms a feature vector. 4. The finger vein recognition method based on multi-source domain migration according to claim 1, wherein the target domain-specified feature transformation network includes a network structure and a pooling layer, and the network structure is used to characterize general features Transformation, the pooling layer is used to pool the mean value of each channel into a value along the channel dimension, so that the mean value of each channel finally forms a feature vector. 5. The method for recognizing finger veins based on multi-source domain migration according to claim 1, wherein said inputting the region of interest of the finger vein picture to be recognized into the embedding network comprises: inputting the finger vein to be recognized After the image is grayscaled, edge extraction operator processing, finger edge acquisition, and ROI interception, the ROI corresponding to the image is obtained. 6. The finger vein recognition method based on multi-source domain migration according to any one of claims 1-5, wherein the outputting the matching result of the input finger vein according to the matching score comprises: according to the level of the matching score, The similarity between the input finger vein image and all registered samples is sorted from high to low, and the final decision matching result is obtained by using the sorting and decision-making algorithm. 7. The finger vein recognition system based on multi-source domain migration is characterized in that, for realizing the method described in any one of claims 1-6, the system comprises: The network building module is used to construct and train a decoupling transfer learning network, the decoupling transfer learning network includes an embedding network, a general feature transformation network and a target domain domain specified feature transformation network, and the embedding network is used to convert the finger vein image into a base Features, the general feature transformation network is used to decouple the basic features to obtain general features, and the target domain domain-specified feature transformation network is used to decouple the basic features to obtain domain-specified features. When training the decoupled transfer learning network, the target domain is used Train with multiple source domains, transfer the domain-specific features and general knowledge of multiple source domains to the target domain, and obtain the target domain-specific feature transformation network; The basic feature extraction module is used to input the region of interest of the finger vein picture to be identified into the embedding network to obtain the basic features; The general feature extraction module is used to input the basic features into the general feature transformation network to obtain the general features; A domain-specified feature extraction module is used to input the target domain domain-specified feature transformation network to obtain domain-specified features; The aggregation module is used to splice the obtained general features and domain-specific features to obtain aggregated features; Calculation module, used to calculate the cosine distance between the aggregation feature and all registered sample features in the registered sample database, and obtain the matching score of the input finger vein picture and the existing finger vein in the database; The matching module is configured to output the matching result of the input finger vein according to the matching score.

Images