CN115393896A - Cross-mode pedestrian re-identification method, device and medium of infrared visible light - Google Patents

Abstract

The embodiment of this specification discloses a cross-modal pedestrian re-identification method, equipment and medium of infrared and visible light, which relates to the field of image recognition technology. The image includes any one or more of visible light images and infrared images; the pedestrian image is input to the pre-trained cross-modal feature generation model, and the pedestrian cross-modal feature corresponding to the pedestrian image is generated, wherein the cross-modal feature generation The model is trained using a bimodal image dataset, and the loss functions include the center maximum mean difference loss function, the intra-class heterogeneity center loss function, and the cross-modal triplet loss function; the pedestrian cross-modal features are combined with the pre-built feature library Compare the features in the feature library to obtain the specified features that meet the requirements in the feature database, so as to determine the identification result of the pedestrian to be identified according to the specified features that meet the requirements.

Description

A cross-modal pedestrian re-identification method, device and medium of infrared and visible light

technical field

This specification relates to the technical field of image recognition, and in particular to a cross-modal pedestrian re-identification method, device and medium of infrared and visible light.

Background technique

Person Re-Identification (Re-ID for short) is an AI technology that uses computer vision technology to retrieve whether a specific pedestrian exists in an image or video sequence. It has important application significance and prospect. Cross-modal pedestrian recognition mainly refers to the unified representation and recognition of pedestrians and other targets based on multi-modal images such as visible light, infrared light, and thermal imaging, using cross-modal neural networks. The ability of visible light cameras to capture effective information in harsh lighting environments, such as nighttime environments, limits its applicability in practice. In contrast, infrared cameras are less dependent on lighting conditions and can capture More information. Dual-mode cameras (RGB and infrared modes) have been widely used in many surveillance systems due to their price advantages. The use of dual-mode cameras can effectively overcome the limitations of single-mode pedestrian matching. Research on cross-modal pedestrian re-identification of visible light infrared images is It is of great significance in practical application.

In harsh lighting environments, especially at night, visible light and infrared imaging can effectively complement each other. Existing cross-modal pedestrian recognition methods are generally studied based on modal shared feature learning or modal image generation. Modal shared feature learning maps images of different modalities to the same feature space, and learns common features between images, ignoring the relationship between modality (whole) and identity (individual), resulting in possible modalities in the extracted features. state difference. Modal image generation supplements the missing information by generating modality-specific images, however, the generated images are usually unstable and the texture of the generated images may be severely missing. To sum up, it can be seen that when performing cross-modal pedestrian recognition, accurate recognition results cannot be obtained.

Contents of the invention

One or more embodiments of this specification provide a cross-modal pedestrian re-identification method, device and medium of infrared and visible light, which are used to solve the following technical problem: accurate recognition results cannot be obtained during cross-modal pedestrian recognition.

One or more embodiments of this specification adopt the following technical solutions:

One or more embodiments of this specification provide a cross-modal pedestrian re-identification method of infrared and visible light. Any one or more of images and infrared images; the pedestrian image is input to a pre-trained cross-modal feature generation model to generate pedestrian cross-modal features corresponding to the pedestrian image, wherein the cross-modal The modality feature generation model is trained using a bimodal image data set, and the loss function includes a center maximum average difference loss function, a heterogeneous center loss function within a class, and a cross-modal triple loss function; the pedestrian cross-modal feature is combined with Comparing the features in the pre-built feature library to obtain the specified features in the feature library that meet the requirements, so as to determine the identification result of the pedestrian to be identified according to the specified features that meet the requirements, wherein the feature library includes Feature information and identity information of multiple pedestrians.

Further, the pedestrian image is input to a pre-trained cross-modal feature generation model, and before the pedestrian cross-modal feature corresponding to the pedestrian image is generated, the method further includes constructing the dual-modal image data set, The data set includes multiple sets of dual-modal images of multiple sample individuals, wherein each set of dual-modal images includes multiple infrared sample images and multiple visible light sample images; using the dual-modal images in the data set , train the pre-built feature generation model, and determine the model parameters, so as to obtain the cross-modal feature generation model under the model parameters.

Further, using the bimodal images in the data set to train a pre-built feature generation model specifically includes: inputting the bimodal images in the data set into the feature generation model, Shorten the distance between the feature centers in the dual-modal image in the Hilbert space, wherein the feature center is the feature center of pedestrian features belonging to different modes in the dual-modal image; by Transformer The feature generation network separates the sample individuals in the bimodal image to train the feature generation model to generate cross-modal features of the sample individuals.

Further, shortening the distance between feature centers in the dual-mode image in the Hilbert space specifically includes: acquiring the visible light sample image, the infrared sample image, and the visible light sample feature extraction function in the dual-modal image and an infrared sample feature extraction function; based on the visible light sample image, the infrared sample image, the visible light sample feature extraction function and the infrared sample feature extraction function, define the center maximum average difference loss function; through the center The maximum average difference loss function maps the feature centers in different modalities to the Hilbert space, shortening the center distance between the feature centers of the two modalities in the bimodal image.

Further, through the center maximum average difference loss function, the feature centers in different modalities are mapped to the Hilbert space, and the center between the feature centers of the two modalities in the bimodal image is shortened After the distance, the method further includes: obtaining the cross-modal class mean center position of the visible light sample image and the cross-modal class mean center position in the infrared sample image respectively; based on the cross-modal class mean center position of the visible light sample image The class mean center position and the cross-modal class mean center position in the infrared sample image define a first center covariance loss, wherein the center covariance loss includes the visible light features in the visible light sample image and the The visible light center covariance loss of the cross-modal class mean center position of the visible light sample image, and the infrared center covariance loss between the infrared feature in the infrared sample image and the cross-modal class mean center position in the infrared sample image; Defining the overall cross-modal class mean center position in the dual-modal image, based on the cross-modal class mean center position, the cross-modal class mean center position of the visible light sample image, and the cross-modal class mean center position in the infrared sample image Modal class mean center position, calculating the second center covariance loss of the cross-modal class mean center position and the overall cross-modal class mean center position under the two modes; according to the first center covariance loss and The second center covariance loss determines the intra-class heterogeneous center loss function; through the intra-class heterogeneous center loss function, controls the relationship between the sample individual in each modality and the two modes in the bimodal image The distance between the feature centers of the states.

Further, through the Transformer feature generation network, the sample individuals in the dual-modal image are separately distinguished, so as to train the feature generation model and generate cross-modal features of the sample individuals, specifically including: obtaining the In the dual-mode image data set, the visible light sample image and the infrared sample image corresponding to the same sample individual; extract the visible light sample features of the visible light sample image, and the infrared feature of the infrared sample image; combine the visible light sample features and the The infrared feature is input to the Transformer feature generation network to generate the output visible light sample feature and the output infrared feature; through the pre-generated cross-modal triplet loss function, the visible light sample feature and the infrared feature are processed constraints to generate the cross-modal features of the sample individual.

Further, the center maximum average difference loss function is:

其中，L_CMMD为所述中心最大平均差异损失函数，

为可见光样本图像，f_v为可见光样本特征提取函数；

为近红外样本图像，f_I为近红外样本特征提取函数，ψ为再生核希尔伯特空间映射函数，P为选取的样本个体的身份种类数量，K为每个身份对应的样本个体的双模态图像数量。

Among them, L _CMMD is the center maximum mean difference loss function,

is the visible light sample image, f _v is the feature extraction function of the visible light sample;

is the near-infrared sample image, f _I is the feature extraction function of the near-infrared sample, ψ is the Hilbert space mapping function of the regeneration kernel, P is the number of identity types of the selected sample individual, K is the pair of sample individuals corresponding to each identity The number of modal images.

Further, the visible light sample feature and the infrared feature are constrained by the pre-generated cross-modal triplet loss function, and before generating the cross-modal feature of the sample individual, the method further includes: Obtain a designated infrared sample image and a designated visible light sample image of a designated sample individual in the dual-mode image data set, and obtain a preset infrared sample image and a preset visible light sample image of a preset sample individual with a different identity from the designated sample individual image; forming an infrared triplet loss function based on the specified infrared sample image and the preset infrared sample image; forming a visible light triplet loss function based on the specified visible light sample image and the preset visible light sample image; In the batch processing group, according to the infrared triplet loss function and the visible light triplet loss function, define a cross-modal triplet loss function for sample individuals with the same identity.

One or more embodiments of this specification provide a cross-modal pedestrian re-identification device for infrared and visible light, including:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to:

Through the dual-mode acquisition device, the pedestrian image corresponding to the pedestrian to be identified is collected, wherein the pedestrian image includes any one or more of visible light images and infrared images; the pedestrian image is input to the pre-trained cross-modal A feature generation model for generating pedestrian cross-modal features corresponding to the pedestrian image, wherein the cross-modal feature generation model uses a dual-modal image data set for training, and the loss function includes a center maximum average difference loss function, an intra-class difference Centroid loss function and cross-modal triplet loss function; compare the cross-modal features of the pedestrian with the features in the pre-built feature library, and obtain the specified features in the feature library that meet the requirements, so as to meet the requirements according to the The specified feature of the pedestrian to be identified is determined to determine the identification result of the pedestrian to be identified, wherein the feature database includes feature information and identity information of a plurality of pedestrians.

A non-volatile computer storage medium provided by one or more embodiments of this specification stores computer-executable instructions, and the computer-executable instructions are set to:

Through the dual-mode acquisition device, the pedestrian image corresponding to the pedestrian to be identified is collected, wherein the pedestrian image includes any one or more of visible light images and infrared images; the pedestrian image is input to the pre-trained cross-modal A feature generation model for generating pedestrian cross-modal features corresponding to the pedestrian image, wherein the cross-modal feature generation model uses a dual-modal image data set for training, and the loss function includes a center maximum average difference loss function, an intra-class difference Centroid loss function and cross-modal triplet loss function; compare the cross-modal features of the pedestrian with the features in the pre-built feature library, and obtain the specified features in the feature library that meet the requirements, so as to meet the requirements according to the The specified feature of the pedestrian to be identified is determined to determine the identification result of the pedestrian to be identified, wherein the feature database includes feature information and identity information of a plurality of pedestrians.

The above-mentioned at least one technical solution adopted in the embodiment of this specification can achieve the following beneficial effects: through the above-mentioned technical solution, combined with the center maximum average difference loss function, the intra-class heterogeneous center loss function and the cross-modal triplet loss function, it can be obtained from the model The model is fully optimized in terms of both mode and identity, and the distance between the two modal distributions is fully shortened to ensure the accuracy of the generated cross-modal features. At the same time, the ability to identify pedestrians is enhanced, and the pedestrian accurate identification.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments described in this specification. Those skilled in the art can also obtain other drawings based on these drawings without any creative effort. In the attached picture:

FIG. 1 is a schematic flowchart of a cross-modal pedestrian re-identification method of infrared and visible light provided by an embodiment of this specification;

FIG. 2 is a schematic diagram of a method for shortening spatial distance and fusing cross-modal features provided by an embodiment of this specification;

FIG. 3 is a schematic structural diagram of a framework for generating cross-modal pedestrian features of infrared and visible light provided by an embodiment of this specification;

FIG. 4 is a schematic structural diagram of an infrared-visible light cross-modal pedestrian re-identification device provided by an embodiment of this specification.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described The embodiments are only some of the embodiments in this specification, not all of them. Based on the embodiments of this specification, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

The embodiment of this specification provides a cross-modal pedestrian re-identification method using infrared and visible light. It should be noted that the execution subject in the embodiment of this specification may be a server, or any device with data processing capabilities. Fig. 1 is a schematic flowchart of a cross-modal pedestrian re-identification method of infrared and visible light provided by the embodiment of this specification. As shown in Fig. 1, it mainly includes the following steps:

In step S101, a pedestrian image corresponding to a pedestrian to be identified is collected by a dual-mode collection device.

Wherein, the pedestrian image includes any one or more of visible light images and infrared images.

In one embodiment of this specification, the dual-mode acquisition device can be understood as a dual-mode camera, and the dual-mode camera includes two modes, one is a visible light mode and the other is an infrared mode. In order to enhance the accuracy of pedestrian re-identification and avoid the influence of visible light camera by dark environment, a dual-mode camera is used to collect pedestrian images of pedestrians to be recognized.

In step S102, the pedestrian image is input to the pre-trained cross-modal feature generation model, and the pedestrian cross-modal feature corresponding to the pedestrian image is generated.

Among them, the cross-modal feature generation model is trained with a bimodal image dataset, and the loss functions include the center maximum average difference loss function, the intra-class heterogeneity center loss function, and the cross-modal triplet loss function.

In one embodiment of the present specification, the pedestrian image corresponding to the pedestrian to be identified is input into the pre-trained cross-modal feature generation model to output the pedestrian cross-modal feature corresponding to the pedestrian image. It should be noted that for the cross-modal generation model in this embodiment, when performing feature extraction, the distance between the two modes is first shortened in the Hilbert space, and the distance between the two modes is realized by the center maximum mean loss function. For the clustering of the same identity in the same state, the individual cross-modal feature center distance loss is calculated through the intra-class heterogeneous center loss function, so as to bring the centers of the two modalities closer; then the Transformer feature generation network and the cross-modal triplet are used The group loss function distinguishes individual pedestrian images with different identities, draws cross-modal images with the same identity closer, and pushes away cross-modal images with different identities.

The pedestrian image is input to the pre-trained cross-modal feature generation model, and before the pedestrian cross-modal feature corresponding to the pedestrian image is generated, the method also includes: constructing the bimodal image data set, which includes multiple Multiple sets of dual-modal images of sample individuals, wherein each set of dual-modal images includes multiple infrared sample images and multiple visible light sample images; using the dual-modal images in the data set, the pre-built feature generation model is Training, to determine the model parameters, so as to obtain the cross-modal feature generation model under the model parameters.

In one embodiment of this specification, a large number of dual-modality image datasets are obtained, including multiple sets of dual-modality images of multiple sample individuals in the data set, and each set of dual-modality images includes multiple infrared sample images and multiple visible light samples. Sample images, the identity of each sample individual is different. The feature generation model is trained by using the bimodal images in the data set to determine the model parameters, and a cross-modal feature generation model under the required model parameters is obtained.

Using the bimodal image in the data set to train the pre-built feature generation model, specifically includes: inputting the bimodal image in the data set into the feature generation model, shortening the feature generation model in Hilbert space The distance between the feature centers in the dual-modal image, wherein the feature center is the feature center of the pedestrian features belonging to different modes in the dual-modal image; through the Transformer feature generation network, in the dual-modal image Separately distinguish the sample individuals to train the feature generation model to generate the cross-modal features of the sample individuals.

Shorten the distance between the feature centers in the dual-modal image in the Hilbert space, specifically including: obtaining the visible light sample image, the infrared sample image, the visible light sample feature extraction function and the infrared sample feature extraction function in the dual-modal image function; based on the visible light sample image, the infrared sample image, the visible light sample feature extraction function and the infrared sample feature extraction function, define the center maximum average difference loss function; through the center maximum average difference loss function, the different modes The feature center of is mapped to the Hilbert space, shortening the center distance between the feature centers of the two modalities in the bimodal image. The centered maximum mean difference loss function is:

为可见光样本图像，f_v为可见光样本特征提取函数；

为近红外样本图像，f_I为近红外样本特征提取函数，ψ为再生核希尔伯特空间映射函数。Among them, L _CMMD is the maximum mean difference loss function of the center,

is the visible light sample image, f _v is the feature extraction function of the visible light sample;

is the near-infrared sample image, f _I is the near-infrared sample feature extraction function, and ψ is the regeneration kernel Hilbert space mapping function.

In one embodiment of the present specification, each type of mold table is regarded as a whole, and the centers of the two modes are drawn closer. If the maximum mean discrepancy loss (Maximum Mean Discrepancy, MMD) is directly used to reduce the inter-domain difference, since each sample participates in shortening the distance between the two distributions, the MMD loss fluctuates violently during the training process, resulting in over-sensitivity to local samples , which is not conducive to stably narrowing the center distance of the two modes. Since the sample center contains the information of each sample to a certain extent, the sample can represent the information of the distribution, so the center maximum mean difference loss (CenterMaximum Mean Discrepancy, CMMD) is designed to measure two modes that are different but related to the distribution Center distance. Randomly select P sample individuals in the data set, corresponding to P types of identities; select K sample data for each identity to form a batch processing group Batch, in which two modal batches have the same identity category. That is to say, P is the number of identity categories of the selected sample individuals, and K is the number of bimodal images of the sample individuals corresponding to each identity.

CMMD mainly cares about the distance between feature centers, and its loss function is defined as:

L_CMMD为该中心最大平均差异损失函数，

为可见光样本图像，f_v为可见光样本特征提取函数；

为近红外样本图像，f_I为近红外样本特征提取函数，ψ为再生核希尔伯特空间映射函数，KN为高斯核函数，定义为：

σ为超参数。in,

L _CMMD is the maximum mean difference loss function of the center,

is the visible light sample image, and f _v is the feature extraction function of the visible light sample;

is the near-infrared sample image, f _I is the near-infrared sample feature extraction function, ψ is the regeneration kernel Hilbert space mapping function, KN is the Gaussian kernel function, defined as:

σ is a hyperparameter.

Through the center maximum average difference loss function, the feature centers in different modalities are mapped to the Hilbert space, and after shortening the center distance between the feature centers of the two modalities in the dual-modal image, the method also Including: obtaining the cross-modal class mean center position of the visible light sample image and the cross-modal class mean center position in the infrared sample image respectively; based on the cross-modal class mean center position of the visible light sample image and the infrared sample image The center position of the cross-modal class mean value of , defines the first center covariance loss, wherein the center covariance loss is the visible light including the visible light features in the visible light sample image and the cross-modal class mean center position of the visible light sample image Center covariance loss, and the infrared center covariance loss between the infrared feature in the infrared sample image and the cross-modal class mean center position in the infrared sample image; define the overall cross-modal class mean center position in the dual-modal image , based on the cross-modal class mean center position, the cross-modal class mean center position of the visible light sample image and the cross-modal class mean center position in the infrared sample image, calculate the cross-modal class mean under the two modalities The second center covariance loss between the center position and the center position of the overall cross-modal class mean; according to the first center covariance loss and the second center covariance loss, determine the heterogeneous center loss function within the class; through the class Inner heterogeneity center loss function, which controls the distance between sample individuals in each modality and the feature centers of the two modalities in this bimodal image.

The center distance between the two modality data is controlled by L _CMMD , and then the distance from each class instance to its class center in each modality is controlled.

First, the images under the two modalities in the bimodal image are taken as a whole, and the overall cross-modal class mean center position in the bimodal image is defined, and the overall cross-modal homogeneous mean center C _i is as follows:

为：

定义红外样本图像中的跨模态类均值中心位置

为：

And define the center position of the cross-modal class mean value of the visible light sample image

for:

Defines the location of the center of the cross-modal class means in the infrared sample image

for:

According to the cross-modal class mean center position of the visible light sample image and the cross-modal class mean center position in the infrared sample image, the first center covariance loss is defined, and the first center covariance loss L _ICHC-A is obtained as follows:

其中，中心协方差损失为可见光样本图像中的其他可见光特征与可见光样本图像的跨模态类均值中心位置的可见光中心协方差损失，以及红外样本图像中的红外特征与红外样本图像中的跨模态类均值中心位置的红外中心协方差损失的和。

Among them, the central covariance loss is the visible central covariance loss between other visible light features in the visible light sample image and the cross-modal class mean center position of the visible light sample image, and the infrared feature in the infrared sample image and the cross-modal class mean in the infrared sample image. The sum of the IR center covariance loss for the center position of the state class mean.

Calculate the cross-modal class mean center position and the overall The second central covariance loss across the modal class mean center position, the obtained second central covariance loss _LICHC-B is as follows:

The sum of the first center covariance loss and the second center covariance loss is used as the intra-class heterogeneous center loss function to obtain the intra-class heterogeneous center loss function L _ICHC as follows: L _ICHC ＝L _ICHC-A +L _ICHC-B .

In one embodiment of this specification, the center maximum average difference loss function and the intra-class heterogeneity center loss function are used as the loss function to shorten the distance between the two modal centers, and the final loss function is: L _Whole = L _CMMD + L _ICHC , where L _Whole is the final loss function.

In order to further control the instance feature similarity, a cross-entropy loss function L _CE is introduced,

是尺度为1×n的FC特征中的第j个概率值。模态距离损失函数L_W定义如下，L_W＝L_CE+λL_Whole其中，λ为超参数。Among them, x _i is the input sample, y _i is the corresponding label,

is the jth probability value in the FC feature with scale 1×n. The modal distance loss function L _W is defined as follows, L _W = L _CE + λL _Whole where λ is a hyperparameter.

Through the Transformer feature generation network, the sample individuals in the dual-modal image are separately distinguished, so as to train the feature generation model and generate the cross-modal features of the sample individual, specifically including: obtaining the dual-modal image data set , the visible light sample image and the infrared sample image corresponding to the same sample individual; extract the visible light sample feature of the visible light sample image, and the infrared feature of the infrared sample image; input the visible light sample feature and the infrared feature to the Transformer feature generation network , to generate the output visible light sample feature and the output infrared feature; through the pre-generated cross-modal triplet loss function, the visible light sample feature and the infrared feature are constrained to generate the cross-modal feature of the sample individual.

The visible light sample feature and the infrared feature are constrained by the pre-generated cross-modal triplet loss function, and before generating the cross-modal feature of the sample individual, the method further includes: in the dual-modal image data set Obtain the specified infrared sample image and the specified visible light sample image of the specified sample individual, and obtain the preset infrared sample image and the preset visible light sample image of the preset sample individual with a different identity from the specified sample individual; based on the specified infrared sample image and the Preset the infrared sample image to form an infrared triplet loss function; based on the specified visible light sample image and the preset visible light sample image, form a visible light triplet loss function; in the batch processing group, according to the infrared triplet loss function And the visible light triplet loss function defines the cross-modal triplet loss function of the sample individual with the same identity.

和

输入融合网络，通过

生成

通过

生成

生成的特征再用跨模态三元组损失函数进行约束。In one embodiment of this specification, in order to reduce the distance between cross-modal images with the same identity, a Transformer feature generation network is introduced for feature generation. The Transformer feature generation network mainly includes two parts: Encoder and Decoder. During training, the same identity and different modal features

and

Enter the fusion network, through

generate

pass

generate

The generated features are then constrained with a cross-modal triplet loss function.

则随机选取与其具有相同身份的红外样本

及具有不同身份的红外样本

形成可见光三元组损失函数L_V2I；同理，可得到红外三元组损失函数L_I2V，其中，可见光三元组损失函数L_V2I以及红外三元组损失函数L_I2V的表达式如下所示：The cross-modal triplet loss function CMT judges the distance between cross-modal images of different identities. For an image x _i in one modality, the CMT loss will pull the features of the image with the same identity as x _i closer and push them away Feature distances from x _i for images with different identities. Suppose the visible light sample is

Then randomly select the infrared sample with the same identity as

and infrared samples with different identities

The visible light triplet loss function L _V2I is formed; similarly, the infrared triplet loss function L _I2V can be obtained, where the expressions of the visible light triplet loss function L _V2I and the infrared triplet loss function L _I2V are as follows:

Among them, γ is a hyperparameter. In the batch processing group, according to the sum of the infrared triplet loss function and the visible light triplet loss function, define the cross-modal triplet loss function of the same identity sample individual, and the cross-modal triplet loss function of the same identity sample individual The L _CMT looks like this:

L _CMT =L _V2I +L _I2V .

In one embodiment of this specification, considering the cross-entropy loss between different identity instances, the identity loss L _I is defined as follows, L _I =L _CE +βL _CMT , where β is a hyperparameter.

Fig. 2 is a schematic diagram of a method for shortening spatial distance and fusing cross-modal features provided by the embodiment of this specification, and Fig. 3 is a schematic structural diagram of a generation framework for cross-modal pedestrian features of infrared and visible light provided by the embodiment of this specification. As shown in Fig. 2 and Fig. 3, when the overall modality is shortened, CMMD maps the modality center to the regenerating kernel Hilbert space, and then shortens the distance between the two modality centers, so that the two modality images The distribution is closer; in individual identity recognition, after Transform transformation and individual cross-modal fusion features, ICHC loss is based on the identity consistency of samples, and each identity sample is regarded as a class, so that all samples with the same identity, no matter Which mode it belongs to, it is close to its sample center. Combining these two losses, the network can be fully optimized from two aspects of modality and identity, and the distance between the two modality distributions can be fully shortened, while enhancing the ability of individual identification.

From the above discussion, it can be seen that there are two stages in the generation of cross-modal features. One stage is to shorten the distance between the feature centers in the bimodal image in the Hilbert space. Through the first stage, the shared features may exist The mode difference; the second stage is to generate shared embedded features through the Transformer feature generation network and the cross-modal triple loss function, and generate features through the second stage to enhance individual recognition ability.

In one embodiment of this specification, when performing model training, the CNN feature extraction network for visible light and infrared images is performed using the center maximum average difference loss function, the intra-class heterogeneity center loss function, the cross-entropy loss function, and a combination thereof. Constraints, constrained modal space distance loss, intra-mode covariance loss, inter-mode covariance loss, and intra-mode identity cross-entropy loss, to achieve the first stage of training. CNN here refers to Convolutional Neural Networks, convolutional neural network . In the second stage, the cross-modal triple loss function is applied to the Transformer feature generation network and all CNN feature extraction networks to realize the second stage of training, and the two stages of training are executed alternately.

In one embodiment of this specification, all loss functions of the first stage Stage1 and the second stage Stage2 can be combined to train network parameters, and Stage1 and Stage2 can also alternately execute the training process in their respective scopes.

Step S103, comparing the cross-modal features of the pedestrian with the features in the pre-built feature library to obtain the specified features in the feature library that meet the requirements, so as to determine the identification result of the pedestrian to be identified according to the specified features that meet the requirements.

Wherein, the feature database includes feature information and identity information of multiple pedestrians.

After obtaining the cross-modal features of the pedestrian, compare the cross-modal features of the pedestrian to be identified with the features in the feature library, and find the specified features in the feature library that meet the requirements. It should be noted that if the specified feature and the cross-modal feature of the pedestrian If the cosine distance between the state features is less than or equal to the preset threshold, it indicates that the specified feature meets the requirements. According to the identity of the pedestrian corresponding to the feature in the feature database, it is used as the identity information of the pedestrian to be identified.

Through the above technical solution, combined with the center maximum average difference loss function, the intra-class heterogeneity center loss function and the cross-modal triplet loss function, the model can be fully optimized from the two aspects of modality and identity, and the two aspects can be fully shortened. The distance between the modal distributions ensures the accuracy of the generated cross-modal features, and at the same time enhances the ability to identify pedestrians, allowing accurate identification of pedestrian identities.

The embodiment of this specification also provides a cross-modal pedestrian re-identification device of infrared and visible light. As shown in FIG. 4, the device includes: at least one processor; and a memory connected to the at least one processor; instructions executable by at least one processor, the instructions being executable by at least one processor to enable the at least one processor to:

The pedestrian image corresponding to the pedestrian to be identified is collected through a dual-mode acquisition device, wherein the pedestrian image includes any one or more of visible light images and infrared images; the pedestrian image is input to the pre-trained cross-modal feature generation model to generate pedestrian cross-modal features corresponding to the pedestrian image, where the cross-modal feature generation model is trained using a dual-modal image dataset, and the loss function includes the center maximum average difference loss function, the intra-class heterogeneity center loss function And the cross-modal triplet loss function; compare the cross-modal features of the pedestrian with the features in the pre-built feature library, and obtain the specified features in the feature library that meet the requirements, so as to determine the specified features according to the specified features that meet the requirements. The identification result of the pedestrian to be identified, wherein the feature database includes feature information and identity information of multiple pedestrians.

The embodiment of this specification also provides a non-volatile computer storage medium, which stores computer-executable instructions, and the computer-executable instructions are set to:

The pedestrian image corresponding to the pedestrian to be identified is collected through a dual-mode acquisition device, wherein the pedestrian image includes any one or more of visible light images and infrared images; the pedestrian image is input to the pre-trained cross-modal feature generation model to generate pedestrian cross-modal features corresponding to the pedestrian image, where the cross-modal feature generation model is trained using a dual-modal image dataset, and the loss function includes the center maximum average difference loss function, the intra-class heterogeneity center loss function And the cross-modal triplet loss function; compare the cross-modal features of the pedestrian with the features in the pre-built feature library, and obtain the specified features in the feature library that meet the requirements, so as to determine the specified features according to the specified features that meet the requirements. The identification result of the pedestrian to be identified, wherein the feature database includes feature information and identity information of multiple pedestrians.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiments.

The foregoing describes specific embodiments of this specification. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

The above description is only one or more embodiments of this specification, and is not intended to limit this specification. Various modifications and changes will occur to one or more embodiments of this description for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included within the scope of claims of this specification.

Claims (10)

1. A cross-mode pedestrian re-identification method of infrared visible light is characterized by comprising the following steps:

acquiring a pedestrian image corresponding to a pedestrian to be identified through a dual-mode acquisition device, wherein the pedestrian image comprises any one or more of a visible light image and an infrared image;

inputting the pedestrian image into a pre-trained cross-modal feature generation model, and generating a pedestrian cross-modal feature corresponding to the pedestrian image, wherein the cross-modal feature generation model is trained by using a bimodal image data set, and the loss function comprises a central maximum average difference loss function, an intra-class heterogeneous central loss function and a cross-modal triplet loss function;

comparing the pedestrian cross-modal characteristics with characteristics in a pre-constructed characteristic library to obtain specified characteristics meeting requirements in the characteristic library, and determining the identity recognition result of the pedestrian to be recognized according to the specified characteristics meeting the requirements, wherein the characteristic library comprises characteristic information and identity information of a plurality of pedestrians.

2. The method according to claim 1, wherein the pedestrian image is input to a pre-trained cross-modal feature generation model, and before the pedestrian cross-modal feature corresponding to the pedestrian image is generated, the method further comprises:

constructing the bimodal image dataset comprising a plurality of sets of bimodal images of a plurality of sample individuals, wherein each set of bimodal images comprises a plurality of infrared sample images and a plurality of visible light sample images;

and training a pre-constructed feature generation model by using the bimodal images in the data set, and determining model parameters to obtain a cross-modal feature generation model under the model parameters.

3. The method according to claim 2, wherein the training of the pre-constructed feature generation model using the bimodal images in the dataset comprises:

inputting the bimodal images in the data set into the feature generation model, and shortening the distance between feature centers in the bimodal images in a Hilbert space, wherein the feature centers are feature centers of pedestrian features belonging to different modalities in the bimodal images;

and through a Transformer feature generation network, sample individuals in the bimodal images are distinguished independently, so that the feature generation model is trained, and cross-modal features of the sample individuals are generated.

4. The method according to claim 3, wherein shortening the distance between feature centers in the bimodal image in the Hilbert space comprises:

acquiring a visible light sample image, an infrared sample image, a visible light sample feature extraction function and an infrared sample feature extraction function in the bimodal image;

defining the central maximum average difference loss function based on the visible light sample image, the infrared sample image, the visible light sample feature extraction function, and the infrared sample feature extraction function;

and mapping the characteristic centers under different modes to the Hilbert space through the maximum average difference loss function of the centers, and shortening the center distance between the characteristic centers of the two modes in the bimodal image.

5. The method according to claim 4, wherein feature centers of different modalities are mapped to the Hilbert space by the center maximum average difference loss function, and after a center distance between feature centers of two modalities in the bimodal image is shortened, the method further comprises:

respectively acquiring a trans-modal mean value center position of the visible light sample image and a trans-modal mean value center position in the infrared sample image;

defining a first center covariance loss based on the cross-modality mean-like center position of the visible light sample image and the cross-modality mean-like center position in the infrared sample image, wherein the center covariance loss is a visible light center covariance loss comprising visible light features in the visible light sample image and the cross-modality mean-like center position of the visible light sample image, and an infrared center covariance loss comprising infrared features in the infrared sample image and the cross-modality mean-like center position in the infrared sample image;

defining the overall cross-modal mean value center position in the dual-modal image, and calculating a second center covariance loss of the cross-modal mean value center position and the overall cross-modal mean value center position under two modalities based on the cross-modal mean value center position, the cross-modal mean value center position of the visible light sample image and the cross-modal mean value center position in the infrared sample image;

determining the intra-class heterogeneous center loss function according to the first center covariance loss and the second center covariance loss;

and controlling the distance between the sample individual in each mode and the characteristic centers of the two modes in the bimodal image through the intra-class heterogeneous center loss function.

6. The method according to claim 3, wherein the method for re-identifying the infrared-visible cross-modal pedestrian is characterized in that sample individuals in the bimodal image are individually distinguished through a transform feature generation network, so as to train the feature generation model to generate cross-modal features of the sample individuals, and specifically comprises:

acquiring a visible light sample image and an infrared sample image corresponding to the same sample individual in the bimodal image dataset;

extracting visible light sample characteristics of the visible light sample image and infrared characteristics of the infrared sample image;

inputting the visible light sample characteristics and the infrared characteristics into the transform characteristic generation network to generate output visible light sample characteristics and output infrared characteristics;

and constraining the visible light sample characteristics and the infrared characteristics through the pre-generated trans-modal triplet loss function to generate the trans-modal characteristics of the sample individual.

7. The method according to claim 5, wherein the central maximum average difference loss function is:

wherein L is _CMMD Is the central maximum average difference loss function,

for visible light sample images, f _v Extracting a function for the characteristics of the visible light sample;

as a near-infrared sample image, f _I The method comprises the steps of extracting features of near infrared samples, psi representing a regeneration nuclear Hilbert space mapping function, P representing the number of identity types of selected sample individuals, and K representing the number of bimodal images of the sample individuals corresponding to each identity.

8. The method of claim 6, wherein before the visible light sample features and the infrared features are constrained by the pre-generated trans-modal triplet loss function to generate the trans-modal features of the individual samples, the method further comprises:

acquiring an appointed infrared sample image and an appointed visible light sample image of an appointed sample individual in the dual-mode image dataset, and acquiring a preset infrared sample image and a preset visible light sample image of a preset sample individual with different identities from the appointed sample individual;

forming an infrared triple loss function based on the specified infrared sample image and the preset infrared sample image;

forming a visible light triplet loss function based on the specified visible light sample image and the preset visible light sample image;

and in a batch processing group, defining a cross-mode triplet loss function of the identity sample individual according to the infrared triplet loss function and the visible light triplet loss function.

9. An infrared visible cross-modal pedestrian re-identification device, the device comprising:

at least one processor; and the number of the first and second groups,

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to:

acquiring a pedestrian image corresponding to a pedestrian to be identified through a dual-mode acquisition device, wherein the pedestrian image comprises any one or more of a visible light image and an infrared image;

inputting the pedestrian image into a pre-trained cross-modal feature generation model, and generating a pedestrian cross-modal feature corresponding to the pedestrian image, wherein the cross-modal feature generation model is trained by using a bimodal image data set, and the loss function comprises a central maximum average difference loss function, an intra-class heterogeneous central loss function and a cross-modal triplet loss function;

comparing the pedestrian cross-modal characteristics with characteristics in a pre-constructed characteristic library to obtain specified characteristics meeting requirements in the characteristic library, and determining the identity recognition result of the pedestrian to be recognized according to the specified characteristics meeting the requirements, wherein the characteristic library comprises characteristic information and identity information of a plurality of pedestrians.

10. A non-transitory computer storage medium storing computer-executable instructions configured to:

acquiring a pedestrian image corresponding to a pedestrian to be identified through a dual-mode acquisition device, wherein the pedestrian image comprises any one or more of a visible light image and an infrared image;

inputting the pedestrian image into a pre-trained cross-modal feature generation model, and generating a pedestrian cross-modal feature corresponding to the pedestrian image, wherein the cross-modal feature generation model is trained by using a bimodal image data set, and the loss function comprises a central maximum average difference loss function, an intra-class heterogeneous central loss function and a cross-modal triplet loss function;

comparing the pedestrian cross-modal characteristics with characteristics in a pre-constructed characteristic library to obtain specified characteristics meeting requirements in the characteristic library, and determining the identity recognition result of the pedestrian to be recognized according to the specified characteristics meeting the requirements, wherein the characteristic library comprises characteristic information and identity information of a plurality of pedestrians.

Images