CN110148468B - Method and device for dynamic face image reconstruction - Google Patents

Abstract

The method and device for reconstructing a dynamic face image provided by the present invention are aimed at the characteristics that the dynamic face image mainly presents high-level visual feature information, and that the facial features with different attributes are processed by different high-level cognitive brain regions. , using three different attributes of advanced feature information and three different advanced cognitive brain regions to obtain the first type of neural response data, the second type of neural response data and the third type of advanced feature information corresponding to the three different attributes of the face. Neural-like response data, construct different high-level cognitive brain regions and dynamic face models from visual image space to brain perception space, as well as the multi-dimensional mapping relationship between the models, and obtain basic face images, face expression images and Face identity image, to realize the reconstruction of multi-dimensional facial features, obtain dynamic face images, and reconstruct some dynamic face images perceived by patients, so that we can have a deeper understanding and recognition of the mechanism of cognitive impairment of mental diseases. Know.

Description

Method and device for dynamic face image reconstruction

technical field

The present invention relates to image processing technology, and in particular, to a method and device for reconstructing a dynamic face image.

Background technique

Reproducing perceived visual objects from brain nerve signals is a frontier technology field that has attracted widespread attention. It refers to collecting functional magnetic resonance imaging (fMRI) signals of the human brain and using images Processing and machine learning algorithms restore the visual images seen by vision. As the most common and important visual perception object we encounter in understanding nature and social interaction, some people have cognitive abilities. And mental disorders such as prosopagnosia, autism, Alzheimer's disease, Parkinson's disease patients have defects in recognizing the high-level feature attributes of dynamic faces. Therefore, it is necessary to use face reconstruction technology to imagine the user's brain. face for image reconstruction.

In the prior art, principal component analysis (Principal Component Analysis, PCA for short) is used to establish a single linear mapping relationship between the characteristic face and the neural response signal, so as to realize the reconstruction of the face image.

However, the existing technology can only reconstruct static face pictures, which is difficult to meet the needs of face multi-dimensional information reconstruction in the field of image reconstruction.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method and device for reconstructing a dynamic face image, which realizes dynamic face reconstruction, reconstructs expression features and identity features in the reconstructed dynamic face image at the same time, enriches the reconstructed information, and improves human performance. Accuracy of face reconstruction.

A first aspect of the embodiments of the present invention provides a method for reconstructing a dynamic face image, including:

The first type of neural response data is extracted, and a basic face image is obtained according to the first type of neural response data and a preset face image reconstruction model.

The second type of neural response data is extracted, and a facial expression image is obtained according to the second type of neural response data and a preset facial expression reconstruction model.

The third type of neural response data is extracted, and a face identity image is obtained according to the third type of neural response data and a preset face identity reconstruction model.

A dynamic face image is acquired according to the basic face image, the face expression image and the face identity image.

Optionally, in a possible implementation manner of the first aspect, obtaining a basic face image according to the first type of neural response data and a preset face image reconstruction model includes:

According to the following formula 1 and the first type of neural response data, a basic face image is obtained.

是人脸图像重建模型中预设的动态人脸基础图像样本的平均图像，Y_test是所述第一类神经响应数据，

是人脸图像重建模型中预设的动态人脸基础图像样本引起的第一类神经响应数据样本的平均数据，s_test是中Y_test的投影坐标，t_test是X_{G_RECON}的投影坐标，W_train是人脸图像重建模型中s_test-t_test变换矩阵，U_train是人脸图像重建模型中Y_test的特征向量，V_train是人脸图像重建模型中预设的动态人脸基础图像样本的特征向量。where X _{G_RECON} is the basic face image,

is the average image of the dynamic face basic image samples preset in the face image reconstruction model, Y _test is the first type of neural response data,

is the average data of the first type of neural response data samples caused by the dynamic face basic image samples preset in the face image reconstruction model, s _test is the projection coordinates of Y _test in the middle, t _test is the projection coordinates of X _{G_RECON} , W _train is the s _test -t _test transformation matrix in the face image reconstruction model, U _train is the feature vector of Y _test in the face image reconstruction model, and V _train is the feature of the dynamic face basic image sample preset in the face image reconstruction model vector.

Optionally, in a possible implementation manner of the first aspect, before the first type of neural response data and the preset face image reconstruction model are acquired, the method further includes:

Obtain a dynamic face basic image training sample and a first type of neural response data training sample caused by the dynamic face basic image sample.

Taking the basic image sample of the dynamic face as the output, and taking the first type of neural response data sample as the input, the st transformation matrix, the eigenvector of the first type of neural response data sample and the dynamic face are paired by the following formula. The eigenvectors of the basic image training samples are used for parameter learning to obtain the s _test -t _test transformation matrix in the face image reconstruction model, the eigenvectors of the first type of neural response data samples in the face image reconstruction model, and the human face in the face image reconstruction model. the feature vector of the base image of the face,

是X的平均图像，Y是所述第一类神经响应数据样本，

是Y的平均数据，s是Y的投影坐标，t是X的投影坐标，W是所述s-t变换矩阵，U是Y的特征向量，V是X的特征向量。Wherein, X is the basic image sample of the dynamic face,

is the average image of X, Y is the first class of neural response data samples,

is the mean data of Y, s is the projected coordinate of Y, t is the projected coordinate of X, W is the st transformation matrix, U is the eigenvector of Y, and V is the eigenvector of X.

According to the s _test -t _test transformation matrix in the face image reconstruction model, the feature vector of the first type of neural response data samples in the face image reconstruction model, and the features of the face basic image in the face image reconstruction model vector to obtain the face image reconstruction model.

Optionally, in a possible implementation manner of the first aspect, obtaining a facial expression image according to the second type of neural response data and a preset facial expression reconstruction model includes:

The facial expression image is obtained according to the following formula 3 and the second type of neural response data.

是人脸表情重建模型中预设的动态人脸表情图像样本的平均图像，Y_{E_test}是第二类神经响应数据，

是人脸表情重建模型中预设的动态人脸表情图像样本引起的第二类神经响应数据样本的平均数据，s_{E_test}是Y_{E_test}的投影坐标，t_{E_test}是X_{E_RECON}的投影坐标，W_{E_train}是人脸表情重建模型中的s_{E_test}-t_{E_test}变换矩阵，U_{E_train}是人脸表情重建模型的Y_{E_test}的特征向量，V_{E_train}是人脸表情重建模型预设的动态人脸表情图像样本的特征向量。where X _{E_RECON} is the facial expression image,

is the average image of the preset dynamic facial expression image samples in the facial expression reconstruction model, Y _{E_test} is the second type of neural response data,

is the average data of the second type of neural response data samples caused by the preset dynamic facial expression image samples in the facial expression reconstruction model, s _{E_test} is the projected coordinate of Y _{E_test} , t _{E_test} is the projected coordinate of X _{E_RECON} , and W _{E_train} is The s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, U _{E_train} is the feature vector of Y _{E_test} of the facial expression reconstruction model, and V _{E_train} is the feature vector of the dynamic facial expression image samples preset by the facial expression reconstruction model .

Optionally, in a possible implementation manner of the first aspect, before the second type of neural response data and the preset facial expression reconstruction model are acquired, the method further includes:

Obtain a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample.

Taking the dynamic facial expression image sample as the output, and taking the second type of neural response data sample as the input, four pairs of s _E -t _E transformation matrices and the second type of neural response data sample feature vectors through the following formula Perform parameter learning with the eigenvectors of the dynamic facial expression image training samples to obtain the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the eigenvectors of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression The feature vector of the facial expression image in the reconstructed model,

是X_E的平均图像，Y_E是所述第二类神经响应数据样本，

是Y_E的平均数据，s_E是Y_E的投影坐标，t_E是X_E的投影坐标，W_E是所述s_E-t_E变换矩阵，U_E是Y_E的特征向量，V_E是X_E的特征向量，id是每一个面部身份个体的标签。Wherein, X _E is the dynamic facial expression image sample,

is the average image of X _E , Y _E is the second class of neural response data samples,

is the mean data of Y _E , s _E is the projected coordinate of Y _E , t _E is the projected coordinate of X _E , W _E is the s _E -t _E transformation matrix, U _E is the eigenvector of Y _E , and V _E is the The feature vector of X _E , id is the label of each facial identity individual.

According to the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the feature vector of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression image in the facial expression reconstruction model. Feature vector to obtain facial expression reconstruction model.

Optionally, in a possible implementation manner of the first aspect, obtaining a face identity image according to the third type of neural response data and a preset face identity reconstruction model includes:

According to the following formula 5 and the third type of neural response data, a face identity image is obtained.

人脸身份重建模型中预设的动态人脸身份图像样本的平均图像，Y_{I_test}所述第三类神经响应数据，

是人脸身份重建模型中预设的动态人脸身份图像样本引起的第三类神经响应数据样本的平均数据，s_{I_test}是Y_{I_test}的投影坐标，t_{I_test}是X_{I_RECON}的投影坐标，W_I-train是人脸身份重建模型中s_{I_test}-t_{I_test}变换矩阵，U_{I_train}是人脸身份重建模型Y_{I_test}的特征向量，V_{I_train}是人脸身份重建模型中预设的动态人脸身份图像样本的特征向量。where X _{I_RECON} is the face identity image,

The average image of the preset dynamic face identity image samples in the face identity reconstruction model, the third type of neural response data described in Y _{I_test} ,

is the average data of the third type of neural response data samples caused by the preset dynamic face identity image samples in the face identity reconstruction model, s _{I_test} is the projected coordinate of Y _{I_test} , t _{I_test} is the projected coordinate of X _{I_RECON} , W _{I- train} is the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, U _{I_train} is the feature vector of the face identity reconstruction model Y _{I_test} , and V _{I_train} is the feature of the dynamic face identity image sample preset in the face identity reconstruction model vector.

Optionally, in a possible implementation manner of the first aspect, before the third type of neural response data and the preset face identity reconstruction model are acquired, the method further includes:

Obtain the training samples of the dynamic face identity image and the third type of neural response data training samples caused by the dynamic face identity image training samples.

Taking the dynamic face identity image samples as the output and the third type of neural response data samples as the input, six pairs of transformation matrices s _I -t _I and the third type of neural response data samples through the following formula Perform parameter learning with the eigenvectors of the dynamic face identity image training samples to obtain the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, the eigenvectors of the third type of neural response data samples in the face identity reconstruction model, and the face feature vector of training samples of identity images,

是X_I的平均图像，Y_I是所述第三类神经响应数据样本，

是Y_I的平均数据，s_I是Y_I的投影坐标，t_I是X_I的投影坐标，W_I是所述s_I-t_I变换矩阵，U_I是Y_I的特征向量，V_I是人脸身份重建模型X_I的特征向量，ex是每一种面部表情的标签。Wherein, X _I is the dynamic face identity image sample,

is the average image of X _I , Y _I is the third class of neural response data samples,

is the mean data of Y _I , s _I is the projected coordinate of Y _I , t _I is the projected coordinate of X _I , W _I is the s _I -t _I transformation matrix, U _I is the eigenvector of Y _I , and V _I is the The feature vector of the face identity reconstruction model X _I , ex is the label of each facial expression.

According to the transformation matrix of s _{I_test -} t _{I_test} in the face identity reconstruction model, the feature vector of the third type of neural response data samples in the face identity reconstruction model, the face identity image in the face identity reconstruction model , and obtain the face identity reconstruction model.

Optionally, in a possible implementation manner of the first aspect, the first type of neural response data is neural response data obtained from the brain region of the primary visual cortex of the user to be tested.

The second type of neural response data is the neural response data obtained from the posterior superior temporal sulcus and amygdala brain regions of the user to be tested.

The third type of neural response data is the neural response data obtained from the brain region of the fusiform gyrus face processing region and the anterior temporal lobe brain region of the user to be tested.

Optionally, in a possible implementation manner of the first aspect, the obtaining a dynamic face image according to the basic face image, the face expression image and the face identity image includes:

An average image of the basic face image, the face expression image and the face identity image is determined as the dynamic face image.

A second aspect of the embodiments of the present invention provides a dynamic face image reconstruction device, including:

The first acquisition module is configured to extract the first type of neural response data, and acquire a basic face image according to the first type of neural response data and a preset face image reconstruction model.

The second obtaining module is configured to extract the second type of neural response data, and obtain a facial expression image according to the second type of neural response data and a preset facial expression reconstruction model.

The third acquisition module is configured to extract the third type of neural response data, and acquire a face identity image according to the third type of neural response data and a preset face identity reconstruction model.

A dynamic face image acquisition module, configured to acquire a dynamic face image according to the basic face image, the face expression image and the face identity image.

Optionally, in a possible implementation manner of the second aspect, the first acquisition module is configured to acquire a basic face image according to the following formula 1 and the first type of neural response data.

是人脸图像重建模型中预设的动态人脸基础图像样本的平均图像，Y_test是所述第一类神经响应数据样本，

是人脸图像重建模型中预设的动态人脸基础图像样本引起的第一类神经响应数据样本的平均数据，s_test是Y_test的投影坐标，t_test是X_{G_RECON}的投影坐标，W_train是人脸图像重建模型中s_test-t_test变换矩阵，U_train是人脸图像重建模型中Y_test的特征向量，V_train是人脸图像重建模型中预设的动态人脸基础图像样本的特征向量。where X _{G_RECON} is the basic face image,

is the average image of the dynamic face basic image samples preset in the face image reconstruction model, Y _test is the first type of neural response data samples,

is the average data of the first type of neural response data samples caused by the dynamic face basic image samples preset in the face image reconstruction model, s _test is the projected coordinate of Y _test , t _test is the projected coordinate of X _{G_RECON} , and W _train is s _test -t _test transformation matrix in the face image reconstruction model, U _train is the feature vector of Y _test in the face image reconstruction model, V _train is the feature vector of the dynamic face basic image sample preset in the face image reconstruction model .

Optionally, in a possible implementation manner of the second aspect, the first obtaining module 401 further includes a training sample for obtaining the dynamic face basic image training sample and the first category caused by the dynamic face basic image sample. Neural response data training samples.

Taking the basic image sample of the dynamic face as the output, and taking the first type of neural response data sample as the input, the st transformation matrix, the eigenvector of the first type of neural response data sample and the dynamic face are paired by the following formula. The eigenvectors of the basic image training samples are used for parameter learning to obtain the s _test -t _test transformation matrix in the face image reconstruction model, the eigenvectors of the first type of neural response data samples in the face image reconstruction model, and the human face in the face image reconstruction model. the feature vector of the base image of the face,

是X的平均图像，Y是所述第一类神经响应数据样本，

是Y的平均数据，s是Y的投影坐标，t是X的投影坐标，W是所述s-t变换矩阵，U是Y的特征向量，V是X的特征向量。Wherein, X is the basic image sample of the dynamic face,

is the average image of X, Y is the first class of neural response data samples,

is the mean data of Y, s is the projected coordinate of Y, t is the projected coordinate of X, W is the st transformation matrix, U is the eigenvector of Y, and V is the eigenvector of X.

According to the s _test -t _test transformation matrix in the face image reconstruction model, the feature vector of the first type of neural response data samples in the face image reconstruction model, and the features of the face basic image in the face image reconstruction model vector to obtain the face image reconstruction model.

Optionally, in a possible implementation manner of the second aspect, the second acquisition module is configured to acquire a facial expression image according to the following formula 3 and the second type of neural response data;

是人脸表情重建模型中预设的动态人脸表情图像样本的平均图像，Y_{E_test}是第二类神经响应数据样本，

是人脸表情重建模型中预设的动态人脸表情图像样本引起的第二类神经响应数据样本的平均数据，s_{E_test}是Y_{E_test}的投影坐标，t_{E_test}是X_{E_RECON}的投影坐标，W_{E_train}是人脸表情重建模型中的s_{E_test}-t_{E_test}变换矩阵，U_{E_train}是人脸表情重建模型的Y_{E_test}的特征向量，V_{E_train}是人脸表情重建模型预设的动态人脸表情图像样本的特征向量。where X _{E_RECON} is the facial expression image,

is the average image of the preset dynamic facial expression image samples in the facial expression reconstruction model, Y _{E_test} is the second type of neural response data samples,

is the average data of the second type of neural response data samples caused by the preset dynamic facial expression image samples in the facial expression reconstruction model, s _{E_test} is the projected coordinate of Y _{E_test} , t _{E_test} is the projected coordinate of X _{E_RECON} , and W _{E_train} is The s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, U _{E_train} is the feature vector of Y _{E_test} of the facial expression reconstruction model, and V _{E_train} is the feature vector of the dynamic facial expression image samples preset by the facial expression reconstruction model .

Optionally, in a possible implementation manner of the second aspect, the second acquisition module further includes a training sample for acquiring a dynamic facial expression image and a second type of nerve caused by the dynamic facial expression image sample. Response data training samples.

Taking the dynamic facial expression image sample as the output, and taking the second type of neural response data sample as the input, four pairs of s _E -t _E transformation matrices and the second type of neural response data sample feature vectors through the following formula Perform parameter learning with the eigenvectors of the dynamic facial expression image training samples to obtain the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the eigenvectors of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression The feature vector of the facial expression image in the reconstructed model,

是X_E的平均图像，Y_E是所述第二类神经响应数据样本，

是Y_E的平均数据，s_E是Y_E的投影坐标，t_E是X_E的投影坐标，W_E是所述s_E-t_E变换矩阵，U_E是Y_E的特征向量，V_E是X_E的特征向量，id是每一个面部身份个体的标签。Wherein, X _E is the dynamic facial expression image sample,

is the average image of X _E , Y _E is the second class of neural response data samples,

is the mean data of Y _E , s _E is the projected coordinate of Y _E , t _E is the projected coordinate of X _E , W _E is the s _E -t _E transformation matrix, U _E is the eigenvector of Y _E , and V _E is the The feature vector of X _E , id is the label of each facial identity individual.

According to the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the feature vector of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression image in the facial expression reconstruction model. Feature vector to obtain facial expression reconstruction model.

Optionally, the third obtaining module is configured to obtain a face identity image according to the following formula 5 and the third type of neural response data.

人脸身份重建模型中预设的动态人脸身份图像样本的平均图像，Y_{I_test}所述第三类神经响应数据样本，

是人脸身份重建模型中预设的动态人脸身份图像样本引起的第三类神经响应数据样本的平均数据，s_{I_test}是Y_{I_test}的投影坐标，t_{I_test}是X_{I_RECON}的投影坐标，W_I-train是人脸身份重建模型中s_{I_test}-t_{I_test}变换矩阵，U_{I_train}是人脸身份重建模型Y_{I_test}的特征向量，V_{I_train}是人脸身份重建模型中预设的动态人脸身份图像样本的特征向量。where X _{I_RECON} is the face identity image,

The average image of the preset dynamic face identity image samples in the face identity reconstruction model, the third type of neural response data samples described in Y _{I_test} ,

is the average data of the third type of neural response data samples caused by the preset dynamic face identity image samples in the face identity reconstruction model, s _{I_test} is the projected coordinate of Y _{I_test} , t _{I_test} is the projected coordinate of X _{I_RECON} , W _{I- train} is the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, U _{I_train} is the feature vector of the face identity reconstruction model Y _{I_test} , and V _{I_train} is the feature of the dynamic face identity image sample preset in the face identity reconstruction model vector.

Optionally, in a possible implementation manner of the second aspect, the third acquisition module further includes a third type of neural response for acquiring a dynamic face identity image training sample and a third type of neural response caused by the dynamic face identity image training sample. Data training samples; with the dynamic face identity image samples as the output and the third type of neural response data samples as the input, six pairs of transformation matrices s _I - t _I and the third type of neural response data through the following formulas The feature vector of the sample and the feature vector of the dynamic face identity image training sample are used for parameter learning to obtain the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, and the characteristics of the third type of neural response data samples in the face identity reconstruction model. vector and feature vector of face identity image training samples,

是X_I的平均图像，Y_I是所述第三类神经响应数据样本，

是Y_I的平均数据，s_I是Y_I的投影坐标，t_I是X_I的投影坐标，W_I是所述s_I-t_I变换矩阵，U_I是Y_I的特征向量，V_I是人脸身份重建模型X_I的特征向量，ex是每一种面部表情的标签。Wherein, X _I is the dynamic face identity image sample,

is the average image of X _I , Y _I is the third class of neural response data samples,

is the mean data of Y _I , s _I is the projected coordinate of Y _I , t _I is the projected coordinate of X _I , W _I is the s _I -t _I transformation matrix, U _I is the eigenvector of Y _I , and V _I is the The feature vector of the face identity reconstruction model X _I , ex is the label of each facial expression.

According to the transformation matrix of s _{I_test -} t _{I_test} in the face identity reconstruction model, the feature vector of the third type of neural response data samples in the face identity reconstruction model, the face identity image in the face identity reconstruction model , and obtain the face identity reconstruction model.

Optionally, in a possible implementation manner of the second aspect, the first type of neural response data is neural response data obtained from the brain region of the primary visual cortex of the user to be tested.

The second type of neural response data is the neural response data obtained from the posterior superior temporal sulcus and amygdala brain regions of the user to be tested.

The third type of neural response data is the neural response data obtained from the brain region of the fusiform gyrus face processing region and the anterior temporal lobe brain region of the user to be tested.

Optionally, in a possible implementation manner of the second aspect, the dynamic face image acquisition module is configured to obtain an average image of the basic face image, the face expression image and the face identity image. , which is determined as the dynamic face image.

The method for reconstructing a dynamic face image provided by the present invention is aimed at the cognition that the dynamic face image mainly presents high-level visual feature information, and the facial features with different attributes are processed by different high-level cognitive brain regions. Features, this scheme uses three different attributes of advanced feature information, and uses three different advanced cognitive brain regions to obtain the first type of neural response data and the second type of neural response data corresponding to the three different attributes of the advanced feature information of the face. and the third type of neural response data, construct different high-level cognitive brain regions and dynamic face models from visual image space to brain perception space, as well as the multi-dimensional mapping relationship between the models, and obtain basic face images, face Expression images and face identity images can be used to reconstruct multi-dimensional facial features, and dynamic face images can be obtained to reconstruct some dynamic face images perceived by patients, which enables us to have a deeper understanding of the cognitive impairment mechanism of mental illness. understanding and cognition.

Description of drawings

1 is a schematic flowchart of a method for reconstructing a dynamic face image provided by the present invention;

2 is a schematic diagram of signal transmission of a method for reconstructing a dynamic face image provided by the present invention;

3 is a schematic structural diagram of an apparatus for reconstructing a dynamic face image provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of a hardware structure of a dynamic face image reconstruction apparatus provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms "first", "second", "third", "fourth", etc. in the description and claims of the present invention and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order. or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein.

It should be understood that, in various embodiments of the present invention, the size of the sequence numbers of each process does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be used in the embodiments of the present invention. Implementation constitutes any limitation.

It should be understood that in the present invention, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to Those steps or elements that are expressly listed may instead include other steps or elements that are not expressly listed or are inherent to the process, method, product or apparatus.

First, the terms involved in the present invention are explained:

Functional Magnetic Resonance Imaging (fMRI): refers to an emerging neuroimaging method, the principle of which is to use magnetic resonance imaging to measure the hemodynamic changes caused by neuronal activity.

The specific application scenario of the present invention can be applied to reconstruct the patients with cognitive and mental disorders such as prosopagnosia, autism, Alzheimer's disease, and Parkinson's disease. The dynamic face images perceived by patients can enable us to have a deeper understanding and cognition of the cognitive impairment mechanism of mental illness. The current face image reconstruction uses Principal Component Analysis (PCA) to To achieve face image reconstruction, the existing technology does not consider the establishment of a mapping relationship for different attributes of high-level feature information, and can only reconstruct static face images, which is difficult to meet the needs of face multi-dimensional information reconstruction in the field of image reconstruction.

The present invention provides a method for reconstructing a dynamic face image, aiming at solving the above technical problems in the prior art, realizing dynamic face reconstruction, and simultaneously reconstructing expression features and identity features in the reconstructed dynamic face image, thereby enriching the The reconstructed information improves the accuracy of face reconstruction.

The technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems will be described in detail below with specific examples. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of a method for reconstructing a dynamic face image provided by the present invention. The execution body of the method shown in FIG. 1 may be software and/or hardware devices. The method shown in FIG. 1 includes steps S101 to S104, and the details are as follows:

S101. Extract the first type of neural response data, and obtain a basic face image according to the first type of neural response data and a preset face image reconstruction model.

Specifically, the first type of neural response data is the neural response data obtained from the brain area of the primary visual cortex of the user to be tested. Different attributes of the human face are processed by different brain areas of the brain. The cortical brain area can perceive the pixel-level low-level visual features of the face. The first type of neural response data adopts fMRI technology to collect the functional magnetic resonance signals of the primary visual cortex of the user's brain to obtain the first type of neural response data. In response to the data, the preset face image reconstruction model can obtain the corresponding basic face image by responding to the first type of neural response data.

S102. Extract the second type of neural response data, and obtain a facial expression image according to the second type of neural response data and a preset facial expression reconstruction model.

Specifically, the second type of neural response data is the neural response data obtained from the posterior superior temporal sulcus and amygdala brain regions of the user to be tested, and the facial expression features of the human face are determined by the posterior superior temporal sulcus and amygdala brain regions. The other brain regions are responsible for processing, and the second type of neural response data uses functional magnetic resonance imaging technology to collect the functional magnetic resonance signals of the posterior superior temporal sulcus and amygdala brain regions of the user to be tested to obtain the second type of neural response data. The preset The corresponding facial expression image can be obtained by responding to the second type of neural response data by the facial expression reconstruction model.

S103: Extract the third type of neural response data, and obtain a face identity image according to the third type of neural response data and a preset face identity reconstruction model.

Specifically, the third type of neural response data is the neural response data obtained from the brain region of the self-fusiform gyrus face processing region and the anterior temporal lobe brain region of the user to be tested, and the facial identity features of the face are determined by the self-fusiform gyrus Brain areas such as the face processing area and the anterior temporal lobe are responsible for processing. The third type of neural response data uses functional magnetic resonance imaging technology to collect the brain area or the anterior side of the face processing area of the fusiform gyrus of the user to be tested. The functional magnetic resonance signals of the temporal lobe brain area are used to obtain the third type of neural response data, and the preset face identity reconstruction model can obtain the corresponding face identity image in response to the third type of neural response data.

S104. Obtain a dynamic face image according to the basic face image, the face expression image, and the face identity image.

Specifically, the average image of the basic face image, the face expression image and the face identity image is determined as the dynamic face image.

In this embodiment, the above steps S101 to S103 are not limited by the described sequence of actions, and steps S101 to S103 may be performed in other sequences or simultaneously.

The method for reconstructing a dynamic face image provided by the above embodiment adopts three different attribute high-level feature information and three different high-level cognitive brain regions to obtain the first type of high-level feature information corresponding to three different attributes of a face. Neural response data, the second type of neural response data, and the third type of neural response data, while building different advanced cognitive brain regions and dynamic face models from visual image space to brain perception space, as well as the multi-dimensional mapping relationship between the models , obtain the basic face image, face expression image and face identity image to realize the reconstruction of multi-dimensional facial features, and obtain dynamic face images, which can reconstruct some dynamic face images perceived by the tested users.

On the basis of the above embodiment, the specific implementation of step S101 (acquiring a basic face image according to the first type of neural response data and a preset face image reconstruction model) may be:

Referring to FIG. 2, FIG. 2 is a schematic diagram of signal transmission of a method for reconstructing a dynamic face image provided by the present invention. The expression of dynamic face in brain perception space includes basic dynamic face perception space, face expression perception space and human face Identity perception space; the expression of dynamic face in image space includes basic image pixel space, facial image expression space and facial image identity space.

According to the following formula 1 (that is, the face image reconstruction model) and the first type of neural response data, the basic face image in step S101 is obtained,

是人脸图像重建模型中预设的动态人脸基础图像样本的平均图像，Y_test是所述第一类神经响应数据样本，

是人脸图像重建模型中预设的动态人脸基础图像样本引起的第一类神经响应数据样本的平均数据，s_test是中Y_test的投影坐标，t_test是X_{G_RECON}的投影坐标，W_train是人脸图像重建模型中s_test-t_test变换矩阵，U_train是人脸图像重建模型中Y_test的特征向量，V_train是人脸图像重建模型中预设的动态人脸基础图像样本的特征向量。where X _{G_RECON} is the basic face image,

is the average image of the dynamic face basic image samples preset in the face image reconstruction model, Y _test is the first type of neural response data samples,

is the average data of the first type of neural response data samples caused by the dynamic face basic image samples preset in the face image reconstruction model, s _test is the projection coordinates of Y _test in the middle, t _test is the projection coordinates of X _{G_RECON} , W _train is the s _test -t _test transformation matrix in the face image reconstruction model, U _train is the feature vector of Y _test in the face image reconstruction model, and V _train is the feature of the dynamic face basic image sample preset in the face image reconstruction model vector.

On the basis of the above embodiment, before acquiring the basic face image according to the first type of neural response data and the preset face image reconstruction model, a process of learning each parameter in the face image reconstruction model may also be included ,details as follows:

S201, using the dynamic face basic image sample as the output quantity and the first type neural response data sample as the input quantity, through the above formula two pairs of st transformation matrix, the feature vector of the first type neural response data sample and dynamic The eigenvectors of the basic face image training samples are used for parameter learning, and the s _test -t _test transformation matrix in the face image reconstruction model, the eigenvectors of the first type of neural response data samples in the face image reconstruction model, and the face image reconstruction model are obtained. The feature vector of the base image of the face.

Among them, the dynamic face is in the basic image pixel space, assuming that X _j is a dynamic face visual image, where j=1,2,...,N,N is the number of dynamic face images, and each dynamic face The face is represented in the form of a single-dimensional vector, and the dynamic face basic image sample X is represented as follows: X=[X ₁ X ₂ ... X _j ... X _N ].

Perform PCA singular value decomposition on X to use these samples to generate a "basic image pixel space", and the projected coordinates of each dynamic face image sample in the basic image pixel space are:

是X的平均图像,V是X的特征向量，按照它对应的特征值的大小由高往低排序，V可以更具体表示V＝[V₁,V₂,...,V_N],V是

的所有(线性无关)特征向量特征向量，每一列为一组特征向量。in

is the average image of X, V is the eigenvector of X, sorted from high to low according to the size of its corresponding eigenvalues, V can be more specifically expressed as V=[V ₁ ,V ₂ ,...,V _N ],V Yes

All (linearly independent) eigenvectors of eigenvectors, each column is a set of eigenvectors.

Under the dynamic face basic image sample X, each dynamic image (not limited to the image in the sample) can be linearly represented by its projected coordinates in this space. This PCA-based decomposition process is reversible, so any A visual image can be reconstructed according to its projected coordinates in the feature space, which can be expressed as:

Among them, the dynamic face is in the basic dynamic face perception space, assuming that Y _j is the neural response distribution of a dynamic face image in a group of brain areas of the primary visual cortex of the brain, where j=1,2,..., N, if Y _j is represented in the form of a one-dimensional vector, then the neural response of the dynamic face image sample set in the primary visual cortex of the brain, that is, the first type of neural response data sample Y, is expressed as Y=[Y ₁ Y ₂ ... Y _j ... Y _N ], using PCA to perform singular value decomposition of Y, the projection coordinates of the neural response of each dynamic face image in this neural response space can be expressed as:

是Y的平均数据，U是Y的特征向量，按照对应的特征值从大到小，U具体表示为U＝[U₁,U₂,...,U_N]。

is the average data of Y, U is the eigenvector of Y, according to the corresponding eigenvalues from large to small, U is specifically expressed as U=[U ₁ , U ₂ ,...,U _N ].

Among them, the multi-dimensional mapping relationship is to express the projection coordinate t of the dynamic face image sample under X as the linear transformation of its projection coordinate s under Y, namely

t=sW Equation 2.4

where W is the s-t transformation matrix. When t and s are known and are full-rank matrices, a solution to the transformation matrix W is:

W=(s ^T s+I) ^-1 s ^T t Equation 2.5

To sum up, formula 2 is formed by formula 2.1, formula 2.3, formula 2.4 and formula 2.5,

是X的平均图像，Y是所述第一类神经响应数据样本，

是Y的平均数据，s是Y的投影坐标，t是X的投影坐标，W是所述s-t变换矩阵，U是Y的特征向量，V是X的特征向量。Wherein, X is the basic image sample of the dynamic face,

is the average image of X, Y is the first class of neural response data samples,

is the mean data of Y, s is the projected coordinate of Y, t is the projected coordinate of X, W is the st transformation matrix, U is the eigenvector of Y, and V is the eigenvector of X.

S202, according to the s _test -t _test transformation matrix in the face image reconstruction model, the feature vector of the first type of neural response data samples in the face image reconstruction model, the face basic image in the face image reconstruction model The feature vector of , obtains the face image reconstruction model as formula 1.

On the basis of the above embodiment, the specific implementation of step S102 (extracting the second type of neural response data, and obtaining a facial expression image according to the second type of neural response data and a preset facial expression reconstruction model) can be Yes:

Obtain the facial expression image according to the following formula 3 (that is, the facial expression reconstruction model) and the second type of neural response data;

是人脸表情重建模型中预设的动态人脸表情图像样本的平均图像，Y_{E_test}是第二类神经响应数据样本，

是人脸表情重建模型中预设的动态人脸表情图像样本引起的第二类神经响应数据样本的平均数据，s_{E_test}是Y_{E_test}的投影坐标，t_{E_test}是X_{E_RECON}的投影坐标，W_{E_train}是人脸表情重建模型中的s_{E_test}-t_{E_test}变换矩阵，U_{E_train}是人脸表情重建模型的Y_{E_test}的特征向量，V_{E_train}是人脸表情重建模型预设的动态人脸表情图像样本的特征向量。where X _{E_RECON} is the facial expression image,

is the average image of the preset dynamic facial expression image samples in the facial expression reconstruction model, Y _{E_test} is the second type of neural response data samples,

is the average data of the second type of neural response data samples caused by the preset dynamic facial expression image samples in the facial expression reconstruction model, s _{E_test} is the projected coordinate of Y _{E_test} , t _{E_test} is the projected coordinate of X _{E_RECON} , and W _{E_train} is The s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, U _{E_train} is the feature vector of Y _{E_test} of the facial expression reconstruction model, and V _{E_train} is the feature vector of the dynamic facial expression image samples preset by the facial expression reconstruction model .

On the basis of the above embodiment, according to the second type of neural response data and the preset facial expression reconstruction model, before acquiring the facial expression image, the process of learning each parameter in the facial expression reconstruction model may also be included, details as follows:

S301, using the dynamic facial expression image sample as the output quantity and the second type of neural response data sample as the input quantity, through the above formula four pairs of _sE - _tE transformation matrix, the second type of neural response data sample The eigenvectors and the eigenvectors of the dynamic facial expression image training samples are used for parameter learning to obtain the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the eigenvectors of the second type of neural response data samples in the facial expression reconstruction model, and the human The feature vector of facial expression images in the facial expression reconstruction model.

Obtain the training samples of dynamic facial expression images and the training samples of the second type of neural response data caused by the dynamic facial expression image samples; wherein, the dynamic faces are in the facial image expression space, and the N samples in the image sample set are re-labeled image, and reintegrate the dynamic face sample set, which is expressed in the form of face expression image training sample X _E as follows:

Each column of X _E is composed of P single-dimensional vectors of different facial identities of the same facial expression, representing a facial expression, and each row of X _E has Q values, representing the same facial identity in an image part Q kinds of expression changes on the position.

Perform PCA-based singular value decomposition on X _E , the projected coordinates of each facial expression under X _E can be expressed as:

是X_E的平均图像，V_E是X_E的特征向量，特征值按从大到小的顺序排列表示为

is the average image of X _E , V _E is the eigenvector of X _E , and the eigenvalues are arranged in descending order and expressed as

Under X _E , each facial expression (not limited to the types of expressions in the sample) can be represented by its projected coordinates in this space. Since the decomposition process of PCA is reversible, any type of facial expression can be expressed according to its The projected coordinates in the expression feature space are reconstructed:

Among them, the dynamic face is in the facial expression perception space, assuming that Yi _,e is the neural response distribution of a dynamic face image in the posterior superior temporal sulcus and amygdala brain area, and Y _i,e is calculated according to each column. Rearrange the neural responses of one class of expressions to obtain the second class of neural response data samples Y _E , perform singular value decomposition on Y _E , and the projection coordinates of the neural responses of each facial expression under this Y _E can be expressed as:

是Y_E的平均数据，U_E是特征向量，按对应特征值由大到小具体表示为

in,

is the average data of Y _E , U _E is the eigenvector, which is specifically expressed as the corresponding eigenvalue from large to small

The projected coordinate t _E of the dynamic face image sample under X _E is expressed as the linear transformation of its projected coordinate s _E under Y _E , and t _E and s _E are defined as:

Here id is the label of each face identity individual, and the established mapping relationship is as follows:

t _E(id) = s _E(id) W _E(id) Equation 4.4

A parsing of W _E(id) can be expressed as:

W _E(id) = (s ^T _E(id) s _E(id) +I) ^-1 s ^T _E(id) t _E(id) Equation 4.5

To sum up, formula 4 is formed by formula 4.1, formula 4.3, formula 4.4 and formula 4.5,

是X_E的平均图像，Y_E是所述第二类神经响应数据样本，

是Y_E的平均数据，s_E是Y_E的投影坐标，t_E是X_E的投影坐标，W_E是所述s_E-t_E变换矩阵，U_E是Y_E的特征向量，V_E是X_E的特征向量。Wherein, X _E is the dynamic facial expression image sample,

is the average image of X _E , Y _E is the second class of neural response data samples,

is the mean data of Y _E , s _E is the projected coordinate of Y _E , t _E is the projected coordinate of X _E , W _E is the s _E -t _E transformation matrix, U _E is the eigenvector of Y _E , and V _E is the Eigenvectors of X _E.

S302, according to the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the feature vector of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression in the facial expression reconstruction model. The feature vector of the image to obtain the facial expression reconstruction model.

On the basis of the above embodiment, the specific implementation of step S103 (extracting the third type of neural response data, and obtaining a face identity image according to the third type of neural response data and the preset face identity reconstruction model) can be Yes:

Obtain a face identity image according to the following formula 5 (that is, the face identity reconstruction model) and the third type of neural response data;

人脸身份重建模型中预设的动态人脸身份图像样本的平均图像，Y_{I_test}所述第三类神经响应数据样本，

是人脸身份重建模型中预设的动态人脸身份图像样本引起的第三类神经响应数据样本的平均数据，s_{I_test}是Y_{I_test}的投影坐标，t_{I_test}是X_{I_RECON}的投影坐标，W_I-train是人脸身份重建模型中s_{I_test}-t_{I_test}变换矩阵，U_{I_train}是人脸身份重建模型Y_{I_test}的特征向量，V_{I_train}是人脸身份重建模型中预设的动态人脸身份图像样本的特征向量。where X _{I_RECON} is the face identity image,

The average image of the preset dynamic face identity image samples in the face identity reconstruction model, the third type of neural response data samples described in Y _{I_test} ,

is the average data of the third type of neural response data samples caused by the preset dynamic face identity image samples in the face identity reconstruction model, s _{I_test} is the projected coordinate of Y _{I_test} , t _{I_test} is the projected coordinate of X _{I_RECON} , W _{I- train} is the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, U _{I_train} is the feature vector of the face identity reconstruction model Y _{I_test} , and V _{I_train} is the feature of the dynamic face identity image sample preset in the face identity reconstruction model vector.

On the basis of the above embodiment, according to the third type of neural response data and the preset face identity reconstruction model, before acquiring the face identity image, the process of learning each parameter in the face image reconstruction model may also be included, details as follows:

S401, using the dynamic face identity image sample as the output quantity and the third type of neural response data sample as the input quantity, through the above formula six pairs of transformation matrices _sI - _tI , the third type of neural response data sample The eigenvectors and the eigenvectors of the dynamic face identity image training samples are used for parameter learning to obtain the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, the eigenvector sum of the third type of neural response data samples in the face identity reconstruction model Feature vector for training samples of face identity images.

Obtain the training samples of the dynamic face identity image and the third type of neural response data training samples caused by the dynamic face identity image training samples.

Among them, the dynamic face in the face image identity space, the labeled N sample images are integrated again, and represented by the dynamic face identity image sample X _I , each column of X _I is composed of Q different faces of the same facial identity The expression single-dimensional vector is spliced together, representing a facial identity individual, and each row of X _I has P values, representing the P individual identity changes of the same facial expression at the local position of the image.

Perform PCA-based singular value decomposition on X _I , and the projected coordinates of each face identity individual in this new identity feature space can be expressed as:

是X_I的平均图像，V_I是X_I的特征向量，特征值按从大到小的顺序排列可以表示为

is the average image of X _I , V _I is the eigenvector of X _I , and the eigenvalues are arranged in descending order and can be expressed as

Under X _I , each facial identity individual (not limited to the identity individuals in the sample) can be represented by its projected coordinates in this space. Since the decomposition process of PCA is reversible, any identity individual can be represented according to its identity The projected coordinates in the feature space are reconstructed:

Among them, the dynamic face is in the face identity perception space, assuming that Y _{i, e} are the neural response distribution of a dynamic face image in the face processing area of the fusiform gyrus and the anterior temporal lobe brain area. _i,e are rearranged according to the neural response data of a face identity individual in each column to obtain Y _I , and Y _I is subjected to PCA singular value decomposition. The projected coordinates of the neural responses of each facial expression under Y _I can be expressed as:

是Y_I的平均数据，U_I是Y_I的特征向量，按对应特征值由大到小具体表示为

here

is the average data of Y _I , and U _I is the eigenvector of Y _I , which is specifically expressed as the corresponding eigenvalue from large to small.

The projected coordinate _tI of a dynamic face image sample in the face image identity space is expressed as a linear transformation of its projected coordinate _sI in the neural response space. Redefine t _I and s _I as:

Here ex is the label for each facial expression. The mapping relationship is as follows:

t _I(ex) = s _I(ex) W _I(ex) Equation 6.4

The analytical solution of W _I(ex) can be expressed as:

W _I(ex) = (s ^T _I(ex) s _I(ex) +I) ^-1 s ^T _I(ex) t _I(ex) Equation 6.5

To sum up, formula 6 is formed by formula 6.1, formula 6.3, formula 6.4 and formula 6.5,

是X_I的平均图像，Y_I是所述第三类神经响应数据样本，

是Y_I的平均数据，s_I是Y_I的投影坐标，t_I是X_I的投影坐标，W_I是所述s_I-t_I变换矩阵，U_I是Y_I的特征向量，V_I是人脸身份重建模型X_I的特征向量。Wherein, X _I is the dynamic face identity image sample,

is the average image of X _I , Y _I is the third class of neural response data samples,

is the mean data of Y _I , s _I is the projected coordinate of Y _I , t _I is the projected coordinate of X _I , W _I is the s _I -t _I transformation matrix, U _I is the eigenvector of Y _I , and V _I is the The feature vector of the face identity reconstruction model _XI .

S402, according to the transformation matrix of s _{I_test -} t _{I_test} in the face identity reconstruction model, the feature vector of the third type of neural response data samples in the face identity reconstruction model, the face in the face identity reconstruction model The feature vector of the identity image to obtain the face identity reconstruction model.

Referring to FIG. 3 , it is a schematic structural diagram of an apparatus for reconstructing a dynamic face image provided by an embodiment of the present invention. The apparatus 40 for reconstructing a dynamic face image as shown in FIG. 3 includes:

The first acquisition module 401 is configured to extract the first type of neural response data, and acquire a basic face image according to the first type of neural response data and a preset face image reconstruction model.

The second obtaining module 402 is configured to extract the second type of neural response data, and obtain a facial expression image according to the second type of neural response data and a preset facial expression reconstruction model.

The third obtaining module 403 is configured to extract the third type of neural response data, and obtain a face identity image according to the third type of neural response data and a preset face identity reconstruction model.

The dynamic face image obtaining module 404 is configured to obtain a dynamic face image according to the basic face image, the face expression image and the face identity image.

The apparatus for dynamic face image reconstruction in the embodiment shown in FIG. 3 can correspondingly be used to execute the steps in the method embodiment shown in FIG. 1 , and the implementation principles and technical effects thereof are similar, and will not be repeated here.

Optionally, the first obtaining module 401 is configured to obtain a basic face image according to the following formula 1 and the first type of neural response data;

是人脸图像重建模型中预设的动态人脸基础图像样本的平均图像，Y_test是所述第一类神经响应数据样本，

是人脸图像重建模型中预设的动态人脸基础图像样本引起的第一类神经响应数据样本的平均数据，s_test是中Y_test的投影坐标，t_test是X_{G_RECON}的投影坐标，W_train是人脸图像重建模型中s_test-t_test变换矩阵，U_train是人脸图像重建模型中Y_test的特征向量，V_train是人脸图像重建模型中预设的动态人脸基础图像样本的特征向量。where X _{G_RECON} is the basic face image,

is the average image of the dynamic face basic image samples preset in the face image reconstruction model, Y _test is the first type of neural response data samples,

is the average data of the first type of neural response data samples caused by the dynamic face basic image samples preset in the face image reconstruction model, s _test is the projection coordinates of Y _test in the middle, t _test is the projection coordinates of X _{G_RECON} , W _train is the s _test -t _test transformation matrix in the face image reconstruction model, U _train is the feature vector of Y _test in the face image reconstruction model, and V _train is the feature of the dynamic face basic image sample preset in the face image reconstruction model vector.

Optionally, the first obtaining module 401 further includes a training sample for obtaining a dynamic human face basic image training sample and a first type of neural response data training sample caused by the dynamic human face basic image sample.

Taking the basic image sample of the dynamic face as the output, and taking the first type of neural response data sample as the input, the st transformation matrix, the eigenvector of the first type of neural response data sample and the dynamic face are paired by the following formula. The eigenvectors of the basic image training samples are used for parameter learning to obtain the s _test -t _test transformation matrix in the face image reconstruction model, the eigenvectors of the first type of neural response data samples in the face image reconstruction model, and the human face in the face image reconstruction model. the feature vector of the base image of the face,

是X的平均图像，Y是所述第一类神经响应数据样本，

是Y的平均数据，s是Y的投影坐标，t是X的投影坐标，W是所述s-t变换矩阵，U是Y的特征向量，V是X的特征向量；根据所述人脸图像重建模型中s_test-t_test变换矩阵、所述人脸图像重建模型中第一类神经响应数据样本的特征向量、所述人脸图像重建模型中人脸基础图像的特征向量，获取人脸图像重建模型。Wherein, X is the basic image sample of the dynamic face,

is the average image of X, Y is the first class of neural response data samples,

is the average data of Y, s is the projection coordinate of Y, t is the projection coordinate of X, W is the st transformation matrix, U is the feature vector of Y, V is the feature vector of X; reconstruct the model according to the face image In the s _test -t _test transformation matrix, the feature vector of the first type of neural response data sample in the described face image reconstruction model, the feature vector of the face basic image in the described face image reconstruction model, obtain the face image reconstruction model .

Optionally, the second obtaining module 402 is configured to obtain a facial expression image according to the following formula 3 and the second type of neural response data;

是人脸表情重建模型中预设的动态人脸表情图像样本的平均图像，Y_{E_test}是第二类神经响应数据样本，

是人脸表情重建模型中预设的动态人脸表情图像样本引起的第二类神经响应数据样本的平均数据，s_{E_test}是Y_{E_test}的投影坐标，t_{E_test}是X_{E_RECON}的投影坐标，W_{E_train}是人脸表情重建模型中的s_{E_test}-t_{E_test}变换矩阵，U_{E_train}是人脸表情重建模型的Y_{E_test}的特征向量，V_{E_train}是人脸表情重建模型预设的动态人脸表情图像样本的特征向量。where X _{E_RECON} is the facial expression image,

is the average image of the preset dynamic facial expression image samples in the facial expression reconstruction model, Y _{E_test} is the second type of neural response data samples,

is the average data of the second type of neural response data samples caused by the preset dynamic facial expression image samples in the facial expression reconstruction model, s _{E_test} is the projected coordinate of Y _{E_test} , t _{E_test} is the projected coordinate of X _{E_RECON} , and W _{E_train} is The s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, U _{E_train} is the feature vector of Y _{E_test} of the facial expression reconstruction model, and V _{E_train} is the feature vector of the dynamic facial expression image samples preset by the facial expression reconstruction model .

Optionally, the second obtaining module 402 further includes a training sample for obtaining a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample.

Taking the dynamic facial expression image sample as the output, and taking the second type of neural response data sample as the input, four pairs of s _E -t _E transformation matrices and the second type of neural response data sample feature vectors through the following formula Perform parameter learning with the eigenvectors of the dynamic facial expression image training samples to obtain the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the eigenvectors of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression The feature vector of the facial expression image in the reconstructed model,

是X_E的平均图像，Y_E是所述第二类神经响应数据样本，

是Y_E的平均数据，s_E是Y_E的投影坐标，t_E是X_E的投影坐标，W_E是所述s_E-t_E变换矩阵，U_E是Y_E的特征向量，V_E是X_E的特征向量；根据所述人脸表情重建模型中的s_{E_test}-t_{E_test}变换矩阵、所述人脸表情重建模型中第二类神经响应数据样本的特征向量、所述人脸表情重建模型中人脸表情图像的特征向量，获取人脸表情重建模型。Wherein, X _E is the dynamic facial expression image sample,

is the average image of X _E , Y _E is the second class of neural response data samples,

is the mean data of Y _E , s _E is the projected coordinate of Y _E , t _E is the projected coordinate of X _E , W _E is the s _E -t _E transformation matrix, U _E is the eigenvector of Y _E , and V _E is the The feature vector of X _E ; according to the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the feature vector of the second type of neural response data samples in the facial expression reconstruction model, the facial expression reconstruction model The feature vector of the facial expression image in the middle is obtained, and the facial expression reconstruction model is obtained.

Optionally, the third obtaining module 403 is configured to obtain a face identity image according to the following formula 5 and the third type of neural response data;

人脸身份重建模型中预设的动态人脸身份图像样本的平均图像，Y_{I_test}所述第三类神经响应数据样本，

是人脸身份重建模型中预设的动态人脸身份图像样本引起的第三类神经响应数据样本的平均数据，s_{I_test}是Y_{I_test}的投影坐标，t_{I_test}是X_{I_RECON}的投影坐标，W_I-train是人脸身份重建模型中s_{I_test}-t_{I_test}变换矩阵，U_{I_train}是人脸身份重建模型Y_{I_test}的特征向量，V_{I_train}是人脸身份重建模型中预设的动态人脸身份图像样本的特征向量。where X _{I_RECON} is the face identity image,

The average image of the preset dynamic face identity image samples in the face identity reconstruction model, the third type of neural response data samples described in Y _{I_test} ,

is the average data of the third type of neural response data samples caused by the preset dynamic face identity image samples in the face identity reconstruction model, s _{I_test} is the projected coordinate of Y _{I_test} , t _{I_test} is the projected coordinate of X _{I_RECON} , W _{I- train} is the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, U _{I_train} is the feature vector of the face identity reconstruction model Y _{I_test} , and V _{I_train} is the feature of the dynamic face identity image sample preset in the face identity reconstruction model vector.

Optionally, the third obtaining module 403 further includes a training sample for obtaining a dynamic face identity image training sample and a third type of neural response data training sample caused by the dynamic face identity image training sample. Taking the dynamic face identity image samples as the output and the third type of neural response data samples as the input, six pairs of transformation matrices s _I -t _I and the third type of neural response data samples through the following formula Perform parameter learning with the eigenvectors of the dynamic face identity image training samples to obtain the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, the eigenvectors of the third type of neural response data samples in the face identity reconstruction model, and the face feature vector of training samples of identity images,

是X_I的平均图像，Y_I是所述第三类神经响应数据样本，

是Y_I的平均数据，s_I是Y_I的投影坐标，t_I是X_I的投影坐标，W_I是所述s_I-t_I变换矩阵，U_I是Y_I的特征向量，V_I是人脸身份重建模型X_I的特征向量；Wherein, X _I is the dynamic face identity image sample,

is the average image of X _I , Y _I is the third class of neural response data samples,

is the mean data of Y _I , s _I is the projected coordinate of Y _I , t _I is the projected coordinate of X _I , W _I is the s _I -t _I transformation matrix, U _I is the eigenvector of Y _I , and V _I is the The feature vector of the face identity reconstruction model _XI ;

According to the transformation matrix of s _{I_test -} t _{I_test} in the face identity reconstruction model, the feature vector of the third type of neural response data samples in the face identity reconstruction model, the face identity image in the face identity reconstruction model , and obtain the face identity reconstruction model.

Optionally, the first type of neural response data is neural response data obtained from the brain region of the primary visual cortex of the user's brain to be tested.

The second type of neural response data is the neural response data obtained from the posterior superior temporal sulcus and amygdala brain regions of the user to be tested.

The third type of neural response data is the neural response data obtained from the brain region of the fusiform gyrus face processing region and the anterior temporal lobe brain region of the user to be tested.

Optionally, the dynamic face image obtaining module 404 is configured to determine the average image of the basic face image, the face expression image and the face identity image as the dynamic face image.

4 is a schematic diagram of a hardware structure of a device provided by an embodiment of the present invention, the device 50 includes: a processor 51, a memory 52, and a computer program; wherein

The memory 52 is used for storing the computer program, and the memory can also be a flash memory (flash). The computer program is, for example, an application program, a functional module, and the like that implement the above-mentioned method.

The processor 51 is configured to execute the computer program stored in the memory, so as to realize each step executed by the terminal in the above method. For details, refer to the relevant descriptions in the foregoing method embodiments.

Optionally, the memory 52 may be independent or integrated with the processor 51 .

When the memory 52 is a device independent of the processor 51, the device may further include:

The bus 53 is used to connect the memory 52 and the processor 51 .

The present invention also provides a readable storage medium, where a computer program is stored in the readable storage medium, and when the computer program is executed by a processor, is used to implement the methods provided by the above-mentioned various embodiments.

The readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media can be any available media that can be accessed by a general purpose or special purpose computer. For example, a readable storage medium is coupled to the processor such that the processor can read information from, and write information to, the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium may be located in application specific integrated circuits (Application Specific Integrated Circuits, ASIC for short). Alternatively, the ASIC may be located in the user equipment. Of course, the processor and the readable storage medium may also exist in the communication device as discrete components. The readable storage medium may be read only memory (ROM), random access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like.

The present invention also provides a program product including execution instructions stored in a readable storage medium. At least one processor of the device can read the execution instruction from the readable storage medium, and the execution of the execution instruction by the at least one processor causes the device to implement the dynamic face image reconstruction method provided by the above-mentioned various embodiments.

In the embodiment of the above-mentioned device, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, referred to as: CPU), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, referred to as: DSP) ), application specific integrated circuit (English: Application Specific Integrated Circuit, referred to as: ASIC) and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the present invention can be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (8)

1. a method for dynamic face image reconstruction, is characterized in that, comprises: Extract the first type of neural response data, and obtain a basic face image according to the first type of neural response data and a preset face image reconstruction model; the first type of neural response data is obtained from the brain of the user to be tested. Neural response data obtained from visual cortex brain regions; Extract the second type of neural response data, and obtain facial expression images according to the second type of neural response data and a preset facial expression reconstruction model; the second type of neural response data is obtained from the back side of the user to be tested. Neural response data obtained from superior temporal sulcus and amygdala brain regions; Extracting the third type of neural response data, and obtaining a face identity image according to the third type of neural response data and a preset face identity reconstruction model; the third type of neural response data is the self-shuttle from the user to be tested. Neural response data obtained from the face processing area of the sigmoid gyrus and the anterior temporal lobe area; obtaining a dynamic face image according to the basic face image, the face expression image and the face identity image; The obtaining a dynamic face image according to the basic face image, the face expression image and the face identity image, including: An average image of the basic face image, the face expression image and the face identity image is determined as the dynamic face image. 2. The method according to claim 1, wherein, obtaining a basic face image according to the first type of neural response data and a preset face image reconstruction model, comprising: Obtain a basic face image according to the following formula 1 and the first type of neural response data;

where X _{G_RECON} is the basic face image,

is the average image of the dynamic face basic image samples preset in the face image reconstruction model, Y _test is the first type of neural response data,

is the average data of the first type of neural response data samples caused by the dynamic face basic image samples preset in the face image reconstruction model, s _test is the projection coordinates of Y _test in the middle, t _test is the projection coordinates of X _{G_RECON} , W _train is the s _test -t _test transformation matrix in the face image reconstruction model, U _train is the feature vector of Y _test in the face image reconstruction model, and V _train is the feature of the dynamic face basic image sample preset in the face image reconstruction model vector. 3. The method according to claim 2, wherein, before acquiring the basic face image of the first type of neural response data and the preset face image reconstruction model, further comprising: Obtaining a dynamic face basic image training sample and a first type of neural response data training sample caused by the dynamic face basic image sample; Taking the basic image sample of the dynamic face as the output, and taking the first type of neural response data sample as the input, the st transformation matrix, the eigenvector of the first type of neural response data sample and the dynamic face are paired by the following formula. The eigenvectors of the basic image training samples are used for parameter learning to obtain the s _test -t _test transformation matrix in the face image reconstruction model, the eigenvectors of the first type of neural response data samples in the face image reconstruction model, and the human face in the face image reconstruction model. the feature vector of the base image of the face,

Wherein, X is the basic image sample of the dynamic face,

is the average image of X, Y is the first class of neural response data samples,

is the average data of Y, s is the projected coordinate of Y, t is the projected coordinate of X, W is the st transformation matrix, U is the eigenvector of Y, and V is the eigenvector of X; According to the s _test -t _test transformation matrix in the face image reconstruction model, the feature vector of the first type of neural response data samples in the face image reconstruction model, and the features of the face basic image in the face image reconstruction model vector to obtain the face image reconstruction model. 4. The method according to claim 1, wherein, obtaining a facial expression image according to the second type of neural response data and a preset facial expression reconstruction model, comprising: Obtain the facial expression image according to the following formula 3 and the second type of neural response data;

where X _{E_RECON} is the facial expression image,

is the average image of the preset dynamic facial expression image samples in the facial expression reconstruction model, Y _{E_test} is the second type of neural response data,

is the average data of the second type of neural response data samples caused by the preset dynamic facial expression image samples in the facial expression reconstruction model, s _{E_test} is the projected coordinate of Y _{E_test} , t _{E_test} is the projected coordinate of X _{E_RECON} , and W _{E_train} is The s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, U _{E_train} is the feature vector of Y _{E_test} of the facial expression reconstruction model, and V _{E_train} is the feature vector of the dynamic facial expression image samples preset by the facial expression reconstruction model . 5. The method according to claim 4, characterized in that, before acquiring the facial expression image by the second type of neural response data and the preset facial expression reconstruction model, further comprising: Obtaining a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample; Taking the dynamic facial expression image sample as the output, and taking the second type of neural response data sample as the input, four pairs of s _E -t _E transformation matrices and the second type of neural response data sample feature vectors through the following formula Perform parameter learning with the eigenvectors of the dynamic facial expression image training samples to obtain the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the eigenvectors of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression The feature vector of the facial expression image in the reconstructed model,

Wherein, X _E is the dynamic facial expression image sample,

is the average image of X _E , Y _E is the second class of neural response data samples,

is the mean data of Y _E , s _E is the projected coordinate of Y _E , t _E is the projected coordinate of X _E , W _E is the s _E -t _E transformation matrix, U _E is the eigenvector of Y _E , and V _E is the The feature vector of X _E ; id is the label of each facial identity individual; According to the s _{E_test} -t _{E_test} transformation matrix in the facial expression reconstruction model, the feature vector of the second type of neural response data samples in the facial expression reconstruction model, and the facial expression image in the facial expression reconstruction model. Feature vector to obtain facial expression reconstruction model. 6. The method according to claim 1, wherein, obtaining a face identity image according to the third type of neural response data and a preset face identity reconstruction model, comprising: Obtain a face identity image according to the following formula 5 and the third type of neural response data;

where X _{I_RECON} is the face identity image,

The average image of the preset dynamic face identity image samples in the face identity reconstruction model, the third type of neural response data described in Y _{I_test} ,

is the average data of the third type of neural response data samples caused by the preset dynamic face identity image samples in the face identity reconstruction model, s _{I_test} is the projected coordinate of Y _{I_test} , t _{I_test} is the projected coordinate of X _{I_RECON} , W _{I- train} is the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, U _{I_train} is the feature vector of the face identity reconstruction model Y _{I_test} , and V _{I_train} is the feature of the dynamic face identity image sample preset in the face identity reconstruction model vector. 7. The method according to claim 6, characterized in that, before acquiring a face identity image in the third type of neural response data and a preset face identity reconstruction model, the method further comprises: Obtaining training samples of dynamic face identity images and training samples of the third type of neural response data caused by the training samples of dynamic face identity images; Taking the dynamic face identity image samples as the output and the third type of neural response data samples as the input, six pairs of transformation matrices s _I -t _I and the third type of neural response data samples through the following formula Perform parameter learning with the eigenvectors of the dynamic face identity image training samples to obtain the s _{I_test} -t _{I_test} transformation matrix in the face identity reconstruction model, the eigenvectors of the third type of neural response data samples in the face identity reconstruction model, and the face feature vector of training samples of identity images,

Wherein, X _I is the dynamic face identity image sample,

is the average image of X _I , Y _I is the third class of neural response data samples,

is the mean data of Y _I , s _I is the projected coordinate of Y _I , t _I is the projected coordinate of X _I , W _I is the s _I -t _I transformation matrix, U _I is the eigenvector of Y _I , and V _I is the The feature vector of the face identity reconstruction model X _I ; ex is the label of each facial expression; According to the transformation matrix of s _{I_test -} t _{I_test} in the face identity reconstruction model, the feature vector of the third type of neural response data samples in the face identity reconstruction model, the face identity image in the face identity reconstruction model to obtain the face identity reconstruction model. 8. A device for dynamic face image reconstruction, comprising: The first acquisition module is used to acquire the first type of neural response data, and obtain the basic face image according to the first type of neural response data and a preset face image reconstruction model; the first type of neural response data is: Neural response data obtained from the primary visual cortex of the user under test; The second acquisition module is configured to acquire the second type of neural response data, and obtain the facial expression image according to the second type of neural response data and the preset facial expression reconstruction model; the second type of neural response data is: Neural response data obtained from the posterior superior temporal sulcus and amygdala brain regions of the tested user; The third acquisition module is used to acquire the third type of neural response data, and obtain the face identity image according to the third type of neural response data and the preset face identity reconstruction model; the third type of neural response data is: Neural response data obtained from the fusiform face processing area and the anterior temporal lobe area of the user to be tested; a dynamic face image acquisition module, configured to acquire a dynamic face image according to the basic face image, the face expression image and the face identity image; The dynamic face image acquisition module is specifically configured to determine the average image of the basic face image, the face expression image and the face identity image as the dynamic face image.

Images