CN111611860A - Micro-expression detection method and detection system - Google Patents

Abstract

A method for detecting the occurrence of micro-expressions, 1) recording normal EEG data and normal facial video data of a subject before stimulation induces micro-expressions; 2) stimulating micro-expressions, recording EEG data and facial video data; 3) marking time Stamping data, matching EEG time data and facial video time data; 4) Obtaining the starting time T _sm of the EEG time data reaction, the start and end frames and top frames of facial expression changes; 5) Determine whether micro-expressions occur. The invention correlates the EEG signal and the facial image information through the time stamp, respectively processes the EEG signal and the facial video data, and then combines the processing results to determine whether a micro-expression occurs, and the accuracy of the determination result is high.

Description

Micro-expression detection method and detection system

technical field

The invention relates to the technical field of EEG signal and facial video signal processing, in particular to a method for detecting whether a micro-expression occurs.

Background technique

As a short-lived facial expression that occurs within 1/25s to 1/2s, microexpressions are considered to be natural emotional expressions that are difficult to control when people are repressed or trying to hide their true emotions, so they are a very important clue in lie detection. , its importance and wide range of application scenarios have received more and more attention.

At present, the detection of micro-expressions is mainly through the method of expression recognition, which to a certain extent can separate a specific expression state from a given static image or dynamic video sequence, so as to determine the psychological emotion of the recognized object. It realizes the computer's understanding and recognition of facial expressions, but the detection accuracy of this method is not high, and it is prone to misjudgment. With the development of EEG technology, its high temporal dynamic resolution and high sensitivity to brain activity make it more direct to detect the occurrence of micro-expressions. The detection is limited by the single detection mode, and the accuracy is low.

The announcement number is CN109344816A, which discloses a method called "A method for real-time detection of facial movements based on EEG signals". By correlating the EEG signals with the corresponding facial action pictures in time, the EEG signals corresponding to each frame of pictures can be extracted. , and then establish a BP neural network by extracting EEG feature information to establish a facial action detection model, so as to achieve the purpose of identifying three types of facial actions through EEG signals. The shortcomings are: 1. The patent does not disclose the specific method of correlating EEG signals and facial action pictures through time, and the processing of EEG signals and facial picture recognition has a certain delay, which cannot solve the problem of data synchronization; 2. Establishing a BP neural network for facial motion detection requires a lot of computation and data processing, and cannot process a large amount of data; 3. The essence of this patent is to recognize facial pictures through EEG signals, not to identify and combine EEG signals and facial pictures separately. Judgment, its accuracy is insufficient.

SUMMARY OF THE INVENTION

One object of the present invention is to provide a method for detecting the occurrence of micro-expressions, which correlates EEG signals and facial image information through time stamps, processes the EEG signals and facial video data respectively, and then determines whether micro-expressions have occurred in combination with the processing results. , the judgment result has high accuracy.

The purpose of the present invention is achieved through such a technical solution, including the steps before the stimulation induces microexpressions and the steps after the stimulation induces microexpressions, recording the normal EEG data and normal facial video data of the subject before the stimulation induces microexpressions, The following steps are included after stimulus-induced microexpressions:

1) Stimulation induces micro-expressions, recording EEG data and facial video data;

2) Marking timestamp data for each section of EEG data and each frame of facial video data, matching and generating EEG time data and facial video time data;

3) Process the EEG data, EEG time data, facial video data and facial video time data, obtain the EEG micro-expression occurrence time T _sm judged by the EEG data and EEG time data, and obtain the facial video data and facial video The start and end frames and the top frame of facial expression changes judged by time data;

4) According to the EEG processed in step 3), determine the micro-expression occurrence time T _sm , the start and end frames and the top frame of the facial expression change, and determine whether the micro-expression occurs.

Further, record the normal state EEG data, normal state facial video data of the tested person, and the concrete method of recording EEG data and facial video data in step 1) is:

Normal EEG data and EEG data were acquired and recorded at 1024Hz sampling rate by recording from 128 electrodes using the Biosemi Active system; normal facial video data and facial video were acquired and recorded at 80 frames per second by the high-speed camera of the Biosemi Active system data.

Further, described in step 2) is that each section of EEG data and each frame of facial video data all mark the concrete method of time stamp data as:

Using the time synchronization module of the Biosemi Active system, the time stamp data in the time synchronization module is synchronously transmitted to the EEG acquisition module and the high-speed camera acquisition module of the Biosemi Active system, so that each piece of EEG data acquired by the EEG acquisition module and the high-speed camera Each frame of acquired facial video data contains synchronized timestamp data, that is, generated EEG time data and facial video time data.

Further, the concrete steps of processing EEG data and EEG time data described in step 3) are as follows:

X(k)表示长度为N的序列的傅里叶变换，k表示频率；3-1) Using the normal EEG data as the baseline data, calculate the normal PSD normal value of the normal Gamma band of the left temporal lobe channel D23, the right temporal lobe channel A09, and the prefrontal lobe channel B26, and the power spectral density represents The power carried by a unit frequency wave, the formula for calculating the normal value of PSD is:

X(k) represents the Fourier transform of a sequence of length N, and k represents the frequency;

3-2) For the EEG time data, set the sliding window duration as W=2s, take 2*(1/fs) as the sliding time, and fs as the EEG sampling frequency; calculate the left temporal lobe channel D23, The PSD sliding window duration value of the right temporal lobe channel A09 and the prefrontal lobe channel B26 in the Gamma frequency band, and compared with the normal PSD value of the corresponding channel; if the left temporal lobe channel D23, the right temporal lobe channel A09, the prefrontal lobe channel The PSD value of any channel in B26 is higher than the normal PSD value, assuming that the EEG data has changed, go to step 3-3); any channel in the left temporal lobe channel D23, the right temporal lobe channel A09, and the prefrontal lobe channel B26 If the PSD value is lower than the normal average PSD value, it will not be processed, and it is determined that the EEG data has not changed;

其中x(n)为信号幅度，N为数据长度，即2s的数据，取能量值均值为门限值G：G＝1/2E；将能量值E与门限值G相比，若E>G，且5ms以内给定采样点En均能持续达到门限值(E1...En>G)，则初步认为Tn是反应起点时刻Ts；同时，设置对比门限值PR，公式为

计算起点Ts之前n个采样点的对比值PRn，若|PN_n-PN_N-1|＝0则认为第一个采样点PRn对应的时刻为反应起点Ts；若反应起点Ts成功找到，则转向步骤3-4)；若反应起点Ts未成功找到，则返回步骤3-2)；3-3) Take the data within the sliding window duration W=2s assuming that the EEG data changes, first calculate the energy value E within the 2s, and the energy formula is

Where x(n) is the signal amplitude, N is the data length, that is, the data of 2s, and the average energy value is the threshold value G: G=1/2E; compare the energy value E with the threshold value G, if E> G, and the given sampling point En can continuously reach the threshold value within 5ms (E1...En>G), then Tn is initially considered to be the response starting time Ts; at the same time, the comparison threshold value PR is set, and the formula is

Calculate the comparison value PRn of n sampling points before the starting point Ts. If |PN _n -PN _N-1 |=0, the moment corresponding to the first sampling point PRn is considered to be the reaction starting point Ts; if the reaction starting point Ts is successfully found, turn to Step 3-4); If the reaction starting point Ts is not found successfully, then return to step 3-2);

则记录该时刻点，即微脑电表情发生时刻T_sm；若不满足条件，则返回步骤3-2)；3-4) At the time of the response starting point Ts, respectively take the brain region channel start time _Ts of the left temporal lobe channel D23, the right temporal lobe channel A09, and the prefrontal lobe channel B26, if the time T _sD23 -T _sB26 >0 or T _sD23 -T _sA09 >0, and must satisfy

Then record this moment, that is, the micro-EEG expression generation moment T _sm ; if the condition is not met, then return to step 3-2);

3-5) Obtain the EEG micro-expression occurrence time T _sm under the EEG data.

Further, the concrete steps of processing face video data and face video time data described in step 3) are:

3-6) Detecting the human face, and detecting the specific position of the human face from the original image of each frame;

3-7) Perform face alignment and face reference point positioning on the face in the face video collection; according to the input face image, use the CLM local constraint model to automatically locate the key facial feature CLM local constraint model;

3-8) Extraction of deformation-based expression features: Using CLM to mark facial feature points, after obtaining the coordinates of facial reference points, calculate the relative slope information between these facial reference points, and extract deformation-based expression features; Track the key points in each area, extract the corresponding displacement information, and extract the distance information between the specific feature points of the expression picture, subtract the distance from the calm picture, obtain the change information of the distance, and extract the movement-based expression features;

3-9) According to the facial feature data extraction result, obtain the start and end frame and the top frame; wherein, by comparing the distance between the feature points and the distance difference k of the calm picture, the threshold value R is set to exceed the first frame image of k>R It is determined as the starting frame; by comparing the images after the starting frame, one frame of image with the maximum value of k is determined as the number of top frames, and the first frame with k<R is regarded as the ending frame.

Further, step 3-6) detects the human face, and the specific steps of detecting the specific position of the human face from the original image of each frame are:

3-6-1) Use local binary pattern (LBP) to extract the response image;

3-6-2) The AdaBoost algorithm is used to process the response image to separate the face area; the LBP algorithm first scans each pixel of the original image line by line, and takes the gray value of each pixel as the threshold value. , binarize the 3*3 adjacent points around it and form an 8-bit binary number in sequence, and use the value of this binary number (0~255) as the point response.

Further, step 3-7) according to the input face image, using the CLM local constraint model to automatically locate the key features of the face The specific steps of the CLM local constraint model are as follows:

计算每个样本图像的形状和平均脸坐标的差值，可以得到一个零均值的形状变化矩阵X，对矩阵X进行PCA变换就可以得到人脸变化的主要成分，记特征值为λ_i，对应的特征向量为p_i；选择最大的k个特征值对应的特征向量组成正交矩阵P＝(p₁，p₂，…，p_k)；形状变化的权重向量b＝(b₁，b₂，…，b_k)^T,b的每个分量表示其在对应的特征向量方向的大小：

对任意的人脸检测图像，其样本形状向量可以表示为：

3-7-1) Modeling the shape of the face model: for M pictures, each picture has N feature points, the coordinates of each feature point are ( _xi , _yi ), and N on an image The vector composed of the coordinates of the feature points is represented by x=[x ₁ y ₁ x ₂ y ₂ ...x _N y _n ] ^T , and the average face coordinates of all images can be obtained:

Calculate the difference between the shape of each sample image and the average face coordinate, and a shape change matrix X with zero mean can be obtained. The main component of face change can be obtained by performing PCA transformation on the matrix X, and the eigenvalue is λ _i , corresponding to The eigenvector of is p _i ; the eigenvectors corresponding to the largest k eigenvalues are selected to form an orthogonal matrix P=(p ₁ , p ₂ , ..., p _k ); the weight vector of shape change b=(b ₁ , b ₂ , ..., b _k ) ^T , each component of b represents its size in the direction of the corresponding eigenvector:

For any face detection image, the sample shape vector can be expressed as:

其中y⁽ⁱ⁾＝{-1，1}i＝1，2，…r,其中y⁽ⁱ⁾＝1为正样本标记，y⁽ⁱ⁾＝-1为负样本标记；则训练的线性支持向量机为：

其中x_i表示样本集的子空间向量，α_i是权重系数，Ms是每个特征点支持向量的数量，b为偏移量；可得：y⁽ⁱ⁾＝W^T·x⁽ⁱ⁾+θ，W^T＝[W₁ W₂…W_n]是每个支持向量的权重系数，θ是便宜量；3-7-2) Establish a patch model for each feature point: take a fixed size patch area around each feature point, mark the patch area containing the feature point as a positive sample; then intercept the same size in the non-feature point area The patch is marked as a negative sample; there are a total of r patches for each feature point, which are composed of a vector (x ⁽¹⁾ , x ⁽²⁾ , ... x ^(r) ) ^T , for each image in the sample set, there are

where y ⁽ⁱ⁾ ={-1,1}i=1,2,...r, where y ⁽ⁱ⁾ =1 is a positive sample mark, and y ⁽ⁱ⁾ =-1 is a negative sample mark; then the linear support for training The vector machine is:

where x _i represents the subspace vector of the sample set, α _i is the weight coefficient, Ms is the number of support vectors for each feature point, and b is the offset; it can be obtained: y ⁽ⁱ⁾ = W ^T x ⁽ⁱ⁾ + θ, W ^T = [W ₁ W ₂ ... W _n ] is the weight coefficient of each support vector, θ is the cheap quantity;

每次优化这个目标函数得到一个新的特征点位置，然后再迭代更新，直到收敛到最大值。3-7-3) Face point fitting: perform a local search through the limited area of the currently estimated feature point position, and generate a similar response map R(X, Y) for each feature point. Fit a quadratic function to the response graph, assuming that R(X, Y) is the maximum value obtained at (x ₀ y ₀ ) in the field, then the quadratic function r(x ₀ , y ₀ )=a(xx ₀ ) ² +b(yy ₀ ) ² +c fits this position. Where a, b, c are the coefficients of the quadratic function, the quadratic function r ⁽ _x , The minimum error between y) and R(x, y); plus the deformation constraint cost function, the objective function of feature point search can be formed, and the objective function can be expressed as:

Each time the objective function is optimized, a new feature point position is obtained, and then iteratively updated until it converges to the maximum value.

Further, the judgment rule described in step 5) is:

Judging whether the micro-expression occurs through the EEG activity response time, if it is determined to occur, the change in the expression within the time threshold TL is searched according to the micro-EEG expression generation time T _sm as the time starting point, if it is judged by the time of the start and end frames of the expression If the expression within 500ms appears, it is finally judged that the micro-expression occurs; if the expression does not appear within 500ms, it is not finally judged that the micro-expression occurs.

Further, the time threshold TL is 500ms to 1000ms.

Another object of the present invention is to provide a micro-expression occurrence detection system.

The purpose of the present invention is achieved through such technical solutions, including:

The data acquisition module is used to record normal EEG data and normal facial video data before stimulation induces microexpressions; record EEG data and facial video data after stimulation induces microexpressions;

The time matching module is used to mark the time stamp of each piece of EEG data and each frame of facial video data to generate EEG time data and facial video time data;

The data processing module processes EEG data, EEG time data, facial video data and facial video time data, calculates the time T _sm of EEG micro-expression occurrence, and the start and end frames of facial expression changes judged by facial video data and facial video time data and top frame;

The micro-expression determination module determines whether the micro-expression occurs or not according to the EEG determination time T _sm of the micro-expression occurrence, the start and end frames and the top frame of the facial expression change.

Owing to adopting the above-mentioned technical scheme, the present invention has the following advantages:

1. The present invention solves the delay problem of EEG signal and facial image recognition by correlating EEG data and facial video data through time stamps, and realizes data synchronization; 2. The present invention combines the change time of EEG data and facial video data 3. This application processes the EEG signal and facial video data respectively, and then combines the processing results to determine whether a micro-expression occurs, and the result of the determination is as follows: High accuracy.

Other advantages, objects, and features of the present invention will be set forth in the description that follows, and will be apparent to those skilled in the art based on a study of the following, to the extent that is taught in the practice of the present invention. The objectives and other advantages of the present invention may be realized and attained by the following description and claims.

Description of drawings

The accompanying drawings of the present invention are described below.

FIG. 1 is a schematic flow chart of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

Example:

A method for detecting the occurrence of micro-expressions, recording the subject's normal EEG data and normal facial video data before stimulating micro-expressions, the left temporal lobe channel D23 in the 128-lead EEG data, and the D23 channel is the left temporal lobe The most representative channel is the right temporal lobe channel A09, the channel A09 is the most representative channel of the right temporal lobe, the prefrontal channel B26, and the channel B26 is the most representative channel of the prefrontal lobe. The face data is trained with 66 face coordinates and provides the face coordinates of 49 points in its output, and the face points are extracted after correcting the head pose in the calm state. An alignment is made between the neutral face of each subject and the average face of all subjects, and all track points of the sequence are recorded using the alignment. Reference points are generated by averaging the coordinates of the inner corners of the eye and nose landmarks. Calculate and average 38 points, including eyebrows, eyes and lips, to the reference point.

The specific method after stimulation induces microexpressions is as follows:

1) Stimulation induces micro-expressions, recording EEG data and facial video data;

2) Marking timestamp data for each section of EEG data and each frame of facial video data, matching and generating EEG time data and facial video time data;

3) process EEG data, EEG time data, facial video data and facial video time data to obtain T _sm , which is the moment when T _sm is judged by EEG as the occurrence of micro-expressions, and obtains the start and end frames and top frames of facial expression changes;

4) According to the T _sm processed in step 3), the start and end frames and the top frame of the facial expression change, determine whether a micro-expression occurs.

The specific method for recording EEG data and facial video data in step 1) is: by using the Biosemi Active system to record from 128 electrodes at a sampling rate of 1024 Hz to acquire and record normal EEG data and EEG data; Acquire and record normal facial video data and facial video data at a speed of 80 frames per second.

Described in step 2) is that each section of EEG data and each frame of face video data are all marked with time stamp data The concrete method is: utilize the time synchronization module of the Biosemi Active system, make the time stamp data in the time synchronization module synchronously transmitted to The EEG acquisition module and the high-speed camera acquisition module of the Biosemi Active system enable each piece of EEG data acquired by the EEG acquisition module and each frame of facial video data acquired by the high-speed camera to contain synchronized timestamp data, that is, the generated EEG time. data and facial video time data.

The specific steps for processing EEG data and EEG time data are as follows:

X(k)表示长度为N的序列的傅里叶变换，k表示频率；3-1) Taking the normal EEG data as the baseline data, calculate the normal value of the power spectral density PSD of the normal Gamma band of the left temporal lobe channel D23, the right temporal lobe channel A09, and the prefrontal lobe channel B26 in the normal state. The power spectral density represents The power carried by a unit frequency wave, the formula for calculating the normal value of PSD is:

X(k) represents the Fourier transform of a sequence of length N, and k represents the frequency;

3-2) For the EEG time data, set the sliding window duration W=2s, take 2*(1/fs) as the sliding time, where fs is the EEG sampling frequency; calculate the left temporal lobe channel D23 in the 2s sliding window , the right temporal lobe channel A09, the prefrontal lobe channel B26 PSD sliding window duration value in the Gamma frequency band, and compared with the normal PSD value of the corresponding channel; if the left temporal lobe channel D23, the right temporal lobe channel A09, the prefrontal lobe channel If the PSD value of any channel in channel B26 is higher than the normal PSD value, assuming that the EEG data has changed, go to step 3-3); any one of the left temporal lobe channel D23, the right temporal lobe channel A09, and the prefrontal lobe channel B26 If the channel PSD value is lower than the normal average PSD value, it will not be processed, and it is determined that the EEG data has not changed;

其中X_i(k)为EEG信号进行FFT变换后的结构，k为数据长度，即2s的数据，取能量值均值为门限值G：G＝1/2E；将能量值E与门限值G相比，若E>G，且5ms以内给定采样点En均能持续达到门限值(E1...En>G)，则初步认为Tn是不同脑区通道中反应的起始时刻T_s；同时，设置对比门限值PR，公式为

计算起点Ts之前n个采样点的对比值PRn，若|PN_n-PN_N-1|＝0则认为第一个采样点PRn对应的时刻为不同脑区通道中反应的起始时刻T；若反应起点Ts成功找到，则转向步骤3-4)；若反应起点Ts未成功找到，则返回步骤3-2)；3-3) Take the data within the sliding window duration W=2s assuming that the EEG data changes, first calculate the energy value E _i within these 2s, the energy formula

Among them, X _i (k) is the structure of the EEG signal after FFT transformation, k is the data length, that is, the data of 2s, and the average value of the energy value is the threshold value G: G=1/2E; the energy value E and the threshold value are Compared with G, if E>G, and the given sampling point En can continuously reach the threshold value within 5ms (E1...En>G), it is preliminarily considered that Tn is the starting time T of the response in different brain area channels _s ; at the same time, set the comparison threshold PR, the formula is

Calculate the contrast value PRn of n sampling points before the starting point Ts, if |PN _n -PN _N-1 |=0, then the time corresponding to the first sampling point PRn is considered to be the starting time T of the responses in different brain area channels; if If the reaction starting point Ts is successfully found, then turn to step 3-4); if the reaction starting point Ts is not successfully found, then return to step 3-2);

则记录通过EEG判别为微表情发生的时刻T_sm；若不满足条件，则返回步骤3-2)；3-4) At the time T _s of the starting point of the reaction, respectively take the starting time T _s of the left temporal lobe channel D23, the right temporal lobe channel A09, and the prefrontal lobe channel B26, if the time T _sD23 -T _sB26 >0 or T _sD23 -T _sA09 >0, that is, the start time of the D23 channel is before the start time of the B26 channel, or the start time of the D23 channel is before the start time of the A09 time, and must meet the

Then record the time T _sm that is judged as the occurrence of micro-expressions by EEG; if the condition is not met, then return to step 3-2);

3-5) Obtain the time T _sm at which the micro-expression is identified by EEG.

The concrete steps of processing face video data and face video time data described in step 3) are:

3-6) Detecting the human face, and detecting the specific position of the human face from the original image of each frame;

3-7) Perform face alignment and face reference point positioning on the face in the face video collection; according to the input face image, use the CLM local constraint model to automatically locate the key facial feature CLM local constraint model;

3-8) Extraction of deformation-based expression features: Using CLM to mark facial feature points, after obtaining the coordinates of facial reference points, calculate the relative slope information between these facial reference points, and extract deformation-based expression features; Track the key points in each area, extract the corresponding displacement information, and extract the distance information between the specific feature points of the expression picture, subtract the distance from the calm picture, obtain the change information of the distance, and extract the movement-based expression features;

3-9) According to the facial feature data extraction result, obtain the start and end frame and the top frame; wherein, by comparing the distance between the feature points and the distance difference k of the calm picture, the threshold value R is set to exceed the first frame image of k>R It is determined as the starting frame; by comparing the images after the starting frame, one frame of image with the maximum value of k is determined as the number of top frames, and the first frame with k<R is regarded as the ending frame.

Step 3-6) detects the human face, and the specific steps of detecting the specific position of the human face from the original image of each frame are:

3-6-1) Use local binary pattern (LBP) to extract the response image;

3-6-2) The AdaBoost algorithm is used to process the response image to separate the face area; the LBP algorithm first scans each pixel of the original image line by line, and takes the gray value of each pixel as the threshold value. , binarize the 3*3 adjacent points around it and form an 8-bit binary number in sequence, and use the value of this binary number (0~255) as the point response.

Step 3-7) According to the input face image, the specific steps of using the CLM local constraint model to automatically locate the key features of the face and the CLM local constraint model are as follows:

计算每个样本图像的形状和平均脸坐标的差值，可以得到一个零均值的形状变化矩阵X，对矩阵X进行PCA变换就可以得到人脸变化的主要成分，记特征值为λ_i，对应的特征向量为p_i；选择最大的k个特征值对应的特征向量组成正交矩阵P＝(p₁，p₂，…，p_k)；形状变化的权重向量b＝(b₁，b₂，…，b_k)^T,b的每个分量表示其在对应的特征向量方向的大小：

对任意的人脸检测图像，其样本形状向量可以表示为：

3-7-1) Modeling the shape of the face model: for M pictures, each picture has N feature points, the coordinates of each feature point are ( _xi , _yi ), and N on an image The vector composed of the coordinates of the feature points is represented by x=[x ₁ y ₁ x ₂ y ₂ ...x _N y _n ] ^T , and the average face coordinates of all images can be obtained:

Calculate the difference between the shape of each sample image and the average face coordinate, and a shape change matrix X with zero mean can be obtained. The main component of face change can be obtained by performing PCA transformation on the matrix X, and the eigenvalue is λ _i , corresponding to The eigenvector of is p _i ; the eigenvectors corresponding to the largest k eigenvalues are selected to form an orthogonal matrix P=(p ₁ , p ₂ , ..., p _k ); the weight vector of shape change b=(b ₁ , b ₂ , ..., b _k ) ^T , each component of b represents its size in the direction of the corresponding eigenvector:

For any face detection image, the sample shape vector can be expressed as:

其中y⁽ⁱ⁾＝{-1，1}i＝1，2，…n,其中y⁽ⁱ⁾＝1为正样本标记，y⁽ⁱ⁾＝-1为负样本标记；则训练的线性支持向量机为：

其中x_i表示样本集的子空间向量，α_i是权重系数，Ms是每个特征点支持向量的数量，b为偏移量；可得：y⁽ⁱ⁾＝W^T·x⁽ⁱ⁾+θ，W^T＝[W₁ W₂ … W_n]是每个支持向量的权重系数，θ是便宜量；3-7-2) Establish a patch model for each feature point: take a fixed size patch area around each feature point, mark the patch area containing the feature point as a positive sample; then intercept the same size in the non-feature point area The patch is marked as a negative sample; there are a total of r patches for each feature point, which are composed of a vector (x ⁽¹⁾ , x ⁽²⁾ , ... x ^(r) ) ^T , for each image in the sample set, there are

where y ⁽ⁱ⁾ ={-1,1}i=1,2,...n, where y ⁽ⁱ⁾ =1 is a positive sample mark, y ⁽ⁱ⁾ =-1 is a negative sample mark; then the linear support for training The vector machine is:

where x _i represents the subspace vector of the sample set, α _i is the weight coefficient, Ms is the number of support vectors for each feature point, and b is the offset; it can be obtained: y ⁽ⁱ⁾ = W ^T x ⁽ⁱ⁾ + θ, W ^T = [W ₁ W ₂ ... W _n ] is the weight coefficient of each support vector, θ is the cheap quantity;

每次优化这个目标函数得到一个新的特征点位置，然后再迭代更新，直到收敛到最大值。3-7-3) Face point fitting: perform a local search through the limited area of the currently estimated feature point position, and generate a similar response map R(X, Y) for each feature point. Fit a quadratic function to the response graph, assuming that R(X, Y) is the maximum value obtained at (x ₀ , y ₀ ) in the field, then the quadratic function r(x ₀ , y ₀ )=a(xx ₀ ) ² +b(yy ₀ ) ² +c fits this position. Where a, b, c are the coefficients of the quadratic function, the quadratic function r ⁽ _x , The minimum error between y) and R(x,y); plus the deformation constraint cost function, the objective function of feature point search can be formed, and the objective function can be expressed as:

Each time the objective function is optimized, a new feature point position is obtained, and then iteratively updated until it converges to the maximum value.

The determination rule described in step 5) is: determine whether the micro-expression occurs through the brain electrical activity response, and if it is determined to occur, then according to the time of occurrence of the micro-expression T _sm is the time starting point to search for the time threshold TL, and the search time threshold is generally 500ms～ The change of the expression within 1000ms, if it is judged by the time of the start and end frames of the expression that there is an expression that occurs within 500ms, it is finally judged that the micro-expression occurs; .

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Modifications or equivalent replacements are made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (10)

1. A method for detecting the occurrence of micro expression comprises the steps before stimulating and inducing micro expression and after stimulating and inducing micro expression, and the steps before stimulating and inducing micro expression, namely recording the normal electroencephalogram data and the normal facial video data of a tested person, and is characterized in that the method comprises the following steps after stimulating and inducing micro expression:

1) stimulating and inducing the micro expression, and recording electroencephalogram data and facial video data;

2) marking time stamp data for each segment of electroencephalogram data and each frame of face video data, and matching to generate electroencephalogram time data and face video time data;

3) processing the electroencephalogram data, the electroencephalogram time data, the face video data and the face video time data to obtain the electroencephalogram microexpression occurrence time T judged by the electroencephalogram data and the electroencephalogram time data_smObtaining start and stop frames and top frames of facial expression changes judged by the facial video data and the facial video time data;

4) judging the occurrence time T of the micro-expression according to the electroencephalogram processed in the step 3)_smAnd judging whether the micro expression occurs or not according to the start-stop frame and the top frame of the facial expression change.

2. The method for detecting the occurrence of the micro-expression according to claim 1, wherein the normal electroencephalogram data and the normal facial video data of the tested person are recorded, and the specific method for recording the electroencephalogram data and the facial video data in the step 1) comprises the following steps:

acquiring and recording normal electroencephalogram data and electroencephalogram data at a 1024Hz sampling rate from 128 electrode records by using a Biosemiactive system; normal face video data and face video data were acquired and recorded at a rate of 80 frames per second by a high-speed camera of the BiosemiActive system.

3. The micro-expression occurrence detection method according to claim 1, wherein the specific method of marking time stamp data for each segment of electroencephalogram data and each frame of facial video data in step 2) is:

the time synchronization module of the Biosemi Active system is utilized to synchronously transmit the timestamp data in the time synchronization module to the electroencephalogram acquisition module and the high-speed camera acquisition module of the Biosemi Active system, so that each segment of electroencephalogram data acquired by the electroencephalogram acquisition module and each frame of facial video data acquired by the high-speed camera comprise the synchronous timestamp data, namely, the electroencephalogram time data and the facial video time data are generated.

4. The micro-expression occurrence detection method according to claim 1, wherein the specific steps of processing the electroencephalogram data and the electroencephalogram time data in step 3) are as follows:

3-1) calculating the PSD normal values of normal Gamma wave bands of the left temporal lobe channel D23, the right temporal lobe channel A09 and the prefrontal lobe channel B26 in normal state by taking normal electroencephalogram data as baseline data, wherein the PSD normal values represent the power carried by unit frequency waves, and the PSD normal value calculation formula is

X (k) represents the fourier transform of a sequence of length N, k representing the frequency;

3-2) setting the duration of a sliding window as W2 s, taking 2 x (1/fs) as the sliding time and fs as the electroencephalogram sampling frequency aiming at the electroencephalogram time data; calculating PSD sliding window time length values of a left temporal lobe channel D23, a right temporal lobe channel A09 and a prefrontal lobe channel B26 in a 2s sliding window in a Gamma frequency band, and comparing the PSD sliding window time length values with normal PSD values of corresponding channels; if the PSD value of any one of the left temporal lobe channel D23, the right temporal lobe channel A09 and the prefrontal lobe channel B26 is higher than the normal PSD value, assuming that the electroencephalogram data change, turning to the step 3-3); the PSD value of any one of a left temporal lobe channel D23, a right temporal lobe channel A09 and a prefrontal lobe channel B26 is lower than the normal average PSD value, and the electroencephalogram data are determined to be unchanged;

3-3) taking the sliding window time length W of the change of the electroencephalogram data as the data in 2s, firstly calculating the energy value E in the 2s, wherein the energy formula is

Where x (N) is the signal amplitude, N is the data length, i.e. 2s of data, and the mean value of the energy values is taken as the threshold value G: G-1/2E; comparing the energy value E with a threshold value G if E>G, and a given sampling point En can continuously reach a threshold value within 5ms (E1.. En)>G) Then, Tn is preliminarily considered as the reaction starting point Ts; at the same time, a contrast threshold value PR is set, and the formula is

Calculating the comparison value PRn of n sampling points before the starting point Ts, if | PN_n-PN_N-1If | ═ 0, the moment corresponding to the first sampling point PRn is considered as a reaction starting point Ts; if the reaction starting point Ts is found successfully, turning to the step 3-4); if the reaction starting point Ts is not found successfully, returning to the step 3-2);

3-4) respectively taking the brain area channel starting time T of the left temporal lobe channel D23, the right temporal lobe channel A09 and the prefrontal lobe channel B26 at the time of a reaction starting point Ts_sIf at time T_sD23-T_sB26>0 or T_sD23-T_sA09>0, and must satisfy

The time point, namely the occurrence time T of the situation of the cerebellar ammeter is recorded_sm(ii) a If the condition is not met, returning to the step 3-2);

3-5) obtaining the electroencephalogram micro-expression occurrence time T under the electroencephalogram data_sm。

5. The method according to claim 4, wherein the step 3) of processing the face video data and the face video time data comprises the following steps:

3-6) detecting the human face, and detecting the specific position of the human face from the original image of each frame;

3-7) carrying out face alignment and face reference point positioning on the face in the face video acquisition; automatically positioning a face key characteristic CLM local constraint model by adopting a CLM local constraint model according to an input face image;

3-8) extracting expression characteristics based on deformation: utilizing CLM to label the facial feature points, obtaining the coordinates of the facial reference points, calculating the related slope information between the facial reference points, and extracting expression features based on deformation; simultaneously tracking key points in the three regions, extracting corresponding displacement information, extracting distance information between specific feature points of the expression pictures, subtracting the distance from the calm pictures to obtain change information of the distance, and extracting expression features based on movement;

3-9) obtaining a start-stop frame and a top frame according to the facial feature data extraction result; setting a threshold value R by comparing the distance between the feature points with the distance difference k of the calm picture, and judging the first frame image exceeding k > R as an initial frame; and judging the frame image with the maximum value of the k value as the top frame number by comparing the images after the initial frame, and taking the first frame when k is less than R as the termination frame.

6. The micro-expression occurrence detection method according to claim 5, wherein the step 3-6) of detecting the human face comprises the specific steps of:

3-6-1) extracting a response image by adopting a Local Binary Pattern (LBP);

3-6-2) processing the response image by adopting an AdaBoost algorithm to separate a human face area; the LBP algorithm firstly scans each pixel point of an original image line by line, binarizes adjacent points of 3 x 3 around each pixel point by taking the gray value of the point as a threshold value, and forms an 8-bit binary number according to the sequence, and takes the value (0-255) of the binary number as the response of the point.

7. The micro-expression occurrence detection method of claim 6, wherein the specific steps of automatically positioning the face key feature CLM local constraint model by using the CLM local constraint model according to the input face image in the step 3-7) are as follows:

3-7-1) modeling the shape of the human face model: for M pictures, each picture has N characteristic points, and the coordinate of each characteristic point is (x)_i、y_i) The vector composed of the coordinates of N feature points on one image is represented by x ═ x₁y₁x₂y₂… x_Ny_N]^TThe mean face coordinates of all images are available:

calculating the difference between the shape of each sample image and the average face coordinate to obtain aThe shape change matrix X with zero mean value is subjected to PCA conversion to obtain the main components of face change, and the characteristic value is recorded as lambda_iThe corresponding feature vector is p_i(ii) a Selecting the eigenvectors corresponding to the largest k eigenvalues to form an orthogonal matrix P ═ P₁，P₂，…，p_k) (ii) a Weight vector b of shape change is (b)₁，b₂，…，b_k)^TEach component of b represents its magnitude in the direction of the corresponding eigenvector:

for any face detection image, the sample shape vector can be expressed as:

3-7-2) establishing a patch model for each feature point: taking a patch area with a fixed size around each feature point, and marking the patch area containing the feature points as a positive sample; then, intercepting a patch with the same size in a non-characteristic point area and marking the patch as a negative sample; there are a total of r patches per feature point, which are grouped into a vector (x)⁽¹⁾，x⁽²⁾，…x^(r))^TFor each image in the sample set, there is

Wherein y is⁽ⁱ⁾1, 2, … r, wherein y is { -1, 1} i { -1, 2, … r⁽ⁱ⁾1 is a positive sample mark, y⁽ⁱ⁾-1 is a negative sample marker; the trained linear support vector machine is:

wherein x_iSubspace vector representing a sample set, α_iIs a weight coefficient, Ms is the number of support vectors for each feature point, b is an offset; the following can be obtained: y is⁽ⁱ⁾＝W^T·x⁽ⁱ⁾+θ，W^T＝[W₁W₂… W_n]Is the weight coefficient of each support vector, θ is the cheap quantity;

3-7-3) fitting face points: a similar response map R (X, Y) is generated for each feature point by performing a local search through the bounding region of the currently estimated feature point position. Fitting a quadratic function to the response plot, assuming R (X, Y) is domain-wide (X)₀，y₀) Where a maximum is obtained, a quadratic function r (x) may be used₀，y₀)＝a(x-x₀)²+b(y-y₀)²Where a, b, and c are coefficients of a quadratic function, using a least squares method of min ∑_x，y[R(x，y)-r(x，y)]²The minimum error between the quadratic functions R (x, y) and R (x, y) can be found; the deformation constraint cost function is added to form an objective function for searching the feature points, and the objective function can be expressed as:

optimizing the objective function each time to obtain a new feature point position, and then iteratively updating until the maximum value is converged.

8. The micro-expression occurrence detection method according to claim 7, wherein the determination rule in step 5) is:

judging whether the micro expression occurs or not according to the electroencephalogram activity reaction moment, and if so, judging the occurrence moment T according to the micro expression_smSearching for the change of the expression in the time threshold TL for the time starting point, and finally judging that the micro expression occurs if the expression occurring in 500ms is judged to occur according to the time of the starting and ending frames of the expression; if no expression occurring within 500ms appears, the micro expression is not judged to occur finally.

9. The method of claim 8, wherein the time threshold TL is 500ms to 1000 ms.

10. A system for microexpression detection using the detection method of any one of claims 1 to 9, wherein said detection system comprises:

the data acquisition module is used for recording normal electroencephalogram data and normal facial video data before the micro expression is induced by stimulation; recording the electroencephalogram data and the facial video data after the micro expression is induced by stimulation;

the time matching module is used for marking the time stamp of each section of electroencephalogram data and each frame of face video data to generate electroencephalogram time data and face video time data;

the data processing module is used for processing the electroencephalogram data, the electroencephalogram time data, the face video data and the face video time data and calculating the occurrence time T of the electroencephalogram micro-expressions_smThe start-stop frame and the top frame of the facial expression change are judged through the facial video data and the facial video time data;

a microexpression judgment module for judging the microexpression occurrence time T according to the electroencephalogram_smAnd judging whether the micro expression occurs or not according to the start-stop frame and the top frame of the facial expression change.

Images