CN111428661A - Method for processing face image based on intelligent human-computer interaction - Google Patents

Abstract

A face image processing method based on intelligent human-computer interaction belongs to the field of artificial intelligence. The facial features related to user information in the present invention are divided into overall facial features, expression features and visual tracking, and a three-dimensional face information can be obtained through combined analysis of the three aspects. The present invention binds specific actions in the user's facial features to behavior information, and realizes human-computer interaction through the facial feature information, thereby reducing the selection error of facial recognition points. The invention locates the basic facial features of the human face by analyzing the captured human face, performs visual tracking for the eye region, binds the facial expression to the behavior information to realize human-computer interaction, reduces the selection error of the facial recognition point, and improves the Facial recognition accuracy.

Description

A method of face image processing based on intelligent human-computer interaction

technical field

The invention belongs to the field of artificial intelligence, and relates to the realization of a method for processing face images, in particular to a method for performing face point detection, expression recognition, and visual tracking through real-time collection of face images and video information to realize face image processing. Methods.

Background technique

Human-computer interaction is the process of information exchange between people and systems. This process can exist in many different types of systems. The earliest human-computer interaction method is realized by manual operation and input of machine language instructions. The medium of interaction is computer. language. With the development of the field of graphics processing, the human-computer interaction medium is gradually changing to the graphical interface, which makes the graphical interface have a better interactive experience, makes the user's interaction more convenient, and feeds back more accurate information to the user. With the development of ubiquitous computing, deep learning and other related technologies, the types of human-computer interaction are constantly enriched, the media used for interaction have become diversified, and the main methods have been continuously improved, including speech recognition, gesture recognition, tracking, etc. Diversified, the amount of information that can be transmitted is greatly increased.

As an important field of pattern recognition, face recognition has always been one of the research hotspots. With the maturity of face analysis technology, face features are widely used in various occasions, and the problem of face image processing has been pushed up. The cusp of the times. Thereby, the present invention locates the feature area of the face, performs face point detection, expression recognition, and visual tracking on the face, and applies the three methods in combination, overcomes the different limitations of each method, and realizes the intelligent human face. computer interaction process.

Face point detection and recognition is one of the important applications in the field of deep learning. Face point recognition refers to locating key points on the face after detecting the face, and preprocessing the data after locating the face feature area. , using the recognition algorithm to extract features to complete face recognition, the process is shown in Figure 1. With the rapid development of deep learning technology, face recognition technology is also becoming more and more mature.

Expression recognition is an important direction for computers to understand human emotions and an important aspect of human-computer interaction. By analyzing expressions, you can capture user information and make decisions, and you can directly judge the user's emotions and psychology. Convolutional neural networks are commonly used to analyze emotions. By analyzing multiple convolutional layers and acquisition layers in the cellular neural network, high-level and multi-level features of the entire face or local area can be extracted, and good facial expression image feature analysis can be obtained. Experience has shown that the application of convolutional neural networks in image recognition is superior to other types of neural networks, and the best expression recognition effect can be achieved through convolutional neural networks.

Visual tracking is another important aspect in the process of human-computer interaction. Through visual tracking, the user's focus can be easily observed, which is more conducive to analyzing the user's area of interest and analyzing the user's choices and preferences. We use the human eye as the input of the computer. By tracking the user's line of sight, we determine the line of sight of the human eye and complete the corresponding human-computer interaction.

Based on the above analysis, it can be seen that the above three methods are respectively aimed at one direction in face recognition, so there are defects in individual applications, which lead to the occurrence of recognition errors. To this end, the present invention proposes a new face recognition method by combining the above three methods. By analyzing the face area of the captured face, the basic facial features of the face are located, and the eye area is visually tracked to identify the face. Expression binding behavior information to realize human-computer interaction, reduce facial recognition point selection errors, and improve facial recognition accuracy.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the problem of face image processing, and proposes a method for realizing face image processing through real-time collection of face images and video information, performing face point detection, expression recognition, and visual tracking.

The method for realizing the present invention is described as follows:

The present invention realizes intelligent human-computer interaction based on the face image processing technology of convolutional neural network, and the specific implementation method is divided into the following three processes:

(1) The method of face point detection

This method adopts a three-layer convolutional neural network structure, adopts the absolute value correction and parameter sharing mechanism to establish the first-layer convolutional neural network, and adopts the idea of multi-level regression to obtain the second and third-layer convolutional neural networks. Due to the large change in the face posture, the detection is unstable, and the relative position of the face point and the detection point may change in a large range, that is, a large relative error occurs. Therefore, the input area of the first-level network is selected to be large. to cover as many predicted locations as possible. The output of the first-level network provides selection conditions for the selection of subsequent detection areas, so the second and third-level detection areas are correspondingly reduced. The predicted location is a circular area centered on the location point with the highest predicted location density of the previous level network.

Re-acquire the predicted position in the new prediction area. This process is repeated several times until the detection area is reduced to 1% of the first-level detection area. The obtained predicted position is the predicted position of each point, and then the Multiple networks for different input areas;

The final predicted position x of a face point can be formally expressed as an n-level cascade, and the mathematical representation of the predicted position x is as follows:

表示，即x₁ ⁽¹⁾为第1层级上的第一个预测位置，第i层级上的第l_i个预测位置

相比较于第i-1层级上相对应的第l_i个预测位置的改变用Δx₁ ⁽i⁾表示。In the formula, x is the prediction position, l _i is the number of prediction positions on the n-level cascade on the first level, and the prediction positions on the i-th level are respectively

Representation, that is, x ₁ ⁽¹⁾ is the first prediction position on the 1st level, and the l _ith prediction position on the i-th level

The change compared to the corresponding li-th predicted position on the _i -1-th level is denoted by Δx ₁ ⁽ i ⁾ .

This method adopts the design of three-layer convolutional neural network. The first layer of convolutional neural network includes three deep convolutional networks with different detection areas. The detection area of the F1 network covers the entire face, the detection area of the EN1 network only covers the eyes and nose area, and the detection area of the NM1 network only covers the nose and mouth. area. The three networks use the above prediction method to predict different areas of the same face at the same time, and the predicted values of the three networks obtained are averaged to reduce the variance to obtain the first-level prediction position of the entire facial feature point, to avoid local features. The face prediction results that are too obvious are deviated from the actual ones. According to the idea of using regression for the prediction position of the first layer, the corresponding prediction positions of the second and third layers can be obtained for the three networks F1, EN1 and NM1 respectively. Since the range of the input area of the second and third levels is strictly limited by the prediction results of the first level, the prediction positions of the second and third levels can achieve extremely high accuracy, but will therefore be strictly limited.

(2) Methods of facial expression recognition

In the facial expression recognition method, an end-to-end learning model is proposed, which synthesizes face images from two angles of posture and expression, and performs facial expression recognition when the posture is unchanged. The structure of the model consists of a generator, two discriminators and a classifier. The image is preprocessed before feeding it into the model, and the face detection algorithm is applied to a base library containing 68 landmarks for face detection. After preprocessing, the facial image is input into the generator G to generate the identity. Specifically, there is a law f(x), so that each input image has a definite and unique identity, and then the identity is associated with Expression encoding e and pose encoding p are cascaded to represent changes in the face. By applying the minimax algorithm between the generator G and the discriminator D, and adding corresponding labels to the input of the decoder, new labels of face images with different poses and expressions can be obtained. Herein, the present invention uses two discriminator structures represented by Datt and Di respectively, wherein the discriminator Datt is used to identify and represent the entanglement degree of the identity, and Di is used to improve the quality of the generated image. After the face image is synthesized, the classifier Cexp is used to complete the facial expression recognition task of the face image. Specifically, the present invention applies the deep learning algorithm to the classifier to ensure that in each representation layer, the key factors of classification are gradually It tends to stabilize while preserving the characteristic information of each facial expression.

(3) The method of visual tracking

和梯度向量g_j之间的点积，得到人脸图像中眼部区域的最佳中心c^*，即瞳孔中心位置，由以下方法给出:The invention adopts the detection-based tracking algorithm, and by analyzing the image gradient vector field capturing the face, the relationship between the possible center and direction of the image gradient is mathematically described, a possible center c is set, and the center position is provided. A gradient vector, normalized so that the displacement is in the same direction as the gradient. By computing the normalized displacement relative to the center position

And the dot product between the gradient vector g _j , get the best center c ^* of the eye region in the face image, that is, the pupil center position, given by the following method:

和梯度向量g₁…g_N，即选取的第j个梯度对应的归一化位移为

梯度向量为g_j，当目标函数取得最大值时所对应的位置变量即为最佳中心位置c^*。其中，选取的不同梯度对应位移d_j可由如下方法得出：For a possible center c, select N different gradients, corresponding to different normalized displacements

and gradient vectors g ₁ ...g _N , that is, the normalized displacement corresponding to the selected j-th gradient is

The gradient vector is g _j , and the position variable corresponding to the maximum value of the objective function is the optimal center position c ^* . Among them, the selected displacements d _j corresponding to different gradients can be obtained by the following method:

Scale the displacement d _j to unit length to obtain the same weights at different positions in the face image. In order to improve the robustness of this method to linear changes in illumination and contrast, the gradient vector g _j is also scaled to unit length, and the target can be obtained. The function produces a maximum value at the pupil center position. In addition, the complexity of the algorithm can be reduced by considering only gradient vectors with significant amplitudes.

The advantages of the method proposed in the present invention applied to the field of face image processing are as follows:

(1) The present invention realizes human-computer interaction by recognizing facial features.

(2) The facial features related to the user information in the present invention are divided into overall facial features, expression features and visual tracking, and a three-dimensional face information can be obtained through the combined analysis of the three aspects.

(3) The present invention binds specific actions in the user's facial features to behavior information, and realizes human-computer interaction through the facial feature information, thereby reducing the selection error of facial recognition points.

Description of drawings

Fig. 1 is a schematic diagram of the human-computer interaction processing process based on face detection;

Figure 2 model block diagram;

Detailed ways

A method for processing face images based on intelligent human-computer interaction of the present invention will now be described in further detail with reference to the model block diagram and implementation examples in FIG. 2 .

A method for processing face images based on intelligent human-computer interaction of the present invention is a method of performing face point detection, expression recognition, and visual tracking through real-time collection of face images and video information to realize face image processing. method. The overall working framework of the invention applied to the field of face recognition is described as follows: first, the face is captured, the key points of the face are located through a three-layer convolutional neural network, and the face expression is recognized by an end-to-end deep learning model. Use different gestures and expressions to achieve face image synthesis and expression recognition, and achieve visual tracking by using image gradients to locate the center of the eye. After obtaining these three features, we combine the three features into a three-layer neural network for training to make the machine respond reasonably based on the combined features.

The present invention adopts the following technical solutions and implementation steps:

(1) The method of face point detection

This method adopts a three-layer convolutional neural network structure, adopts the absolute value correction and parameter sharing mechanism to establish the first-layer convolutional neural network, and adopts the idea of multi-level regression to obtain the second and third-layer convolutional neural networks. Due to the large change in the face posture, the detection is unstable, and the relative position of the face point and the detection point may change in a large range, that is, a large relative error occurs. Therefore, the input area of the first-level network is selected to be large. to cover as many predicted locations as possible. The output of the first-level network provides selection conditions for the selection of subsequent detection areas, so the second and third-level detection areas are correspondingly reduced. The predicted location is a circular area centered on the location point with the highest predicted location density of the previous level network.

Re-acquire the predicted position in the new prediction area. This process is repeated several times until the detection area is reduced to 1% of the first-level detection area. The obtained predicted position is the predicted position of each point, and then the Multiple networks for different input areas;

The final predicted position x of a face point can be formally expressed as an n-level cascade, and the mathematical representation of the predicted position x is as follows:

表示，即x₁ ⁽¹⁾为第1层级上的第一个预测位置，第i层级上的第l_i个预测位置

相比较于第i-1层级上相对应的第l_i个预测位置的改变用Δx₁ ⁽ⁱ⁾表示。In the formula, x is the prediction position, l _i is the number of prediction positions on the n-level cascade on the first level, and the prediction positions on the i-th level are respectively

Representation, that is, x ₁ ⁽¹⁾ is the first prediction position on the 1st level, and the l _ith prediction position on the i-th level

The change compared to the corresponding li-th predicted position on the _i -1-th level is denoted by Δx ₁ ⁽ⁱ⁾ .

This method adopts the design of three-layer convolutional neural network. The first layer of convolutional neural network includes three deep convolutional networks with different detection areas. The F1 network detection area covers the entire face, the EN1 network detection area only covers the eyes and nose area, and the NM1 network detection area only covers the nose and mouth. area. The three networks use the above prediction method to predict different areas of the same face at the same time, and the predicted values of the three networks are averaged to reduce the variance to obtain the first-layer predicted position of the entire facial feature point, avoiding the local characteristics. The face prediction results that are too obvious are deviated from the actual ones. According to the idea of using regression for the prediction position of the first layer, the corresponding prediction positions of the second and third layers can be obtained for the three networks F1, EN1 and NM1 respectively. Since the range of the input area of the second and third levels is strictly limited by the prediction results of the first level, the prediction positions of the second and third levels can achieve extremely high accuracy, but will be strictly limited.

(2) Methods of facial expression recognition

In the facial expression recognition method, an end-to-end learning model is proposed, which synthesizes face images from two angles of posture and expression, and performs facial expression recognition when the posture is unchanged. The structure of the model consists of a generator, two discriminators and a classifier. The image is preprocessed before feeding it into the model, and the face detection algorithm is applied to a base library containing 68 landmarks for face detection. After preprocessing, the facial image is input into the generator G to generate the identity. Specifically, there is a law f(x), so that each input image has a definite and unique identity, and then the identity is associated with Expression encoding e and pose encoding p are cascaded to represent changes in the face. By applying the minimax algorithm between the generator G and the discriminator D, and adding corresponding labels to the input of the decoder, new labels of face images with different poses and expressions can be obtained. Herein, the present invention uses two discriminator structures represented by Datt and Di respectively, wherein the discriminator Datt is used to identify and represent the entanglement degree of the identity, and Di is used to improve the quality of the generated image. After the face image is synthesized, the classifier Cexp is used to complete the facial expression recognition task of the face image. Specifically, the present invention applies the deep learning algorithm to the classifier to ensure that in each representation layer, the key factors of classification are gradually It tends to stabilize while preserving the characteristic information of each facial expression.

(3) The method of visual tracking

和梯度向量g_j之间的点积，得到人脸图像中眼部区域的最佳中心c^*，即瞳孔中心位置，由以下方法给出:The invention adopts the detection-based tracking algorithm, and by analyzing the image gradient vector field capturing the face, the relationship between the possible center and direction of the image gradient is mathematically described, a possible center c is set, and the center position is provided. A gradient vector, normalized so that the displacement is in the same direction as the gradient. By computing the normalized displacement relative to the center position

And the dot product between the gradient vector g _j , get the best center c ^* of the eye region in the face image, that is, the pupil center position, given by the following method:

和梯度向量g₁…g_N，即选取的第个梯度对应的归一化位移为

梯度向量为g_j，当目标函数取得最大值时所对应的位置变量即为最佳中心位置c^*。其中，选取的不同梯度对应位移d_j可由如下方法得出：For a possible center c, select N different gradients, corresponding to different normalized displacements

and the gradient vector g ₁ ...g _N , that is, the normalized displacement corresponding to the selected gradient is

The gradient vector is g _j , and the position variable corresponding to the maximum value of the objective function is the optimal center position c ^* . Among them, the selected displacements d _j corresponding to different gradients can be obtained by the following method:

Scale the displacement d _j to unit length to obtain the same weights at different positions in the face image. In order to improve the robustness of this method to linear changes in illumination and contrast, the gradient vector g _j is also scaled to unit length, and the target can be obtained. The function produces a maximum value at the pupil center position. In addition, the complexity of the algorithm can be reduced by considering only gradient vectors with significant amplitudes.

Claims (2)

1. A method for processing face images based on intelligent human-computer interaction is characterized in that the specific implementation method comprises the following three processes:

method for detecting face points

Adopting a three-layer convolutional neural network structure, adopting an absolute value correction and parameter sharing mechanism to establish a first layer of convolutional neural network, and adopting a multi-layer regression idea to obtain two-layer and three-layer convolutional neural networks;

the input area of the first-level network is selected to be larger so as to cover the predicted positions as much as possible; correspondingly reducing the second and third detection areas, wherein the selection conditions of the detection areas are as follows: a circular area including 75% of predicted positions among all predicted positions obtained by the previous network, the circular area being centered at a position point where the density of predicted positions of the previous network is highest;

the prediction position is obtained again in the new prediction area, the process is repeated for many times until the detection area is reduced to 1% of the first-stage detection area, the obtained prediction position is the prediction position of each point, and then a plurality of networks of different input areas of each level are obtained;

the final predicted position x of the face point is represented as a cascade of n levels, and the mathematical representation of the predicted position x is as follows:

where x is the predicted position, l_iThe number of predicted positions on the ith level is n levels of cascade connection, and the predicted positions on the ith level are respectively used

Is represented by, i.e. x₁ ⁽¹⁾For the first predicted position at level 1, l at level i_iA predicted position

Compared with the corresponding l-th level on the i-1 level_iΔ x for the change of predicted position₁ ⁽ⁱ⁾Represents;

(II) facial expression recognition method

In the facial expression recognition method, an end-to-end learning model is provided, the model carries out facial image synthesis through two angles of gesture and expression, and carries out facial expression recognition when the gesture is unchanged; the structure of the model consists of a generator, two discriminators and a classifier;

preprocessing is carried out before the image is transmitted into the model, and a face detection algorithm is applied to a basic library containing 68 mark points for face detection; after preprocessing, inputting facial images into a generator G to generate identity marks, wherein each input image has a determined and unique identity mark, and then cascading the identity marks with an expression code e and a posture code p to express the change of the face; applying a maximum and minimum algorithm between the generator G and the discriminator D, and adding corresponding labels at the input end of a decoder to obtain new labels of face images with different postures and expressions;

two discriminator structures are respectively represented by Datt and Di, wherein the discriminator Datt is used for identifying and representing the entanglement degree of the mark, and the Di is used for improving the quality of the generated image; after the face image is synthesized, the classifier Cexp is used for completing the facial expression recognition task of the face image;

(III) visual tracking method

Adopting a tracking algorithm based on detection, analyzing an image gradient vector field of a captured face, describing the relation between the possible center and the direction of the image gradient by a mathematical method, setting a possible center c, providing a gradient vector at the center position, and enabling the displacement to be the same as the gradient direction through normalization;

by calculating a normalized displacement related to the position of the centre

And gradient vector g_jThe optimal center c of the eye region in the face image is obtained^*I.e., the pupil center position, is given by:

aiming at a possible center c, selecting N different gradients which respectively correspond to different normalized displacements

And gradient vector g₁…g_NI.e. the selected normalized displacement corresponding to the jth gradient is

Gradient vector is g_jWhen the objective function obtains the maximum value, the corresponding position variable is the optimal center position c^*(ii) a Wherein the selected different gradients correspond to a displacement d_jThe method comprises the following steps:

will displace d_jZooming to unit length to obtain the same weight and gradient vector g of different positions in the face image_jAnd the unit length is also scaled, so that the maximum value of the target function at the pupil center position is obtained.

2. The method of claim 1, wherein:

the first layer of convolutional neural network comprises three deep convolutional networks with different detection areas, wherein F1 network detection areas cover the whole face, EN1 network detection areas only cover the eyes and nose areas, and NM1 network detection areas only cover the nose and mouth areas; the three networks simultaneously predict different areas of the same face, the obtained predicted values of the three networks are averaged, and the corresponding second-layer predicted positions and the corresponding third-layer predicted positions are obtained respectively according to the three networks F1, FN1 and NM1 by adopting a regression idea according to the first-layer predicted positions.

Images