CN1794265A - Method and device for distinguishing face expression based on video frequency - Google Patents

Abstract

The present invention proposes a video-based facial expression recognition method and device. When performing real-time video facial expression recognition, the method mainly applies the ASM contour extraction algorithm to the extraction of feature vectors, and according to the position of the eyes of the face. The face image is extracted, and the normalized eigenface is generated from the position of the chin of the face, and the most effective features in the eigenface are extracted with the AdaBoost algorithm, and the purpose of facial expression recognition is finally achieved. The present invention can eliminate the influence of light in use, and in the method, the face image has been specially processed, so that the gray mean value and variance of the left and right parts of the face are basically consistent, and the method of the present invention is aimed at the video of a commonly used USB camera. Algorithms that automatically detect and track faces in real time and recognize the four common expressions of positive faces can achieve better technical and commercial results.

Description

Video-based facial expression recognition method and device

technical field

The invention relates to a recognition method, in particular to a video-based recognition method and device for human facial expressions.

Background technique

With the deepening of human-computer interaction research and great application prospects, facial expression recognition has become a research hotspot in the field of pattern recognition and artificial intelligence. But the real-time recognition of human facial expressions is a very difficult problem, many theories are not perfect, and there are almost no mature commercial results. The difficulty of facial expression recognition is that the same expression made by different people is quite different, and the difference between different expressions is also very subtle. In addition, lighting and facial posture will also affect the accuracy of recognition. Expression recognition methods are generally based on statistics, that is, extracting feature vectors from face images, then training classifiers, and finally performing recognition.

Feature extraction is the key to the success of recognition. Currently, the features used for expression recognition can be divided into two types: local features and overall features. Facial expression recognition based on local features is to use the position, size and mutual position of each person's facial features (eyebrows, eyes, nose, mouth and facial contours, etc.) to extract features and achieve facial expression recognition. Purpose. The recognition based on the overall features of the face starts from the whole face image, and proposes the features that reflect the whole to realize the recognition of facial expressions. The amount of data of local features is relatively small, but it uses limited features to represent the entire image, which will lose useful information. Moreover, the accurate and automatic extraction of facial features is a difficult problem.

In the existing technology, someone proposes to use the Fisher criterion function to identify the facial expression of the human face, that is, identify the overall features of the human face, and use the back propagation algorithm to identify the human face. The basic steps of recognition are: a. Preprocessing the received image; b. Extracting local features of the face; c. Extracting overall features; d. Fusion of local and overall features; facial expression recognition. However, this recognition method only analyzes and judges the characteristics of the human face more clearly. Although it can roughly reflect the facial expressions on the human face, it is still affected by external factors such as lighting and cannot be accurately, quickly and automatically. Extract it.

Contents of the invention

The technical problem to be solved by the present invention is: the existing technology cannot accurately and automatically extract human facial expressions. The present invention proposes a facial expression recognition method in the case of video, and the purpose is to be able to solve the defects existing in the prior art. The method is based on the overall features of the face, and generates a standard face according to the automatically extracted chin contour of the face, and then uses the AdaBoost algorithm to select the most effective features to obtain a robust recognition result.

The method of the present invention is an algorithm that can automatically detect, track and recognize the common facial expressions on the face in real time, especially the four most common expressions, and can avoid recognition Facial expressions are affected by factors such as lighting.

The purpose of the present invention is achieved like this:

A method for facial expression recognition based on video, comprising the following steps:

Collect the facial expression image data of the face from the video data input by the USB camera, and preprocess the image data;

Real-time extraction of the position of the face in the preprocessed image;

Positioning the human eyes in the human faces in the determined image according to the human eye classifier;

Extract the image area containing the face according to the determined position of the human eyes and the information of the face classifier, and perform normalization processing;

Positioning of facial organs;

Determine the position of the chin of the face according to the positioning of the facial organs, determine the face area in the image, generate the eigenface, and use it as a classification sample;

Calculate the Gabor feature of the eigenface image;

Select the calculated Gabor features;

Construct a support vector machine classifier from the selected features;

According to the constructed classifier, the result of facial expression recognition is obtained.

When collecting data input from a USB camera, the following face image tracking steps are included:

Before obtaining the tracking target, search each frame of image to detect the existence of face image;

If one or more faces are detected in a frame of images, the detected faces are tracked in the next two frames of images, and the faces tracked in these two frames of images are detected and verified, and a judgment is made on the detection results ;

After the face is detected in all three frames of images at the same position, the algorithm considers that there is a face image at this position, and then executes the real-time face detection algorithm to extract the position of the face in the image;

If it is detected that there are multiple face images in the scene, select the largest face image to start tracking, and continue to track the face in subsequent frames, such as the similarity between the tracking results of the next frame and the previous frame in adjacent frames If the value is too low, or if no upright face is detected in the area where a tracking target is located for a long time, the tracking will stop.

The normalization process is realized by a resampling algorithm: the resampling algorithm is scaling, rotation and translation transformation, so that the detected position of the human eye overlaps with the position of the human eye classifier for positioning.

The target extraction method is adopted for the positioning of the facial organs, and the target extraction method is an active shape model algorithm.

The concrete steps of described active shape model algorithm are:

Extract the contour information of the face from the video data, and establish a sample unit;

Normalize and align the samples in the sample unit, and then perform principal component analysis transformation;

The gray level information of each control point in the transformed contour information is analyzed for the principal component, as the basis for point search;

The average profile calculated by principal component analysis is used as the initial value of the profile search, and the iterative search is performed to obtain the final result.

The steps of the iterative search are:

Obtain the initial translation value according to the grayscale information, align the new contour obtained according to the grayscale search to the average contour, and calculate the aligned parameter value;

Calculate the change in shape based on the aligned data and statistics calculated by principal component analysis;

Invert the changed shape to the original position according to the aligned parameter value to get the result of a search;

Repeat the above search steps and continue to iterate until convergence to get the final result.

The generating eigenface is to compare the extracted human face outline with the human face in the human face classifier, and adjust the inclination.

There is also a step of processing the generated eigenface between generating the eigenface and calculating the Gabor feature of the eigenface image: normalizing the gray level of the left and right parts of the formed eigenface, so that the gray mean value of the left and right parts and The variance is the same.

A gray filter band is set between the left and right parts of the eigenface.

The established support vector machine classifier is a multi-class classifier, which is one-to-one, one-to-many or decision tree.

The present invention also proposes a video-based facial expression recognition device, including a video data acquisition unit, an image processing unit, a face information database, and a facial expression recognition unit;

The video data collection unit collects the face image of the video and transmits it to the image processing unit;

The image processing unit calls the face information from the face information database and compares it with the collected face image, then calculates the face data, and transmits the calculated data to the facial expression recognition unit;

The facial expression recognition unit recognizes the collected face images according to the recognition information stored in the face information database.

It also includes a display unit for displaying the recognized facial expressions.

The image processing unit includes a comparison unit, a feature generation unit, a calculation unit and a classifier unit;

The comparison unit compares the image information of the face with the image information in the face database, detects the face and eyes, extracts the face image according to the positions of the eyes, and transmits the face information to the feature generation unit ;

The feature generation unit locates the facial organs, generates the eigenface according to the chin of the face, and sends the eigenface as a sample to the calculation unit;

The computing unit calculates the Gabor feature of the eigenface image, and adopts the AdaBoost algorithm to select the feature, and then transmits the selected feature to the classifier unit;

The classifier unit constructs a support vector machine classifier according to the selected features, and transmits the classifier information to the facial expression recognition unit.

The video data collection unit also includes a video data tracking unit, which tracks and detects the face data of the video data, and judges whether to collect the input data.

The technical scheme of the above-mentioned method of the present invention makes the facial expression of accurate people's face can be extracted automatically in the video situation, and this method has adopted Adaboost and ASM algorithm, can eliminate the influence of illumination, has carried out special to people's face image in the method processing, so that the gray mean value and variance of the left and right parts of the human face are basically consistent, and the method of the present invention develops a real-time automatic detection and tracking of the human face and can identify the frontal human face for the video data of the commonly used USB camera. The algorithm of the four common expressions can achieve better technical and commercial results.

Description of drawings

FIG. 1 is a flow chart of the method for video-based facial expression recognition of the present invention.

FIG. 2 is a schematic diagram of expression collection in an embodiment of the video-based facial expression recognition method of the present invention.

Fig. 3 is a schematic diagram of the normalization processing of the face image shape.

Figure 4 is a schematic diagram of the detection of the ASM algorithm.

Figure 5a shows the eigenfaces of the collected face contours.

Figure 5b shows the standard eigenface.

Figure 6 is a schematic diagram of eigenface generation.

FIG. 7 is a schematic diagram of Gabor features of the image at different scales and angles when calculating the Gabor features of the feature face image.

Fig. 8 is a schematic diagram of a one-to-one classifier according to the present invention.

Fig. 9 is a recognition effect diagram of the method described in the present invention.

Fig. 10 is a structural block diagram of the device of the present invention.

Detailed ways

The present invention provides a video-based human face and facial expression recognition method. The method is made for the video data of a commonly used USB camera. The method can automatically detect and track the human face in real time and can recognize the frontal human face Common expressions.

With reference to shown in Fig. 1 of the present invention, it is the flow chart of identification method described in the present invention, and the steps that it specifically comprises are as follows:

First, collect the facial expression image, the collection step is specifically: collect the facial expression image data of the human face from the video data input by the USB camera, and preprocess the image data;

In an embodiment of the present invention, the image acquisition process also includes a face data tracking step, the purpose of this step is to detect multiple faces in the shooting scene in real time, and for one of the faces such as the largest face to continue Tracking and continuous verification during the tracking process to determine the presence or absence of a face. This tracking step can detect faces with a depth rotation of -20 to 20 degrees and a plane rotation of -20 to 20 degrees, and can detect faces with different skin colors, faces under different lighting conditions, or faces with glasses, etc. The tracking algorithm is not affected by the posture of the face, and the profile and rotated faces can also be tracked.

This trace step is achieved in the following way:

Before obtaining the tracking target, search each frame image to detect whether there is a face; if one or more faces are detected in a certain frame image, then track these faces in the next two frames of images, and compare the two The face tracked in the frame image is detected and verified to determine whether the previous detection result is a real face; only after a face is detected in a certain position in three frames, the algorithm considers that the face exists at this position, and continues to process the face image To judge and identify. In this tracking step, if there are multiple faces in the scene, select one of them to track. Continue to track the face in subsequent frames. If the similarity between the tracking results of the next frame and the previous frame in the adjacent frames is too low, stop tracking; if no frontal upright person is detected in the area where a tracking target is located for a long time face, it thinks that the tracking value of the target is not great, and stops tracking the target. After the previous tracking target stops tracking, face detection is performed again in subsequent images until a new face is found, the new face is tracked, and the steps of face tracking are repeated.

With reference to the content shown in Fig. 1, after gathering facial expression image, carry out face detection step then, the face detection in the present embodiment is to adopt real-time face detection algorithm based on video to extract people's face in real time in the pre-preparation in fact. The position in the processed image; the recognition method can be referred to as shown in Figure 2. The current algorithm can recognize different expressions, such as neutral, laughing, angry and surprised, and the recognition algorithm is based on statistical principles. Before carrying out the identification of the method described in the present invention, must at first collect a large amount of samples, can record the facial expression video of the collected person by USB camera, the image that contains facial expression that separates from video file is used as to carry out An initial sample of statistics to form an initial sample for use in the recognition process.

In the method of the present invention, the purpose of face detection is to determine the position of the face in the collected image, and after the position of the face is determined, the detection of both eyes can be performed. Referring to the binocular detection step in Fig. 1 simultaneously, this step is to make location according to the people's eyes in the people's face in the determined image according to the people's eyes classifier; This step is after the image area of the people's face detected, based on The human eye classifier is used to determine the position of the human eye. The human eye classifier is generally established based on statistical method detection, that is, the classifier is first trained according to the human eye samples, and then detected according to the classifier.

Referring to the contents shown in Fig. 1 and Fig. 3, an image containing only a human face is extracted according to the position of the eyes. This step is to extract the image area containing a human face according to the determined position of the human eyes and the information of the human eye classifier, and perform classification. One treatment. The normalization process can be referred to in Figure 3. The image captured from the video in Figure 3a refers to the standard face template in Figure 3b, and finally the normalized result shown in Figure 3c is obtained. This is because in the case of video, the size of the face area will change with the distance between the real face and the USB camera, which is very unfavorable to the algorithm of organ positioning. An image is resampled from the data. The position of the eyes in the image is fixed and the connection is horizontal. The resampled image covers the entire area of the face.

The resampling algorithm is a simple transformation of scaling, rotation and translation, that is, after the above-mentioned transformation, the detected eyes overlap with the positions of the eyes in the standard face image. The standard image size can be: 120*148. The specific calculation formula is:

x=λ(x′cosθ+y′sinθ)+x ₀

y=λ(-x'sinθ+y'cosθ)+y ₀

Let λcosθ=a, λsinθ=b, then the formula can be written as:

x=ax'+by'+x ₀

y=-bx'+ay'+y ₀

There are only four unknowns in the above formula, two equations can be listed for each point, and all unknowns can be solved with two points. So this transformation can be done by the position of the eyes.

The size of the face image processed by the above resampling algorithm is the same as that of the pre-trained standard image, and the detected eyes (point × in Figure 3) are the same as those in the standard image after rotation and translation.

After carrying out the extraction of above-mentioned people's face image, continue to refer to Fig. 1, carry out people's face organ location, this location people's face organ adopts target extraction algorithm to realize, can adopt ASM (Active Shape Model, Active Shape Model) in the embodiment of the present invention ) algorithm, the purpose of this step is to accurately extract the region of the face, and remove irrelevant background information in the image. In the method of the present invention, the position of the general outline of the human face needs to be determined, and the ASM introduces the statistical information of the existing human face outline as a constraint condition, which is used to control the change of the outline shape in the outline search. The ASM method can quickly and accurately extract the contour of the face and locate the facial organs.

Wherein, the concrete steps of described ASM algorithm are:

Firstly, the contour information of the face is extracted from the video data, and the sample unit is established;

Then, normalize and align the samples in the sample unit, and then perform principal components analysis (PCA for short) transformation;

The gray information of each control point in the contour information processed in the PCA transformation is used as the basis for point search;

Furthermore, the average contour calculated by principal component analysis is used as the initial value of the contour search, and the iterative search is carried out to obtain the final result.

When performing the ASM algorithm, the specific steps of the iterative search are:

Obtain the initial translation value according to the grayscale information, align the new contour obtained according to the grayscale search to the average contour, and calculate the aligned parameter value;

Calculate the change in shape based on the aligned data and statistics calculated by principal component analysis;

Invert the changed shape to the original position according to the aligned parameter value to get the result of a search;

Repeat the search step and iterate until convergence to get the final result.

In the method of the present invention, in order to improve the search speed and accuracy, pyramid images can also be introduced for hierarchical search. And, when the present invention carries out ASM algorithm, owing to introducing PCA calculation and statistics method to control the variation of human face contour, the algorithm of ASM can find out the contour of human face more accurately, and the speed of algorithm is also faster, and the iterative search The calculation can converge within 1 second. In the algorithm scheme of the present invention, the detected position of the human eye can be used to determine the initial position of the outline. At the same time, in order to improve the accuracy of organ positioning, the present invention makes the images stored in the database The size is consistent with the actual detected image size. When implementing the present invention, in fact also can adopt AAM (Active Aspect Model, active appearance model) algorithm to realize the search to the outline of human face, because this algorithm is often used in the prior art, so no longer in the present embodiment repeat.

From the above description, combined with the content shown in FIG. 4, it can be seen that the algorithm of the present invention can better restore the position of the chin in the face, and can well maintain the overall shape of the outline.

Continue to refer to the content in Figure 1, after positioning the facial organs, determine the position of the chin of the face according to the positioning of the facial organs, determine the face area in the image, generate a feature face, and use it as a classification sample; in the middle of this step , when generating eigenfaces, the samples used for classification should contain the main area of the face, and remove those useless information that will affect the recognition effect. In the process of facial expression recognition, only the positive facial expression recognition is considered Under the background, the main factors affecting recognition are background and illumination. The method of the present invention is to extract the position of the chin according to the ASM algorithm, and the face area in the image can be extracted separately as an eigenface image for facial expression recognition. The size of the eigenface is fixed, and the size of the general eigenface Setting it to 64*64 can meet the recognition rate and speed requirements. If the eigenface is too small, the recognition rate will decrease, and if it is too large, it will affect the efficiency of the algorithm.

Combining the contents shown in Figure 5 and Figure 6 at the same time, Figure 5a is the collected eigenface, Figure 5b is the standard eigenface, and Figure 5a has multiple parallel straight lines from top to bottom, which can be seen from the position of the straight lines, where A line determines the position of the chin of the face, and Figure 5b shows the standard eigenface in the face classifier, the standard eigenface is obtained by training before recognition, and the figure is also marked from top to bottom with With the same number of multiple parallel lines in Figure 5a, the position of the corresponding lines in Figure 5a can also determine the position of the chin of the face; as can be seen from the comparison between Figure 5a and Figure 5b, the face actually extracted according to the video input The size of the silhouette is not the same as that of the standard eigenface, and there may be a skew. The method of the present invention can sample along the calculated inclination angle, as shown in Figure 5a, the multiple lines correspond to the same number of lines in Figure 5b, and the relationship between the corresponding lines can be the actual face area Resampling is an image that is exactly the same size as the standard eigenface. After such sampling, the face detected in the video can be converted into a face image with the same size and angle as the standard eigenface. This is a process of standardizing the collected face images. Standardization here refers to geometrically transforming the faces detected in the video to make them consistent with the standard eigenfaces we set. The purpose of standardization is to facilitate the generation of samples and the extraction of features, and improve the accuracy of recognition.

Referring to the content of FIG. 6 at the same time, the left picture is the face image extracted from the video data, and the right side is the eigenface obtained after resampling. The size of the eigenface is preferably 64*64. The recognition of the present invention is based on image grayscale information, so the illumination will affect our final recognition result. In order to remove the influence of illumination, we process the generated eigenfaces. The method is to normalize the gray levels of the left and right sides of the eigenface, so that the gray mean and variance of the left and right parts are the same. At the same time, in order to avoid a grayscale jump in the middle, a grayscale transition zone is set up in the middle of the left and right faces according to this method, so that the grayscale can smoothly transition from the left part of the face to the right part.

Continue to refer to the content of Figure 1, after the eigenface is generated, the Gabor feature of the eigenface image is calculated; as shown in Figure 7, 5 scales and Gabor features in 6 directions can be calculated for each pixel of the eigenface image , that is, each pixel can get a 30-dimensional vector, and the Gabor features of all pixels of the 64*64 image can be combined to get a 122880-dimensional feature vector. In actual calculation, in order to speed up the calculation, the present invention uses fast Fourier transform (FFT) to calculate the Gabor feature.

Referring to the content shown in Figure 1, after calculating the Gabor feature of the eigenface image, the Gabor feature calculated needs to be selected; in the method of the present invention, the dimension of the Gabor feature vector calculated according to the eigenface is as high as 122880 dimensions, this will bring great trouble to the training and calculation of the present invention, and cause the algorithm to be inefficient. Therefore, the present invention uses the AdaBoost algorithm to select features, and the Adaboost method extracts the most effective part of the features from the original vector. as a sample for classification. The basic principle of the AdaBoost algorithm is to continuously combine weak classifiers to form a strong classifier with strong classification ability. In the process of using AdaBoost for calculation, we can select a series of features with the best classification ability, and obtain the final classifier according to the weights obtained from training. The Adaboost algorithm itself is realized by changing the data distribution. It determines the weight value of each example according to whether the classification of each example in each training set is correct, and the overall classification accuracy of the last time. Finally, the classifiers obtained from each training are fused together as the final decision classifier.

With reference to the content shown in Fig. 1 and Fig. 8, after carrying out feature selection, support vector machine (SVM) classifier is constructed by the feature selected; For example, the method of the present invention adopts AdaBoost algorithm to select 2000 dimension features as training samples , of course, in practical applications, features such as 3000 dimensions and 4000 dimensions can also be selected as training samples. In this embodiment, 2000 dimensions are taken as an example to form an SVM classifier. In this embodiment, four types of expressions are basically distinguished , and thus is a multi-class classifier. In fact, multi-class classifiers are relatively simple two-class classifiers. In the embodiment of the present invention, since at least four expressions are to be recognized, each expression can be regarded as a class, so it is a multi-class classifier. And SVM can construct linear classifier and nonlinear classifier. In the method of the present invention, both classifiers can be implemented, but the speed of recognition will be faster by using a linear classifier. Therefore, without affecting the recognition rate, using a linear classifier is a preferred embodiment of the present invention. The design of the multi-class classifier described in the present invention can have multiple options: one-to-one, one-to-many, decision tree, etc. One-to-one is to design a classifier between every two classes. For example, if the present invention has four classes, then there are 6 combinations, and the present invention can construct 6 classifiers. If it is one-to-many, then we can design a classifier between each class and other classes, and four classes can get four classifiers. Complex decision trees can also be designed.

In this embodiment, a one-to-one design method is used for illustration, and the function of the one-to-one classifier is to divide two classes. In expression recognition, if any combination of two classes (for example, there are 6 combinations of four expressions) is used to design a classifier using the above method, 6 one-to-one classifiers can be obtained. Using these one-to-one classifiers, we can distinguish four expressions.

The principle can be shown in Figure 8, using 6 lines to represent 6 classifiers, where line 11 separates neutral expressions from laughing expressions; line 12 separates angry and laughing expressions; line 13 separates surprised and laughing expressions Line 21 separates neutral and angry expressions; line 22 separates neutral and surprised expressions; line 23 separates surprised and angry expressions.

At last, referring to Fig. 1, after obtaining the SVM classifier, the present invention can carry out real-time facial expression recognition, in the implementation process of the present invention, at first each frame in the video is carried out face detection, then face is carried out Track and extract the position of both eyes; if the tracking is successful, perform facial expression recognition on the face in the current image, and give the recognition result in real time; at the same time, refer to the content shown in Figure 9, the left side is the video data input by the USB camera , the small window on the right is the result of facial expression recognition.

The method of the present invention can be applied to a video-based facial expression recognition device, as shown in Figure 10, the device includes a video data acquisition unit 1, an image processing unit 2, a face information database 3 and a facial expression recognition unit 4; Video data acquisition unit 1 collects the face image of video and sends it to image processing unit 2; The image contrast of the two, and adopt AdaBoost calculation unit 123 to carry out calculation to face data and send to described facial expression recognition unit 4; The image is recognized. The device also includes a display unit 5 for displaying the recognized facial expressions.

Wherein said image processing unit 2 comprises a comparison unit 121, a feature generation unit 122, a calculation unit 123 and a classifier unit 124; Contrast, detect the face and eyes, and extract the face image according to the position of the eyes, and send the face image information to the feature generation unit 122; the feature generation unit 122 locates the facial organs, generates Eigenface, the eigenface is sent to the calculation unit 123 as a sample; the calculation unit 123 calculates the Gabor feature of the eigenface image, and adopts the AdaBoost algorithm to select the feature, and then the selected feature is sent to the classifier unit 124; the described The classifier unit 124 constructs a support vector machine classifier according to the selected features, and sends the classifier information to the facial expression recognition unit 4 . The video data collection unit 1 also includes a video data tracking unit 111. The video data tracking unit 111 tracks and detects the face data of the video data, judges whether to collect, and executes the face tracking step in the method of the present invention.

The method of the present invention can automatically extract the facial expression of accurate people's face in video situation, and this method has adopted Adaboost and ASM algorithm, can eliminate the influence of illumination, has carried out special processing to people's face image in the method, makes The gray mean value and variance of the left and right parts of the human face are basically the same, and the method of the present invention develops a real-time automatic detection and tracking of the human face for the video data of the commonly used USB camera and can recognize the common expressions of the frontal human face. Algorithms can achieve better commercial results.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention within.

Claims (14)

1, a kind of facial expression recognition method based on video, it is characterized in that, comprising the following steps: Collect the facial expression image data of the face from the video data input by the USB camera, and preprocess the image data; Real-time extraction of the position of the face in the preprocessed image; Positioning the human eyes in the human faces in the determined image according to the human eye classifier; Extract the image area containing the face according to the determined position of the human eyes and the information of the face classifier, and perform normalization processing; Positioning of facial organs; Determine the position of the chin of the face according to the positioning of the facial organs, determine the face area in the image, generate the eigenface, and use it as a classification sample; Calculating the Gabor feature of the eigenface image based on the classification sample; Select the calculated Gabor features; Construct a support vector machine classifier from the selected features; According to the constructed classifier, the result of facial expression recognition is obtained. 2, the video-based facial expression recognition method as claimed in claim 1 is characterized in that, when collecting from the data of USB camera input, comprise following face tracking steps: Before obtaining the tracking target, search each frame of image to detect whether there is a face; If one or more faces are detected in a frame of images, the detected faces are tracked in the next two frames of images, and the faces tracked in these two frames of images are detected and verified, and a judgment is made on the detection results ; After the face is detected in the three frames of images at the same position, the algorithm considers that there is a face at that position. At this time, the real-time face detection algorithm is executed to extract the position of the face in the image; If multiple faces are detected in the scene, select one of the faces to start tracking, and continue to track the face in subsequent frames, if the similarity between the tracking results of the next frame and the previous frame in adjacent frames is too low , or if no upright face is detected for a long time in the area where a tracking target is located, the tracking will stop. 3. The facial expression recognition method based on video as claimed in claim 1, wherein said normalization process is realized by a resampling algorithm: said resampling algorithm is scaling, rotation and translation transformation, so that the detected The position of the human eye overlaps with the position of the human eye classifier for localization. 4. The facial expression recognition method in the case of video as claimed in claim 1, characterized in that, the described positioning of human face organs adopts a target extraction method, and the target extraction method is an active shape model algorithm. 5. The video-based facial expression recognition method according to claim 4, wherein the specific steps of the active shape model algorithm are: Extract the contour information of the face from the video data, and establish a sample unit; Normalize and align the samples in the sample unit, and then perform principal component analysis transformation; The gray level information of each control point in the transformed contour information is analyzed for the principal component, as the basis for point search; The average contour calculated by principal component analysis is used as the initial value of the contour search, and the iterative search is performed to obtain the face contour. 6. The video-based facial expression recognition method as claimed in claim 5, wherein the iterative search step is: Obtain the initial translation value according to the grayscale information, align the new contour obtained according to the grayscale search to the average contour, and calculate the aligned parameter value; Calculate the change in shape based on the aligned data and statistics calculated by principal component analysis; Invert the changed shape to the position of the new contour according to the aligned parameter value to obtain the result of a search; Repeat the above search steps, and continue to iterate until the face contour is converged. 7. The facial expression recognition method based on video as claimed in claim 1, wherein the generation of said eigenface is to compare the extracted human face contour with the human face in the human face classifier, and adjust the gradient . 8. The facial expression recognition method based on video as claimed in claim 1, characterized in that, between generating the eigenface and calculating the Gabor feature of the eigenface image, there is also a step of processing the generated eigenface: forming The left and right parts of the eigenface are normalized to the gray level, so that the gray mean and variance of the left and right parts are the same. 9. The video-based facial expression recognition method according to claim 8, characterized in that a gray filter band is set between the left and right parts of the eigenface. 10. The video-based facial expression recognition method according to claim 1, characterized in that the established support vector machine classifier is a multi-class classifier, which is one-to-one, one-to-many or decision tree. 11. A video-based facial expression recognition device, characterized in that: Including video data acquisition unit, image processing unit, face information database and facial expression recognition unit; The video data collection unit collects the face image of the video and transmits it to the image processing unit; The image processing unit calls the face information from the face information database and compares it with the collected face image, then calculates the face data, and transmits the calculated data to the facial expression recognition unit; The facial expression recognition unit recognizes the collected face images according to the recognition information stored in the face information database. 12. The video-based facial expression recognition device according to claim 11, further comprising a display unit for displaying the recognized facial expressions. 13. The video-based facial expression recognition device according to claim 11, wherein said image processing unit includes a comparison unit, a feature generation unit, a calculation unit and a classifier unit; The comparison unit compares the image information of the face with the image information in the face database, detects the face and eyes, extracts the face image according to the position of the eyes, and transmits the face image information to the feature generation unit; The feature generation unit locates the facial organs, generates the eigenface according to the chin of the face, and sends the eigenface as a sample to the calculation unit; The computing unit calculates the Gabor feature of the eigenface image, and adopts the AdaBoost algorithm to select the feature, and then transmits the selected feature to the classifier unit; The classifier unit constructs a support vector machine classifier according to the selected features, and transmits the classifier information to the facial expression recognition unit. 14. The facial expression recognition device based on video as claimed in claim 11, wherein said video data acquisition unit also includes a video data tracking unit, and the video data tracking unit performs face data on video data. Track and detect to determine whether to collect the input data.

Images