TW201909036A - Multi-dimensional emotion discrimination system and method for face image based on neural network - Google Patents

Abstract

The present invention provides a system and method for multi-dimensional emotion discrimination of a face image based on a neural network. The system comprises: a face location module configured to identify a face region in an image to be detected and extract a face image in the image to be detected; a feature extraction module configured to extract emotion features of the face image; an identification module configured to identify the emotion features to obtain emotion information; and an output module configured to output the emotion information. The system and method for multi-dimensional emotion discrimination of the face image based on the neural network can adapt different face angles, skin colors and face shapes to extract emotion feature vectors and represent the multi-dimensional emotion to which the face image belongs so as to improve the accuracy of face emotion analysis.

Description

Multi-dimensional emotion discrimination system and method for face image based on neural network

The invention belongs to the field of image processing technology, and particularly relates to a multi-dimensional emotion discrimination system and method for a face image based on a neural network.

People use expressions, gestures, body and language to convey messages and communication. Recognizing facial emotions is one of the most direct ways to understand the message that humans convey. The traditional face emotion recognition field mainly includes the following aspects:

Emotion recognition for keypoint detection: Identify the area where the face is located, use traditional algorithms to locate facial features and contours of keypoints, and extract keypoint features as features for emotion recognition. The method is limited by the accuracy of keypoint positioning, and only Face contour, lack of changes in facial muscles, this way is too general, and emotions are difficult to identify accurately.

On the other hand, the classification of emotion recognition: it is difficult to explain discrete changes in human emotions. For example, anger and sadness are not separated by a line. Human emotions are mixed and continuous. If one face uses only one type, Emotions indicate that this approach is too general and cannot describe people's delicate emotions carefully.

The present invention addresses the shortcomings in the prior art. The present invention provides a multi-dimensional emotion discrimination system and method for a face image based on a neural network to improve the accuracy of facial emotion analysis.

A facial image multi-dimensional emotion discrimination system based on a neural network of the present invention includes a face positioning module, a feature extraction module, a recognition module, and an output module. The face positioning module is used to identify a face region in an image to be detected, and a face detection algorithm is used to extract a face image in the image to be detected; the feature extraction module is used to extract the face The emotional characteristics of the image; the recognition module is used to identify the emotional characteristics to obtain emotional information; the output module is used to output the emotional information.

In one embodiment of the present invention, the feature extraction module described above extracts the emotional features of the face image through a convolutional neural network.

In one embodiment of the present invention, the training of the above-mentioned convolutional neural network includes: training using training data describing a facial image through a multi-dimensional emotion vector; the feature extraction module input is the face image Like, the output is a multi-dimensional emotion vector.

In an embodiment of the present invention, the output of the feature extraction module and the output of the recognition module are both multi-dimensional vectors, and the multi-dimensional vectors include multiple emotion categories.

In one embodiment of the present invention, the aforementioned emotion categories include anger, disgust, fear, happiness, sadness, surprise, or neutrality.

The present invention further provides a neural network-based multi-dimensional emotion discrimination method for a face image, which is applicable to the above-mentioned neural network-based multi-dimensional emotion discrimination system. The steps include: identifying a face region in an image to be detected, The face detection algorithm is used to extract the face image in the image to be detected; then the emotional features of the face image are extracted; the emotional features are identified to obtain the emotional information; and the emotional information is output.

In one embodiment of the present invention, the aforementioned emotional information is a multi-dimensional vector, and the multi-dimensional vector includes a plurality of emotion categories.

In one embodiment of the present invention, the aforementioned emotion categories include anger, disgust, fear, happiness, sadness, surprise, or neutrality.

It can be known from the above technical solution that the neural network-based facial image multi-dimensional emotion discrimination system and method provided by the present invention adapt to different facial angles, skin colors, and face shapes to extract emotional feature vectors and reflect the belonging of the facial image. Multi-dimensional emotion, improve the accuracy of facial emotion analysis.

In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced below. Throughout the drawings, similar elements or portions are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn to actual scale.

The embodiments of the technical solution of the present invention will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to explain the technical solution of the present invention more clearly, and therefore are only examples, and cannot be used to limit the protection scope of the present invention. It should be noted that, unless otherwise stated, the technical terms or scientific terms used in this application shall have the ordinary meanings understood by those skilled in the art to which the present invention belongs.

A multi-dimensional emotion discrimination system for a face image based on a neural network, as shown in FIGS. 1 and 2, includes a face positioning module 110 for identifying a face region in an image to be detected 200 and extracting a to-be-detected image. Face image 210 in image 200; the method of face positioning is to extract a face image by performing a face detection algorithm (not limited to a machine learning method) in an image, a video, or an image to be detected 200 Like 210. A feature extraction module 120 is used to extract the emotional features 220 of the face image 210; according to the input face image 210, a deep learning model is obtained by training a neural network, and the deep learning model is used at the last layer during the test A feature vector corresponding to the face image 210 is extracted from the feature layer. The recognition module 130 is configured to identify the emotional feature 220 to obtain the emotional information 230; according to the feature vector described by each input face image 210, a multi-dimensional emotion discrimination classifier (which can simultaneously describe the possibility of different emotions occurring) The classifier has a probability output value of 0 to 1 for each category) to obtain emotional information 230. The output module 140 is configured to output the emotional information 230.

In this embodiment, the face positioning module is configured to obtain positions of all human face areas in one or more images to be detected, and obtain all human face images in the images to be detected. The feature extraction module uses a convolutional neural network to extract the emotional features of a face image. The recognition module is used for judging the emotional characteristics of the human face to be tested for emotional information. The system can adapt to different face angles, skin tones and face shapes to extract emotional feature vectors, reflecting the multi-dimensional emotions of the face image.

In this embodiment, there are two main problems to be solved: First, the traditional face image processing technology uses only the key points of the face as features, and the effect is not good. Second, the problem of inaccurate results in describing faces with only one emotion category.

In this embodiment, the input of the system is the entire face image, and its output is a judgment of multi-dimensional emotion. The input of the entire image can take into account the subtle changes in facial expression muscles to achieve high-precision emotional feature extraction. Multi-dimensional emotional description to accurately describe the emotional response of the human face.

In this embodiment, the feature extraction module extracts the emotional features of the face image through a convolutional neural network.

In this embodiment, the training of the convolutional neural network includes: training using training data describing a facial image through a multi-dimensional emotion vector; the feature extraction module input is the face image, and the output is Multi-dimensional emotions vector. The convolutional neural network of the feature extraction module includes stacking and residual convolution and pooling layers, which makes extracting facial emotional features more robust and able to adapt to and learn different facial angles, skin colors, and face shapes. When training the convolutional neural network, the facial image of the training data is described by a multi-dimensional emotion vector. The feature extraction module takes a facial image as an input and extracts high-dimensional abstract features from stacked convolutional layers.

In this embodiment, the recognition module uses a target function to train its neural network, so that the recognition module can achieve the effect of describing a multi-dimensional emotion vector. Due to the design of the neural network of the feature extraction module and the recognition module, the present invention can provide accurate emotion vector analysis and prediction in highly variable face images.

In this embodiment, the output of the feature extraction module and the output of the recognition module are both multi-dimensional vectors, and the multi-dimensional vectors include multiple emotion categories. This emotion category includes angry, disgusted, scared, happy, sad, surprised or neutral. The types of emotions that this invention can handle are not limited to the previous ones.

In this embodiment, a neural network-based multi-dimensional emotion discrimination method for a facial image is applicable to the above-mentioned neural network-based multi-dimensional emotion discrimination system, and includes: identifying a face region in an image to be detected, and Extracting a face image from an image to be detected; extracting the emotional characteristics of the face image; identifying the emotional characteristics to obtain emotional information; and outputting the emotional information.

In this embodiment, the method can make robust predictions for images of various faces regardless of the angle and light source of the face. And it can output multi-dimensional emotional results, accurately describe the emotional changes of the face, and can effectively solve the problem of facial emotion recognition.

In this embodiment, the emotion information is a multi-dimensional vector, and the multi-dimensional vector includes a plurality of emotion categories.

In this embodiment, the emotion category includes angry, disgusted, scared, happy, sad, surprised or neutral.

In this embodiment, the following applications are further included:

Application scenario 1: Advertising observation and feedback, with the screen as the carrier, can be placed in public places or personal computers. When customers watch the advertisement, they analyze the content of the advertisement and the customer's facial emotional response, understand the attraction and interest of the masses to the advertisement, and predict the product response or adjust the advertisement content.

Application Scenario 2: Shopping malls, with cameras as carriers, are placed on shelves to observe the emotional response of customers when selecting items. You can adjust the placement of items on the shelves and understand the products that customers love, to do sales analysis and make more sales. Strategy.

Application Scenario 3: Mobile APP, using mobile APP as a carrier, can analyze user's emotions when watching videos or social software, and use smart chat robots to care about the mood of users, and improve the stickiness between smart chat robots and users.

In summary, the neural network-based facial image multi-dimensional emotion discrimination system and method provided by the present invention adapt to different facial angles, skin colors, and face shapes to extract emotional feature vectors, reflecting the multi-dimensional belonging of the facial image. Emotion, improve the accuracy of facial emotion analysis.

Although the present invention is disclosed in the foregoing embodiment as above, it is not intended to limit the present invention. Any person skilled in the art of similarity, without departing from the spirit and scope of the present invention, makes equivalent substitutions for modifications and retouching. Within the scope of patent protection.

110‧‧‧Face positioning module

120‧‧‧Feature Extraction Module

130‧‧‧Identification Module

140‧‧‧output module

200‧‧‧Image to be detected

210‧‧‧face image

220‧‧‧ Emotional characteristics

230‧‧‧ Emotional Information

[Fig. 1] A structural block diagram of a facial image multi-dimensional emotion discrimination system based on a neural network. [Figure 2] An image processing method in a multi-dimensional emotion discrimination method of a face image based on a neural network.

Claims (8)

A multi-dimensional emotion discrimination system for a face image based on a neural network, comprising: a face positioning module for identifying a face region in an image to be detected; and using a face detection algorithm to extract the to be detected A face image in the image; a feature extraction module for extracting emotional characteristics of the face image; a recognition module for identifying the emotional characteristics to obtain emotional information; and an output module for Output the emotional information. According to the neural network-based facial image multi-dimensional emotion discrimination system according to item 1 of the scope of the patent application, the feature extraction module extracts the emotional features of the face image through a convolutional neural network. As described in item 2 of the scope of the patent application, a neural network-based multi-dimensional emotion discrimination system for a face image, wherein the training of the convolutional neural network includes: training that describes a facial image through a multi-dimensional emotion vector The data is used for training; the input of the feature extraction module is the face image, and the output is a multi-dimensional emotion vector. According to the neural network-based facial image multi-dimensional emotion discrimination system according to item 1 of the scope of patent application, the output of the feature extraction module and the output of the recognition module are both multi-dimensional vectors, and the multi-dimensional vector includes multiple Emotion categories. The neural network-based facial image multi-dimensional emotion discrimination system according to item 4 of the scope of the patent application, wherein the emotion category includes angry, disgusted, scared, happy, sad, surprised or neutral. A multi-dimensional emotion discrimination method for a facial image based on a neural network, which is applicable to the first scope of the patent application. The multi-dimensional emotion discrimination system for a facial image based on a neural network includes: identifying a face region in an image to be detected, A face detection algorithm is used to extract a face image in the image to be detected; extract the emotional characteristics of the face image; identify the emotional characteristics to obtain emotional information; and output the emotional information. As described in item 6 of the scope of the patent application, a neural network-based multi-dimensional emotion discrimination method for a face image, wherein the emotion information is a multi-dimensional vector, and the multi-dimensional vector includes a plurality of emotion categories. As described in item 7 of the scope of the patent application, a neural network-based multi-dimensional emotion discrimination method for a facial image, wherein the emotion category includes anger, disgust, fear, happy, sadness, surprise, or neutrality.