CN111816211B - Emotion recognition method and device, storage medium and electronic equipment - Google Patents

Abstract

The embodiment of the present application discloses an emotion recognition method, device, storage medium, and electronic equipment, wherein the electronic equipment can obtain the text content input by the user, and perform emotion recognition according to the text content and the pre-trained first emotion recognition model, and obtain The user's first candidate emotion, and then obtain the voice content during the user's input text content, and perform emotion recognition based on the voice content and the pre-trained second emotion recognition model to obtain the user's second candidate emotion, and finally according to the first candidate emotion and The second candidate emotion identifies the user's target emotion. Therefore, the embodiment of the present application recognizes the user's emotion based on different information sources, and finally determines the user's target emotion by combining the emotion recognition results obtained from different information sources, so as to realize accurate recognition of the user's emotion.

Description

Emotion recognition method, device, storage medium and electronic equipment

technical field

The present application relates to the technical field of data processing, in particular to an emotion recognition method, device, storage medium and electronic equipment.

Background technique

As a group with extremely strong emotional factors, human beings have emotions such as joy, anger, worry, thought, sadness, fear and shock. The electronic device can provide intelligent services to the user by recognizing the user's emotion, for example, "tell" a joke to the user when it is recognized that the user is unhappy. However, in related technologies, the emotion recognition method based on the emotion dictionary is usually used to identify the user's emotion, for example, to identify the emotion words in the text input by the user, and then match the corresponding emotion according to the emotion dictionary, but this emotion recognition method does not Inaccurate.

Contents of the invention

Embodiments of the present application provide an emotion recognition method, device, storage medium, and electronic equipment, capable of accurately recognizing user emotions.

In the first aspect, an embodiment of the present application provides an emotion recognition method applied to an electronic device, and the emotion recognition method includes:

Obtaining the text content input by the user, and performing emotion recognition according to the text content and the pre-trained first emotion recognition model, to obtain the first candidate emotion of the user;

Acquiring the voice content during which the user inputs the text content, and performing emotion recognition according to the voice content and the pre-trained second emotion recognition model, to obtain the second candidate emotion of the user;

The user's target emotion is determined according to the first candidate emotion and the second candidate emotion.

In the second aspect, the embodiment of the present application provides an emotion recognition device, which is applied to electronic equipment, and the emotion recognition device includes:

The first emotion recognition module is used to obtain the text content input by the user, and perform emotion recognition according to the text content and the pre-trained first emotion recognition model to obtain the first candidate emotion of the user;

The second emotion recognition module is used to obtain the voice content during the period when the user inputs the text content, and perform emotion recognition according to the voice content and the pre-trained second emotion recognition model to obtain the second candidate emotion of the user;

A target emotion recognition module, configured to determine the user's target emotion according to the first candidate emotion and the second candidate emotion.

In a third aspect, the embodiment of the present application provides a storage medium on which a computer program is stored, and when the computer program is run on a computer, the computer is made to perform the steps in the emotion recognition method provided in the embodiment of the present application. step.

In a fourth aspect, the embodiment of the present application provides an electronic device, including a processor and a memory, the memory has a computer program, and the processor is used to execute the emotion provided by the embodiment of the present application by calling the computer program. Identify the steps in the method.

In the embodiment of the present application, the electronic device can obtain the text content input by the user, perform emotion recognition according to the text content and the pre-trained first emotion recognition model, obtain the user's first candidate emotion, and then obtain the voice during the user input text content content, and perform emotion recognition according to the voice content and the pre-trained second emotion recognition model to obtain the user's second candidate emotion, and finally determine the user's target emotion according to the first candidate emotion and the second candidate emotion. Therefore, the embodiment of the present application recognizes the user's emotion based on different information sources, and finally determines the user's target emotion by combining the emotion recognition results obtained from different information sources, so as to realize accurate recognition of the user's emotion.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a schematic structural diagram of a panoramic perception architecture provided by an embodiment of the present application.

Fig. 2 is a schematic flowchart of an emotion recognition method provided by an embodiment of the present application.

Fig. 3 is another schematic flowchart of the emotion recognition method provided by the embodiment of the present application.

Fig. 4 is a schematic diagram of an application scenario of an emotion recognition method provided by an embodiment of the present application.

Fig. 5 is a schematic structural diagram of an emotion recognition device provided by an embodiment of the present application.

FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

FIG. 7 is another schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

Referring to the drawings, where the same reference numerals represent the same components, the principles of the present application are exemplified by implementation in a suitable computing environment. The following description is based on illustrated specific embodiments of the present application, which should not be construed as limiting other specific embodiments of the present application that are not described in detail here.

With the miniaturization and intelligence of sensors, electronic devices such as mobile phones and tablet computers integrate more and more sensors, such as light sensors, distance sensors, position sensors, acceleration sensors, and gravity sensors, etc. Electronic equipment can collect more data with less power consumption through its configured sensors. At the same time, the electronic device will also collect data related to its own state and data related to the user state during operation. Generally speaking, electronic devices can obtain data related to the external environment (such as temperature, light, location, sound, weather, etc.), data related to user status (such as posture, speed, mobile phone usage habits, basic personal information, etc.) Data related to device status (such as power consumption, resource usage status, network status, etc.).

In the embodiment of the present application, in order to process the data acquired by the electronic device and provide intelligent services to users, a panoramic perception architecture is proposed. Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of the panoramic perception architecture provided by the embodiment of the present application, which is applied to electronic devices and includes a bottom-up information perception layer, a data processing layer, a feature extraction layer, a scene modeling layer, and an intelligent service layer.

As the bottom layer of the panoramic perception architecture, the information perception layer is used to obtain raw data that can describe various types of scenarios of users, including dynamic data and static data. Among them, the information perception layer is composed of multiple sensors for data collection, including but not limited to the distance sensor used to detect the distance between the electronic device and external objects, the magnetic field information used to detect the environment where the electronic device is located The magnetic field sensor, the light sensor used to detect the light information of the environment where the electronic device is located, the acceleration sensor used to detect the acceleration data of the electronic device, the fingerprint sensor used to collect the user's fingerprint information, the Hall sensor used to sense the magnetic field information Sensors, position sensors for detecting the current geographic location of electronic devices, gyroscopes for detecting angular velocity of electronic devices in various directions, inertial sensors for detecting motion data of electronic devices, and sensing the attitude of electronic devices A posture sensor for information, a barometer for detecting the air pressure in the environment where the electronic device is located, and a heart rate sensor for detecting the user's heart rate information, etc.

As the sub-bottom layer of the panoramic perception architecture, the data processing layer is used to process the original data obtained by the information perception layer, and eliminate the noise and inconsistency of the original data. Among them, the data processing layer can perform data cleaning, data integration, data transformation, data reduction and other processing on the data obtained by the information perception layer.

As the middle layer of the panoramic perception architecture, the feature extraction layer is used to perform feature extraction on the data processed by the data processing layer, so as to extract the features contained in the data. Among them, the feature extraction layer can extract features or process the extracted features through methods such as filtering, packaging, and integration.

The filtering method refers to filtering the extracted features to delete redundant feature data. The wrapping method is used to filter the extracted features. The integration method refers to the integration of multiple feature extraction methods to construct a more efficient and accurate feature extraction method for feature extraction.

As the second level of the panoramic perception architecture, the scene modeling layer is used to build a model based on the features extracted by the feature extraction layer, and the obtained model can be used to represent the state of the electronic device, the user state, or the environment state. For example, the scenario modeling layer can construct a key value model, a pattern identification model, a graph model, an entity relationship model, an object-oriented model, etc. according to the features extracted by the feature extraction layer.

As the highest layer of the panoramic perception architecture, the intelligent service layer is used to provide intelligent services based on the model constructed by the scenario modeling layer. For example, the intelligent service layer can provide users with basic application services, provide intelligent system optimization services for electronic devices, and provide users with personalized intelligent services.

In addition, the panoramic perception architecture also includes an algorithm library, including but not limited to the illustrated Markov algorithm, hidden Dirichlet distribution algorithm, Bayesian classification algorithm, support vector machine, K-means clustering algorithm , K nearest neighbor algorithm, conditional random field, residual network, long short-term memory network, convolutional neural network and recurrent neural network and other algorithms.

Based on the panoramic perception framework provided by the embodiment of the present application, the embodiment of the present application provides an emotion recognition method. The execution subject of the emotion recognition method may be the emotion recognition device provided in the embodiment of the present application, or an electronic equipment, wherein the emotion recognition device can be implemented in hardware or software. Wherein, the electronic device may be a smart phone, a tablet computer, a palmtop computer, a notebook computer, or a desktop computer, etc., which are equipped with a processor and have processing capabilities.

Based on the activity prediction method provided by the embodiment of the present application, the panoramic data can be obtained at the information perception layer and provided to the data processing layer; at the data processing layer, the text content input by the user can be screened out from the panoramic data, and during the user input text content The voice content of the audio content is provided to the feature extraction layer; in the feature extraction layer, the feature extraction of the aforementioned text content and voice content is performed respectively, and the feature vector corresponding to the text content and the feature vector corresponding to the voice content are obtained, which are provided to the scene modeling layer; The modeling layer performs emotion recognition on the feature vector corresponding to the text content and the feature vector corresponding to the sound content, and obtains the first candidate emotion corresponding to the text content and the second candidate emotion corresponding to the sound content, and then according to the first candidate emotion and the second candidate emotion Candidate emotions determine the user's target emotion and provide it to the intelligent service layer; in the intelligent service layer, corresponding operations are performed according to the user's target emotion, such as sending the aforementioned text content and target emotion to the corresponding target device, so that other users can view user input In addition to the text content, it is also possible to know the emotions of the user when entering the text content, which is helpful for better communication.

Please refer to FIG. 2 . FIG. 2 is a schematic flowchart of an emotion recognition method provided in an embodiment of the present application. As shown in Figure 2, the flow of the emotion recognition method provided in the embodiment of the present application may be as follows:

In 101, the text content input by the user is obtained, and emotion recognition is performed according to the text content and a pre-trained first emotion recognition model to obtain the user's first candidate emotion.

In the embodiment of the present application, the electronic device detects the user input, and triggers the user's emotion recognition when detecting the user input text content. Wherein, the text content includes but not limited to words, sentences and articles. For example, when a user is having a conversation with other users through an instant messaging application installed on the electronic device, when the electronic device detects chat text content input by the user, it will trigger emotion recognition on the user.

When triggering the user's emotion recognition, the electronic device first acquires the text content input by the user, so as to perform preliminary recognition of the user's emotion according to the text content. It should be noted that, in the embodiment of the present application, a first emotion recognition model for recognizing user emotions according to text content input by the user is pre-trained. For example, the initial convolutional neural network model is established in advance, and the text content samples are obtained and the emotions corresponding to the text content samples are calibrated to obtain the corresponding emotional labels. The network is trained, and the trained convolutional neural network is used as the first emotion recognition model for emotion recognition of the user's text content. Wherein, the first emotion recognition model may be stored locally in the electronic device, or may be stored in a remote server. In this way, after the electronic device obtains the text content input by the user, it further acquires locally the first emotion recognition model for performing emotion recognition on the text content input by the user, or acquires the first emotion recognition model for the user input from the remote server. The first emotion recognition model for emotion recognition of text content. After acquiring the first emotion recognition model, the electronic device uses the first emotion recognition model to perform emotion recognition on the text content input by the user, and records the recognized emotion as the first candidate emotion of the user.

In 102, the voice content during the user inputting the text content is acquired, and emotion recognition is performed according to the voice content and the pre-trained second emotion recognition model to obtain the user's second candidate emotion.

It should be noted that in the embodiment of the present application, the electronic device not only performs emotion recognition according to the text input by the user, but also performs emotion recognition according to the user's voice. Correspondingly, in the embodiment of the present application, a second emotion recognition model for recognizing the user's emotion according to the user's voice content is also pre-trained. For example, the initial convolutional neural network model is established in advance, and the sound content samples are obtained and the emotions corresponding to the sound content samples are calibrated to obtain the corresponding emotional labels, and then the initial convolutional neural network is The network performs supervised training, and the trained convolutional neural network is used as a second emotion recognition model for emotion recognition of the user's voice content. Wherein, the second emotion recognition model may be stored locally in the electronic device, or may be stored in a remote server. In this way, after the electronic device acquires the voice content during the user input text content, it further acquires locally the second emotion recognition model for emotional recognition of the user's voice content, or acquires the second emotion recognition model from the remote server for user The second emotion recognition model for emotion recognition of voice content. After acquiring the second emotion recognition model, the electronic device uses the second emotion recognition model to perform emotion recognition on the voice content during the user inputting text content, and records the recognized emotion as the user's second candidate emotion.

Wherein, when the electronic device detects the user inputting the text content, it activates the built-in microphone or the external microphone to collect the sound, so as to collect the sound content during the user inputting the aforementioned text content. In this way, when the electronic device performs emotion recognition according to the user's voice, it can directly acquire the previously collected voice content during the user input text content.

In 103, the user's target emotion is determined according to the first candidate emotion and the second candidate emotion.

According to the above description, those skilled in the art can understand that the first candidate emotion and the second candidate emotion obtained above are obtained through independent information sources. Therefore, in order to ensure the accuracy of user emotion recognition, the electronic device also uses The first candidate emotion and the second candidate emotion are comprehensively analyzed to finally determine the user's target emotion.

As can be seen from the above, in the embodiment of the present application, the electronic device can obtain the text content input by the user, and perform emotion recognition according to the text content and the pre-trained first emotion recognition model, obtain the user's first candidate emotion, and then obtain the user input text The voice content during the content period, and perform emotion recognition according to the voice content and the pre-trained second emotion recognition model to obtain the user's second candidate emotion, and finally determine the user's target emotion according to the first candidate emotion and the second candidate emotion. Therefore, the embodiment of the present application recognizes the user's emotion based on different information sources, and finally determines the user's target emotion by combining the emotion recognition results obtained from different information sources, so as to realize accurate recognition of the user's emotion.

In an embodiment, "determining the user's target emotion according to the first candidate emotion and the second candidate emotion" includes:

The first candidate emotion and the second candidate emotion are input into a pre-trained Bayesian classifier for classification, and the user's target emotion output by the Bayesian classifier is obtained.

Since the first candidate emotion and the second candidate emotion obtained above are obtained through independent information sources, in order to combine the first candidate emotion and the second candidate emotion to obtain the user's target emotion, the embodiment of the present application also pre-trains the A Bayesian classifier for quadratic sentiment classification of candidate sentiments from text sources as well as candidate sentiments from sound sources. For example, it is possible to obtain the emotional samples of the text source and the corresponding emotional samples of the sound source, carry out emotional calibration on the emotional samples of the text source and the corresponding emotional text of the sound source, obtain the corresponding emotional labels, and then use the emotional samples of the text source to and the emotional text of the corresponding sound source, and the corresponding emotional label to obtain a Bayesian classifier.

Wherein, the trained Bayesian classifier can be stored locally in the electronic device, or can be stored in a remote server. In this way, when the electronic device determines the user's target emotion according to the first candidate emotion and the second candidate emotion, it further obtains the Bayesian classifier locally, or obtains the Bayesian classifier from a remote server, so that the previously acquired The first candidate emotion from the text source and the second candidate emotion from the sound source are input into the Bayesian classifier for classification, and the emotion output by the Bayesian classifier is used as the target emotion finally obtained by performing emotion recognition on the user.

In one embodiment, "performing emotion recognition based on the text content and the pre-trained first emotion recognition model to obtain the user's first candidate emotion" includes:

(1) Perform feature extraction on the text content to obtain the corresponding feature vector;

(2) Convert the feature vector into a corresponding feature tensor, and input the feature tensor into the first emotion recognition model for emotion recognition, and obtain the user's first candidate emotion output by the first emotion recognition model.

In the embodiment of the present application, when the electronic device performs emotion recognition according to the text content and the pre-trained first emotion recognition model, it does not directly input the original text content into the first emotion recognition model for prediction, but the original After the text content is processed, features that can represent the original text content are input into the first emotion recognition model for emotion recognition.

Wherein, the electronic device first uses a preset feature extraction technology to perform feature extraction on the text content input by the user, and converts the text content into a corresponding vector, which is recorded as a feature vector. Then, the electronic device further combines the feature vectors corresponding to the text content into a tensor, which is denoted as a feature tensor.

It should be noted that, like vectors and matrices, tensor is also a data structure, but tensor is a three-dimensional and above data structure, where the dimension of data is called the order of tensor, and tensor can be regarded as The generalization of vectors and matrices in multi-dimensional space regards vectors as first-order tensors and matrices as second-order tensors.

Correspondingly, when the first emotion recognition model is trained, it is not trained according to the original text content sample, but after obtaining the corresponding feature tensor of the text content sample in the same way, using its feature tensor and the calibrated emotion labels for training. In this way, after the feature vector is converted into a corresponding feature tensor, the feature tensor can be input into the first emotion recognition model for emotion recognition, and the first candidate emotion of the user output by the first emotion recognition model can be obtained.

In one embodiment, "extracting features from text content to obtain corresponding feature vectors" includes:

Extract the keywords included in the text content, and map the keywords to the vector space through the word embedding model to obtain the feature vector.

Those of ordinary skill in the art can understand that not all content in the text content input by the user is meaningful, and if feature extraction is performed on the complete text content, the overall efficiency of user emotion recognition will be affected. Therefore, in the embodiment of the present application, when the electronic device extracts the features of the text content input by the user, it first uses the preset keyword extraction algorithm to extract the keywords in the text content, and uses the extracted keywords to represent the complete Text content, reduce the content that needs to be extracted, and achieve the purpose of improving the efficiency of emotion recognition. The embodiment of the present application does not specifically limit which keyword extraction algorithm is used to extract keywords from the text content, and a suitable keyword extraction algorithm can be selected by those skilled in the art according to actual needs. For example, in this embodiment of the application, the electronic device can use the ID-TIF algorithm to extract keywords from the text content input by the user. Assume that the text content input by the user is the sentence "I miss you very much today", and use the ID-TIF algorithm to extract keywords from the text content input by the user. Key words are extracted, and the key words are "today" and "missing you".

After the electronic device extracts keywords that can represent the text content, it further maps the keywords extracted from the text content to a vector space through a word embedding model to obtain a feature vector corresponding to the text content. Among them, word embedding models include but are not limited to Word2vec model, GloVe model, FastText model, and ELMo model.

In one embodiment, before "inputting the feature tensor into the first emotion recognition model for emotion recognition", it also includes:

Zero pad the feature tensor.

Those of ordinary skill in the art can understand that, when a user inputs text content to an electronic device, since the length of each input text content is different, the data amount of the feature tensor obtained for each text content input by the user is also are different, and cannot be aligned within the feature tensor. Therefore, in the embodiment of the present application, before inputting the feature tensor into the first emotion recognition model for emotion recognition, the electronic device also performs zero-fill processing on the feature tensor, so that the feature tensor is internally aligned, and its data volume reaches the preset The amount of data.

Correspondingly, during the training of the first emotion recognition model, the feature tensors corresponding to the text content samples are also aligned and zero-filled so that they are internally aligned and the data volume reaches the preset data volume.

In one embodiment, "performing emotion recognition based on the voice content and the pre-trained second emotion recognition model to obtain the user's second candidate emotion" includes:

(1) dividing the sound content into a plurality of sub-sound contents;

(2) Input the content of a plurality of sub-sounds into the second emotion recognition model to carry out emotion recognition respectively, and obtain a plurality of corresponding candidate emotions;

(3) Determine the second candidate emotion of the user according to the plurality of candidate emotions.

In the embodiment of the present application, when the electronic device performs emotion recognition based on the voice content and the pre-trained second emotion recognition model to obtain the user's second candidate emotion, it first divides the voice content during the user's input of the aforementioned text content, and divides it into Divided into multiple sub-sound contents of the same length. Wherein, when dividing the sound content, two adjacent sub-sound contents may or may not have the same sound part.

After the complete sound content is divided into multiple sub-sound contents, the electronic device respectively converts the multiple sub-sound contents into corresponding spectrograms, and uses the spectrogram to represent the corresponding sub-sound contents, for example, fast Fourier transform or Convert the spectrogram by means of Mel-frequency cepstral coefficients.

Correspondingly, when the second emotion recognition model is trained, it is not trained according to the original sound content samples, but after the sound content samples are converted into corresponding spectrograms, the corresponding spectrograms and calibrated emotional labels are used for training . In this way, after the electronic device converts a plurality of sub-sound contents obtained by division into corresponding spectrograms, respectively input the spectrograms corresponding to each sub-sound content into the second emotion recognition model for emotion recognition, and obtain the corresponding output of the second emotion recognition model. Candidate emotions for each sub-voice content.

Then, the electronic device further determines the second candidate emotion of the user according to the obtained plurality of candidate emotions. For example, the electronic device may determine whether the proportion of the same candidate emotion in all candidate emotions among the plurality of candidate emotions reaches a preset ratio, and if so, may determine the same candidate emotion as the user's second candidate emotion.

It should be noted that the embodiment of the present application does not specifically limit the specific value of the preset ratio, which can be set by those skilled in the art according to actual needs. For example, the preset ratio is set to 60% in the embodiment of the present application. For example, by dividing the voice content, a total of 5 sub-sound contents are obtained, and these 5 sub-sound contents are respectively converted into corresponding spectrograms, and the emotion recognition is carried out through the second emotion recognition model to obtain 5 candidate emotions. Among the candidate emotions, three candidate emotions are the same, all of which are "happy". At this time, "happy" is determined as the second candidate emotion of the user.

In an embodiment, after "determining the user's target emotion according to the first candidate emotion and the second candidate emotion", it also includes:

Send the text content and target emotion to the corresponding target device.

Those of ordinary skill in the art can understand that when a user communicates with other users based on text content through an electronic device, the user does not know the other user's emotion, and the other user does not know the user's emotion. For this reason, in this embodiment of the application, when the text content entered by the user is used for communication with other users, the text content entered by the user and the recognized target emotion are sent to the corresponding target device, that is, the communication with the user Electronic Devices of Other Users. In this way, other users can not only view the text content input by the user, but also know the emotion of the user when inputting the text content, which is helpful for better communication.

Please refer to FIG. 3 and FIG. 4 in conjunction. FIG. 3 is another schematic flow chart of the emotion recognition method provided by the embodiment of the present application, and FIG. 4 is a schematic diagram of an application scenario of the emotion recognition method. The emotion recognition method can be applied to electronic equipment, and the process of the emotion recognition method can include:

In 201, the electronic device acquires text content input by the user when the user communicates with other users.

Those of ordinary skill in the art can understand that when a user communicates with other users based on text content through an electronic device, the user does not know the other user's emotion, and the other user does not know the user's emotion. For this reason, in this embodiment of the application, when the text content entered by the user is used for communication with other users, the user's emotion can be recognized, and the text content entered by the user and the recognized emotion can be sent to other users' electronic devices. devices to help users better communicate with others.

Among them, the electronic device first identifies whether the user communicates with other users. For example, it can be determined whether the user communicates with other users by identifying whether the application program running in the foreground is a communication application program (such as an instant messaging application and a text message, etc.). Wherein, if the application program running in the foreground is a communication application program, the electronic device determines that the user is communicating with other users. When the electronic device recognizes that the user communicates with other users, it detects the user input, and when it detects the text input by the user, it triggers emotion recognition for the user. Wherein, the text content includes but not limited to words, sentences and articles.

In 202, the electronic device extracts keywords included in the aforementioned text content, and maps the extracted keywords to a vector space through a word embedding model to obtain corresponding feature vectors.

In 203, the electronic device converts the aforementioned feature vectors into corresponding feature tensors, and inputs the feature tensors into the pre-trained first emotion recognition model for emotion recognition to obtain the user's first candidate emotions.

It should be noted that, in the embodiment of the present application, a first emotion recognition model for recognizing user emotions according to text content input by the user is pre-trained. In this way, after the electronic device acquires the text content input by the user, it can initially recognize the user's emotion according to the text content and the pre-trained first emotion recognition model.

Wherein, the electronic device first uses a preset keyword extraction algorithm to extract keywords in the text content, and uses the extracted keywords to represent the complete text content. The embodiment of the present application does not specifically limit which keyword extraction algorithm is used to extract keywords from the text content, and a suitable keyword extraction algorithm can be selected by those skilled in the art according to actual needs. For example, in this embodiment of the application, the electronic device can use the ID-TIF algorithm to extract keywords from the text content input by the user. Assume that the text content input by the user is the sentence "I miss you very much today", and use the ID-TIF algorithm to extract keywords from the text content input by the user. Key words are extracted, and the key words are "today" and "missing you".

After the electronic device extracts keywords that can represent the text content, it further maps the keywords extracted from the text content to a vector space through a word embedding model to obtain a feature vector corresponding to the text content. Among them, word embedding models include but are not limited to Word2vec model, GloVe model, FastText model, and ELMo model.

Then, the electronic device further combines the feature vectors corresponding to the text content into a tensor, which is denoted as a feature tensor. Like vectors and matrices, tensor is also a data structure, but tensor is a three-dimensional and above data structure, where the dimension of data is called the order of tensor, and tensor can be regarded as vector and matrix in multidimensional Generalization in space, consider vectors as first-order tensors, and matrices as second-order tensors. After converting the feature vector into a corresponding feature tensor, the feature tensor can be input into the first emotion recognition model for emotion recognition, and the first candidate emotion of the user output by the first emotion recognition model can be obtained.

In 204, the electronic device acquires the sound content during the user inputting the text content.

It should be noted that in the embodiment of the present application, the electronic device not only performs emotion recognition according to the text input by the user, but also performs emotion recognition according to the user's voice. Wherein, when the electronic device detects the user inputting the text content, it activates the built-in microphone or the external microphone to collect the sound, so as to collect the sound content during the user inputting the aforementioned text content. In this way, the electronic device can directly acquire the previously collected sound content during the user inputting the text content.

In 205, the electronic device performs emotion recognition according to the aforementioned voice content and the pre-trained second emotion recognition model to obtain the second candidate emotion of the user.

In the embodiment of the present application, a second emotion recognition model for recognizing the user's emotion according to the user's voice content is also pre-trained. Wherein, the electronic device converts the foregoing sound content into a corresponding spectrogram, and uses the spectrogram to represent the sound content. For example, the conversion of the spectrogram may be performed by using fast Fourier transform or Mel frequency cepstral coefficient.

After the electronic device converts the sound content into a corresponding spectrogram, the converted spectrogram is input to the second emotion recognition model for emotion recognition, and the second candidate emotion of the user output by the second emotion recognition model is obtained.

In 206, the electronic device inputs the first candidate emotion and the second candidate emotion into a pre-trained Bayesian classifier for classification, and obtains the user's target emotion output by the Bayesian classifier.

According to the above description, those skilled in the art can understand that the first candidate emotion and the second candidate emotion obtained above are obtained through independent information sources. Therefore, in order to ensure the accuracy of user emotion recognition, the electronic device also uses The first candidate emotion and the second candidate emotion are comprehensively analyzed to finally determine the user's target emotion.

In the embodiment of the application, a Bayesian classifier for performing secondary sentiment classification on candidate sentiments from text sources and candidate sentiments from voice sources is also pre-trained. For example, it is possible to obtain the emotional samples of the text source and the corresponding emotional samples of the sound source, carry out emotional calibration on the emotional samples of the text source and the corresponding emotional text of the sound source, obtain the corresponding emotional labels, and then use the emotional samples of the text source to and the emotional text of the corresponding sound source, and the corresponding emotional label to obtain a Bayesian classifier.

Wherein, the trained Bayesian classifier can be stored locally in the electronic device, or can be stored in a remote server. In this way, when the electronic device determines the user's target emotion according to the first candidate emotion and the second candidate emotion, it further obtains the Bayesian classifier locally, or obtains the Bayesian classifier from a remote server, so that the previously acquired The first candidate emotion from the text source and the second candidate emotion from the sound source are input into the Bayesian classifier for classification, and the emotion output by the Bayesian classifier is used as the target emotion finally obtained by performing emotion recognition on the user.

In 207, the electronic device sends the aforementioned text content and target emotion to other user's electronic devices.

After recognizing the user's target emotion, the electronic device sends the text content input by the user and the recognized target emotion to the corresponding target device, that is, the electronic device of other users communicating with the user. In this way, other users can not only view the text content input by the user, but also know the emotion of the user when inputting the text content, which is helpful for better communication.

The embodiment of the present application also provides an emotion recognition device. Please refer to FIG. 5 , which is a schematic structural diagram of an emotion recognition device provided by an embodiment of the present application. Where the emotion recognition device is applied to electronic equipment, the emotion recognition device includes a first emotion recognition module 301, a second emotion recognition module 302, a target emotion recognition module 303 and a behavior prediction module 304, as follows:

The first emotion recognition module 301 is used to obtain the text content input by the user, and perform emotion recognition according to the text content and the pre-trained first emotion recognition model, to obtain the user's first candidate emotion;

The second emotion recognition module 302 is used to obtain the voice content during the user input text content, and perform emotion recognition according to the voice content and the pre-trained second emotion recognition model to obtain the user's second candidate emotion;

A target emotion identification module 303, configured to determine the user's target emotion according to the first candidate emotion and the second candidate emotion.

In an embodiment, when determining the user's target emotion according to the first candidate emotion and the second candidate emotion, the target emotion identification module 303 may be used to:

The first candidate emotion and the second candidate emotion are input into a pre-trained Bayesian classifier for classification, and the user's target emotion output by the Bayesian classifier is obtained.

In an embodiment, when performing emotion recognition according to the text content and the pre-trained first emotion recognition model to obtain the user's first candidate emotion, the first emotion recognition module 301 can be used to:

Perform feature extraction on the text content to obtain the corresponding feature vector;

The feature vector is converted into a corresponding feature tensor, and the feature tensor is input into the first emotion recognition model for emotion recognition, and the user's first candidate emotion output by the first emotion recognition model is obtained.

In an embodiment, when performing feature extraction on the text content to obtain the corresponding feature vector, the first emotion recognition module 301 can be used to:

Extract the keywords included in the text content, and map the keywords to the vector space through the word embedding model to obtain the feature vector.

In one embodiment, before inputting the feature tensor into the first emotion recognition model for emotion recognition, it also includes:

Zero pad the feature tensor.

In an embodiment, when performing emotion recognition according to the voice content and the pre-trained second emotion recognition model to obtain the second candidate emotion of the user, the second emotion recognition module 302 can be used to:

dividing the sound content into a plurality of sub-sound contents;

Inputting the content of a plurality of sub-sounds into the second emotion recognition model for emotion recognition to obtain a plurality of corresponding candidate emotions;

A second candidate emotion of the user is determined according to the plurality of candidate emotions.

In one embodiment, the emotion recognition device further includes a content sending module, configured to send the text content and the target emotion to the corresponding target device after determining the user's target emotion according to the first candidate emotion and the second candidate emotion.

It should be noted that the emotion recognition device provided in the embodiment of the present application belongs to the same concept as the emotion recognition method in the above embodiment, and any method provided in the embodiment of the emotion recognition method can be run on the emotion recognition device, and its specific implementation For details on the process, see the embodiment of the emotion recognition method, which will not be repeated here.

An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the stored computer program is executed on a computer, the computer is made to perform the steps in the emotion recognition method provided in this embodiment. Wherein, the storage medium may be a magnetic disk, an optical disk, a read only memory (Read Only Memory, ROM), or a random access device (Random Access Memory, RAM), and the like.

The embodiment of the present application also provides an electronic device, including a memory and a processor, and the processor executes the steps in the emotion recognition method provided in this embodiment by invoking a computer program stored in the memory.

In an embodiment, an electronic device is also provided. Referring to FIG. 6 , the electronic device includes a processor 401 and a memory 402 . Wherein, the processor 401 is electrically connected with the memory 402 .

The processor 401 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or loading the computer program stored in the memory 402 and calling the data stored in the memory 402, the electronic device executes various functions and process data.

The memory 402 can be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402 . The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, computer programs required by at least one function (such as sound playback function, image playback function, etc.); Data created by the use of electronic devices, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 402 may further include a memory controller to provide the processor 401 with access to the memory 402 .

In this embodiment of the application, the processor 401 in the electronic device will follow the steps below to load the instructions corresponding to the process of one or more computer programs into the memory 402, and run and store them in the memory 402 by the processor 401. A computer program to achieve various functions, as follows:

Obtaining the text content input by the user, and performing emotion recognition according to the text content and the pre-trained first emotion recognition model, to obtain the user's first candidate emotion;

Acquiring the voice content during the period when the user inputs the text content, and performing emotion recognition according to the voice content and the pre-trained second emotion recognition model, to obtain the user's second candidate emotion;

The user's target emotion is determined according to the first candidate emotion and the second candidate emotion.

Please refer to FIG. 7 . FIG. 7 is another schematic structural diagram of an electronic device provided by an embodiment of the present application. The difference from the electronic device shown in FIG. 6 is that the electronic device also includes components such as an input unit 403 and an output unit 404 .

Among them, the input unit 403 can be used to receive input numbers, character information or user characteristic information (such as fingerprints), and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

The output unit 404 can be used to display information input by the user or information provided to the user, such as a screen.

In this embodiment of the application, the processor 401 in the electronic device will follow the steps below to load the instructions corresponding to the process of one or more computer programs into the memory 402, and run and store them in the memory 402 by the processor 401. A computer program to achieve various functions, as follows:

Obtaining the text content input by the user, and performing emotion recognition according to the text content and the pre-trained first emotion recognition model, to obtain the user's first candidate emotion;

Acquiring the voice content during the period when the user inputs the text content, and performing emotion recognition according to the voice content and the pre-trained second emotion recognition model, to obtain the user's second candidate emotion;

The user's target emotion is determined according to the first candidate emotion and the second candidate emotion.

In an embodiment, when determining the user's target emotion according to the first candidate emotion and the second candidate emotion, the processor 401 may execute:

The first candidate emotion and the second candidate emotion are input into a pre-trained Bayesian classifier for classification, and the user's target emotion output by the Bayesian classifier is obtained.

In an embodiment, when performing emotion recognition according to the text content and the pre-trained first emotion recognition model to obtain the user's first candidate emotion, the processor 401 may execute:

Perform feature extraction on the text content to obtain the corresponding feature vector;

The feature vector is converted into a corresponding feature tensor, and the feature tensor is input into the first emotion recognition model for emotion recognition, and the user's first candidate emotion output by the first emotion recognition model is obtained.

In an embodiment, when performing feature extraction on the text content to obtain the corresponding feature vector, the processor 401 may execute:

Extract the keywords included in the text content, and map the keywords to the vector space through the word embedding model to obtain the feature vector.

In an embodiment, before inputting the feature tensor into the first emotion recognition model for emotion recognition, the processor 401 may execute:

Zero pad the feature tensor.

In an embodiment, when performing emotion recognition according to the voice content and the pre-trained second emotion recognition model to obtain the second candidate emotion of the user, the processor 401 may execute:

dividing the sound content into a plurality of sub-sound contents;

Inputting the content of a plurality of sub-sounds into the second emotion recognition model for emotion recognition to obtain a plurality of corresponding candidate emotions;

A second candidate emotion of the user is determined according to the plurality of candidate emotions.

In an embodiment, after determining the user's target emotion according to the first candidate emotion and the second candidate emotion, the processor 401 may execute:

Send the text content and target emotion to the corresponding target device.

It should be noted that the electronic device provided in the embodiment of the present application belongs to the same idea as the emotion recognition method in the above embodiment, any method provided in the embodiment of the emotion recognition method can be run on the electronic device, and its specific implementation process is detailed See the embodiment of the emotion recognition method, which will not be repeated here.

It should be noted that, for the emotion recognition method of the embodiment of the present application, those skilled in the art can understand that all or part of the process of implementing the emotion recognition method of the embodiment of the application can be completed by controlling the relevant hardware through a computer program , the computer program may be stored in a computer-readable storage medium, such as stored in a memory of an electronic device, and executed by at least one processor in the electronic device, and the execution process may include, for example, an emotion recognition method Example flow. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, and the like.

For the emotion recognition device of the embodiment of the present application, its various functional modules can be integrated into one processing chip, or each module can exist separately physically, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as read-only memory, magnetic disk or optical disk, etc. .

The above describes in detail the emotion recognition method, device, storage medium and electronic equipment provided by the embodiment of the present application. In this paper, specific examples are used to illustrate the principle and implementation of the present application. The description of the above embodiment is only It is used to help understand the method and its core idea of this application; at the same time, for those skilled in the art, according to the idea of this application, there will be changes in the specific implementation and application scope. In summary, this specification The content should not be construed as a limitation of the application.

Claims (8)

1. A method for emotion recognition, applied to an electronic device, comprising:

the data processing layer in the panoramic sensing architecture acquires text content input by a user from panoramic data and provides the text content to the feature extraction layer, and the scene modeling layer carries out emotion recognition according to the text content and a first emotion recognition model trained in advance to obtain a first candidate emotion of the user, wherein the first candidate emotion comprises the following specific steps: the feature extraction layer performs feature extraction on the text content to obtain feature vectors corresponding to the text content; converting the feature vector of the text content into a corresponding feature tensor at the scene modeling layer, inputting the feature tensor into the first emotion recognition model for emotion recognition, and obtaining a first candidate emotion of the user output by the first emotion recognition model, wherein the feature tensor is a data structure with three dimensions and more than three dimensions;

The data processing layer in the panoramic sensing architecture acquires sound content during the text content input period of a user from panoramic data and provides the sound content to the feature extraction layer, and the scene modeling layer carries out emotion recognition according to the sound content and a pre-trained second emotion recognition model to obtain a second candidate emotion of the user;

determining a target emotion of the user according to the first candidate emotion and the second candidate emotion;

and sending the text content and the target emotion to electronic equipment of other users so that the other users can view the text content for input and know the emotion when the user inputs the text content.

2. The emotion recognition method of claim 1, wherein the determining the target emotion of the user from the first candidate emotion and the second candidate emotion comprises:

and inputting the first candidate emotion and the second candidate emotion into a pre-trained Bayesian classifier to classify, so as to obtain the target emotion of the user output by the Bayesian classifier.

3. The emotion recognition method according to claim 1, wherein the feature extraction of the text content to obtain a corresponding feature vector includes:

And extracting keywords included in the text content, and mapping the keywords to a vector space through a word embedding model to obtain the feature vector.

4. The emotion recognition method of claim 1, wherein before inputting the feature tensor into the first emotion recognition model for emotion recognition, further comprising:

and performing zero filling processing on the characteristic tensor.

5. The method of claim 1, wherein the performing emotion recognition based on the sound content and a pre-trained second emotion recognition model to obtain a second candidate emotion of the user comprises:

dividing the sound content into a plurality of sub-sound contents;

respectively inputting the plurality of sub-sound contents into the second emotion recognition model to perform emotion recognition to obtain a plurality of corresponding candidate emotions;

and determining a second candidate emotion of the user according to the plurality of candidate emotions.

6. An emotion recognition device applied to an electronic apparatus, comprising:

the first emotion recognition module is used for acquiring text content input by a user from panoramic data by the data processing layer in the panoramic sensing architecture and providing the text content to the feature extraction layer, and the scene modeling layer carries out emotion recognition according to the text content and a first emotion recognition model trained in advance to obtain a first candidate emotion of the user, wherein the first candidate emotion comprises the following specific steps of: the feature extraction layer performs feature extraction on the text content to obtain feature vectors corresponding to the text content; converting the feature vector of the text content into a corresponding feature tensor at the scene modeling layer, inputting the feature tensor into the first emotion recognition model for emotion recognition, and obtaining a first candidate emotion of the user output by the first emotion recognition model, wherein the feature tensor is a data structure with three dimensions and more than three dimensions;

The second emotion recognition module is used for acquiring sound content of the user during the text content input period from panoramic data by the data processing layer in the panoramic sensing framework and providing the sound content to the feature extraction layer, and the scene modeling layer carries out emotion recognition according to the sound content and a pre-trained second emotion recognition model to obtain a second candidate emotion of the user;

a target emotion recognition module for determining a target emotion of the user according to the first candidate emotion and the second candidate emotion; and the content sending module is used for sending the text content and the target emotion to electronic equipment of other users after determining the target situation of the user according to the first candidate emotion and the second follow-up situation so that the other users can view the text content used for input and know the emotion when the user inputs the text content.

7. A storage medium having stored thereon a computer program, which when run on a computer causes the computer to perform the emotion recognition method of any of claims 1 to 5.

8. An electronic device comprising a processor and a memory, the memory storing a computer program, wherein the processor is configured to perform the emotion recognition method of any of claims 1 to 5 by invoking the computer program.

Images