CN117407486A - Multimodal dialogue emotion recognition method based on multimodal voting - Google Patents

Abstract

The invention provides a multimode dialogue emotion recognition method based on multimode voting, which comprises the steps of firstly obtaining multimode data generated by speaking of at least 1 speaker, respectively constructing emotion classification tasks of 3 modes aiming at text data, audio data and picture data, and carrying out first emotion classification; merging the multi-mode data by utilizing a multi-head attention mechanism and carrying out second emotion classification; carrying out third emotion classification after the multi-mode emotion feature vector is fused with the time sequence context information; hard voting is carried out on the three emotion classification results, and emotion categories with the largest number of votes obtained by each speaker are respectively used as final emotion classification results to complete multi-mode dialogue emotion recognition; according to the method, the multi-modal interaction mode is optimized, emotion interference is avoided, meanwhile, interaction between a historical dialogue and a speaker is modeled, emotion features contained in each modal are mined in a finer mode, and accuracy and robustness of emotion classification can be enhanced.

Description

A multi-modal dialogue emotion recognition method based on multi-model voting

Technical field

The present invention relates to the technical fields of deep learning and dialogue emotion classification, and more specifically, to a multi-modal dialogue emotion recognition method based on multi-model voting.

Background technique

Conversational emotion classification (ERC) has long been a research direction of great interest in the fields of multimodal classification and natural language processing (NLP). In daily human communication, identifying and tracking the emotional state of conversation participants is crucial to progress in fields such as human-computer interaction, conversation analysis, and video understanding, and has a wide range of potential applications. With the development of streaming media services, multi-modal dialogue emotion recognition has demonstrated a wide range of application scenarios and important significance in the fields of intelligent customer service, social media analysis, emotion-driven content recommendation, emotion analysis research, and emotion-driven human-computer interaction. Taking emotion-driven human-computer interaction systems as an example, this technology can identify and analyze the emotional states expressed by users during conversations, such as anger, satisfaction, frustration, etc. The system can adaptively adjust the interaction method, tone and feedback according to the different emotional needs of users, thereby conducting emotional interaction with users and providing more personalized and humanized services.

Traditional multi-modal emotion recognition methods have some limitations and cannot effectively solve problems such as opposite emotions expressed between modalities, multiple emotions expressed in a single modality, and modal influences between interlocutors, which have an impact on emotion discrimination.

Emotion recognition in multi-modal dialogues usually has the following difficulties: First, in multi-modal dialogue scenarios, unlike traditional single-sentence multi-modal emotion recognition, multi-modal emotion recognition faces more challenging problems. This is because there are multiple factors that affect the speaker's emotional state in the conversation, including multi-modal context, interlocutor stimulation, own emotional inertia, conversation scene and personality characteristics, etc. Therefore, emotion modeling and prediction in different modalities and angles are needed for multimodal dialogue; secondly, there may be conflicts in the emotions expressed between different modalities, for example, a person is smiling but expressing sad emotions. Therefore, Special attention needs to be paid to handling this situation during modal interaction; third, a single modality will also present multiple emotions, so a more fine-grained emotion classification method is needed; fourth, different speakers in the conversation There may be mutual influence between the speaker and the interlocutor, and the speaker's conversational emotions may be stimulated by the interlocutor's expression when the interlocutor is not speaking. Therefore, during the modeling process, the consideration of interactive information is not limited to the speaker's own video, audio, and text features, but also the corresponding video features of the interlocutor. Fifth, when classifying conversation emotions, it is necessary to combine historical conversation content to analyze the current speaker's emotions. Even if the current conversation content has no obvious emotional tendency, the emotional change trend can be analyzed from historical multi-modal conversation data to make better emotional judgments.

The prior art discloses a multi-modal classification method and system based on dynamic context representation and modal fusion, which solves the problem of insufficient analysis of the characteristics of each modality and failure to perform targeted processing according to its characteristics. problem, in which this method performs global context representation, local context representation and direct mapping representation on the features of each modality respectively, and then fuses the above representations according to the dynamic path selection method to obtain the initial fusion features of each modality; all modes are Perform full fusion, partial fusion and biased fusion processing on the initial fusion features of the state to obtain full fusion results, partial fusion results and biased fusion results, and then fuse them through the dynamic path selection method to obtain the final multi-modal fusion used for classification. features, which improves the accuracy of the final recognition task category; although the methods in this prior art can solve the problem that the modal interaction form is too simple, and the characteristics of each modality are not fully analyzed and not based on its characteristics The problem is dealt with in a targeted manner. By distinguishing and dealing with the inconsistency in the amount of information of different modalities in the multi-modal process, the noise caused by the modalities with less information in the modal fusion process is reduced, but usually one Conversation content will contain a variety of emotions. This method cannot mine the emotional characteristics contained in different modalities at a finer granularity. Moreover, when the emotions expressed by different modalities are inconsistent, it will cause interference to modal fusion.

Contents of the invention

In order to overcome the above-mentioned defects of the existing technology in dialogue emotion recognition, which do not conduct fine-grained mining of modalities, modal interactions are susceptible to interference, and the recognition accuracy is low, the present invention provides a multi-modal dialogue emotion based on multi-model voting. The recognition method avoids emotional interference by optimizing multi-modal interaction methods. At the same time, it models historical conversations and interactions between speakers, and mines the emotional characteristics contained in each modality in a more detailed way, which can enhance Accuracy and robustness of sentiment classification.

In order to solve the above technical problems, the technical solutions of the present invention are as follows:

A multi-modal dialogue emotion recognition method based on multi-model voting, including the following steps:

S1: Obtain multi-modal data generated by at least one speaker's dialogue; the multi-modal data includes text data, audio data and picture data; each modal data includes at least one emotion category to be recognized;

S2: Input the text data, audio data and picture data into the preset text encoder, audio encoder and picture encoder respectively for feature extraction, and obtain text features, audio features and picture features respectively;

S3: Input the text features, audio features and picture features into the preset text emotion classifier, audio emotion classifier and picture emotion classifier respectively for the first emotion classification, and obtain the text emotion classification results, audio emotion classification results and Image emotion classification results;

S4: Calculate the penalty factor corresponding to each modality based on the text emotion classification results, audio emotion classification results and picture emotion classification results; multiply the text features, audio features and picture features with the penalty factors corresponding to each modality respectively, Obtain the text weight reduction vector, audio weight reduction vector and picture weight reduction vector respectively;

S5: Input the text down-weighting vector, audio down-weighting vector and picture down-weighting vector into the preset multi-head attention layer to perform fusion and interaction of multi-modal features to obtain multi-modal emotional feature vectors;

S6: Input the multi-modal emotion feature vector into the preset multi-modal emotion classifier for the second emotion classification, and obtain the multi-modal fusion emotion classification result;

S7: Decompose the multi-modal emotional feature vector into several multi-modal emotional feature sub-vectors, and re-splice all multi-modal emotional feature sub-vectors in time series to obtain an emotional feature vector that integrates time series features;

S8: Input the emotional feature vector fused with time series features into the trained bidirectional RNN classifier for the third emotion classification, and obtain the time series context interactive emotion classification results;

S9: Conduct a hard vote on the text emotion classification results, audio emotion classification results, picture emotion classification results, multi-modal fusion emotion classification results and temporal context interactive emotion classification results, and use the emotion category with the most votes for each speaker as the The final emotion classification result completes multi-modal dialogue emotion recognition.

Preferably, the multi-modal data in step S1 is obtained by extracting text data, audio data and picture data respectively from the preset conversation video data of at least one speaker, and obtaining the multi-modal data. .

Preferably, step S1 also includes preprocessing the acquired audio data and picture data. The specific method is:

Perform sampling rate adjustment, denoising and audio gain adjustment on the audio data in sequence, and use a sliding window to extract audio fragments to complete the preprocessing of the audio data;

Perform deduplication operations on the image data, specifically:

S1.1: Use the optical flow algorithm to perform optical flow calculations on two pictures of adjacent frames of the conversation video data, and obtain the optical flow calculation results;

S1.2: Calculate the change amplitude of the two images in adjacent frames based on the optical flow calculation results, and determine whether the change amplitude is greater than the preset threshold. If it is greater, the two images in the adjacent frames are retained in the deduplication result set. Execute step S1.3; otherwise, directly execute step S1.3;

S1.3: Starting from the first frame of the conversation video data, repeat steps S1.1~S1.2 to update the deduplication result set until the last frame of the conversation video data is traversed. The deduplication result set obtained in the last update is saved as the deduplication image data to complete the preprocessing of the image data.

Preferably, the optical flow algorithm in step S1.1 is specifically any one of the Lucas-Kanade algorithm, the Farneback algorithm and the FlowNet algorithm.

Preferably, the change amplitude in step S1.2 is specifically the displacement vector size or similarity measure of each pixel in the two pictures of adjacent frames; the similarity measure includes Euclidean distance and angle change.

Preferably, in step S2, the preset text encoder is specifically a Transformer encoder, and the Transformer encoder is specifically any one of the BERT model, MacBERT model, RoBERTa model and ERNIE model;

The preset audio encoder is specifically the HuBERT model;

The preset picture encoder is specifically a ViT model.

Preferably, the text emotion classifier, audio emotion classifier and picture emotion classifier preset in step S3, and the multi-modal emotion classifier preset in step S6 are all CRF classifiers.

Preferably, in step S4, the specific method for calculating the penalty factor corresponding to each mode is:

According to the text emotion classification results, audio emotion classification results and picture emotion classification results, the type n of the emotion category is obtained, and the total number of occurrences of each emotion category is counted respectively, and the proportion of the i-th emotion category is calculated. ,Specifically:

in, is the number of occurrences of the i-th emotion category;

For the classification results of each modality, the proportions of all emotional categories contained in it are added up, and the summation result is used as the penalty factor for the corresponding modality.

Preferably, the specific method of step S7 is:

Decompose the multi-modal emotion feature vector into several multi-modal emotion feature sub-vectors according to different speakers, and each multi-modal emotion feature sub-vector corresponds to each speaker one-to-one;

The multi-modal emotional feature subvectors include text downweighted subvectors, audio downweighted subvectors and picture downweighted subvectors;

Fuse the image downweighted subvector corresponding to each speaker with the multimodal emotion feature subvectors corresponding to all other speakers to obtain the fused multimodal emotion feature subvector corresponding to each speaker;

The fused multi-modal emotional feature sub-vectors corresponding to all speakers are re-spliced according to the speaking order of different speakers to obtain the emotional feature vector of fused temporal features.

Preferably, the bidirectional RNN classifier in step S8 is specifically:

The bidirectional RNN classifier includes a forward RNN classifier and a backward RNN classifier with the same structure and arranged in parallel;

Both the forward RNN classifier and the backward RNN classifier include several sequentially connected RNN classification layers;

All RNN classification layers have the same structure, and the specific structure is any one of LSTM network, GRU network and PQRNN network.

Compared with the existing technology, the beneficial effects of the technical solution of the present invention are:

The present invention provides a multi-modal dialogue emotion recognition method based on multi-model voting. First, multi-modal data generated by at least one speaker's dialogue is obtained; text data, audio data and picture data are respectively input into a preset text encoder. , audio encoder and picture encoder for feature extraction to obtain text features, audio features and picture features respectively; input text features, audio features and picture features into the preset text emotion classifier, audio emotion classifier and picture emotion respectively. The first emotion classification is performed in the classifier, and the text emotion classification results, audio emotion classification results and picture emotion classification results are obtained respectively; based on the text emotion classification results, audio emotion classification results and picture emotion classification results, the corresponding values of each modality are calculated respectively. Penalty factor; multiply the text features, audio features, and picture features by the penalty factors corresponding to each modality to obtain the text weight reduction vector, audio weight reduction vector, and picture weight reduction vector respectively; multiply the text weight reduction vector, audio weight reduction vector The weight vector and the picture weight reduction vector are jointly input into the preset multi-head attention layer for fusion and interaction of multi-modal features to obtain multi-modal emotion feature vectors; the multi-modal emotion feature vectors are input into the preset multi-modal emotion classification The second emotion classification is performed in the processor to obtain the multi-modal fusion emotion classification results; the multi-modal emotion feature vector is decomposed into several multi-modal emotion feature sub-vectors, and all multi-modal emotion feature sub-vectors are processed in time sequence. Re-splicing to obtain the emotional feature vector fused with temporal features; input the emotional feature vector fused with temporal features into the trained bidirectional RNN classifier for the third emotion classification to obtain the temporal context interactive emotion classification results; finally, the text emotion classification results , audio emotion classification results, picture emotion classification results, multi-modal fusion emotion classification results and temporal context interaction emotion classification results are jointly conducted hard voting, and the emotion category with the most votes for each speaker is regarded as the final emotion classification result. Complete multi-modal dialogue emotion recognition;

By constructing fine-grained emotion classification tasks for each modality, the present invention can solve the problem that the existing technology cannot identify a variety of different emotions expressed in different modalities. Through this training task of emotional segment extraction, which is more difficult than ordinary emotion classification training, , which improves the sensitivity of the classification model to the occurrence position of modal emotional features, thereby improving the judgment accuracy of the model; secondly, the present invention uses the fine-grained emotion classification tasks of each modality to guide the biased fusion interaction of each modality, avoiding the modal Errors and noise in emotional information affect the interaction results; in addition, the present invention also provides a time-series context interaction method for multi-modal data, while combining contextual information and the characteristics of interaction between different speakers in the dialogue to better understand the dialogue The emotional content of the conversation; when one speaker listens and another speaker speaks, the present invention also considers the emotional impact of the listener's expression and picture features on the current speaker, and can more accurately identify the emotional content in the conversation, Effectively enhances the accuracy and robustness of emotion classification.

Description of the drawings

Figure 1 is a flow chart of a multi-modal dialogue emotion recognition method based on multi-model voting provided in Embodiment 1.

Figure 2 is a schematic diagram of text data emotion segments and emotion labels provided in Embodiment 2.

Figure 3 is a schematic diagram of the text encoder and text emotion classifier provided in Embodiment 2.

Figure 4 is a schematic diagram of the audio encoder and audio emotion classifier provided in Embodiment 2.

Figure 5 is a schematic diagram of the image data deduplication operation provided in Embodiment 2.

Figure 6 is a schematic diagram of the third emotion classification provided in Embodiment 2.

Detailed ways

The drawings are for illustrative purposes only and should not be construed as limiting the application;

In order to better illustrate this embodiment, some components in the drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product;

It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

The technical solution of the present invention will be further described below with reference to the accompanying drawings and examples.

Example 1

As shown in Figure 1, this embodiment provides a multi-modal dialogue emotion recognition method based on multi-model voting, which includes the following steps:

S1: Obtain multi-modal data generated by at least one speaker's dialogue; the multi-modal data includes text data, audio data and picture data; each modal data includes at least one emotion category to be recognized;

S2: Input the text data, audio data and picture data into the preset text encoder, audio encoder and picture encoder respectively for feature extraction, and obtain text features, audio features and picture features respectively;

S3: Input the text features, audio features and picture features into the preset text emotion classifier, audio emotion classifier and picture emotion classifier respectively for the first emotion classification, and obtain the text emotion classification results, audio emotion classification results and Image emotion classification results;

S4: Calculate the penalty factor corresponding to each modality based on the text emotion classification results, audio emotion classification results and picture emotion classification results; multiply the text features, audio features and picture features with the penalty factors corresponding to each modality respectively, Obtain the text weight reduction vector, audio weight reduction vector and picture weight reduction vector respectively;

S5: Input the text down-weighting vector, audio down-weighting vector and picture down-weighting vector into the preset multi-head attention layer to perform fusion and interaction of multi-modal features to obtain multi-modal emotional feature vectors;

S6: Input the multi-modal emotion feature vector into the preset multi-modal emotion classifier for the second emotion classification, and obtain the multi-modal fusion emotion classification result;

S7: Decompose the multi-modal emotional feature vector into several multi-modal emotional feature sub-vectors, and re-splice all multi-modal emotional feature sub-vectors in time series to obtain an emotional feature vector that integrates time series features;

S8: Input the emotional feature vector fused with time series features into the trained bidirectional RNN classifier for the third emotion classification, and obtain the time series context interactive emotion classification results;

S9: Conduct a hard vote on the text emotion classification results, audio emotion classification results, picture emotion classification results, multi-modal fusion emotion classification results and temporal context interactive emotion classification results, and use the emotion category with the most votes for each speaker as the The final emotion classification result completes multi-modal dialogue emotion recognition.

In the specific implementation process, multi-modal data generated by at least one speaker's dialogue is first obtained; text data, audio data and picture data are respectively input into the preset text encoder, audio encoder and picture encoder for feature extraction. , obtain text features, audio features and image features respectively;

Input text features, audio features and picture features into the preset text emotion classifier, audio emotion classifier and picture emotion classifier respectively for the first emotion classification, and obtain the text emotion classification results, audio emotion classification results and picture emotion respectively. Classification results;

Calculate the penalty factor corresponding to each modality based on the text emotion classification results, audio emotion classification results and picture emotion classification results respectively; multiply the text features, audio features and picture features with the penalty factors corresponding to each modality to obtain respectively Text downweight vector, audio downweight vector and image downweight vector;

Input the text down-weighting vector, audio down-weighting vector and picture down-weighting vector into the preset multi-head attention layer to perform fusion and interaction of multi-modal features to obtain multi-modal emotional feature vectors;

Input the multi-modal emotion feature vector into the preset multi-modal emotion classifier for the second emotion classification to obtain the multi-modal fusion emotion classification result;

Decompose the multi-modal emotional feature vector into several multi-modal emotional feature sub-vectors, and re-splice all multi-modal emotional feature sub-vectors in time series to obtain an emotional feature vector that integrates time series features;

Input the emotional feature vector fused with temporal features into the trained bidirectional RNN classifier for the third emotion classification, and obtain the temporal context interactive emotion classification results;

Finally, the text emotion classification results, audio emotion classification results, picture emotion classification results, multi-modal fusion emotion classification results and temporal context interaction emotion classification results are jointly put into hard voting, and the emotion category with the largest number of votes for each speaker is selected as the other. The final emotion classification result completes multi-modal dialogue emotion recognition;

This method avoids emotional interference by optimizing multi-modal interaction methods, while modeling historical conversations and interactions between speakers, mining the emotional characteristics contained in each modality in a more detailed way, and can enhance emotions. Classification accuracy and robustness.

Example 2

This embodiment provides a multi-modal dialogue emotion recognition method based on multi-model voting, which includes the following steps:

S1: Obtain multi-modal data generated by at least one speaker's dialogue; the multi-modal data includes text data, audio data and picture data; each modal data includes at least one emotion category to be recognized;

S2: Input the text data, audio data and picture data into the preset text encoder, audio encoder and picture encoder respectively for feature extraction, and obtain text features, audio features and picture features respectively;

S3: Input the text features, audio features and picture features into the preset text emotion classifier, audio emotion classifier and picture emotion classifier respectively for the first emotion classification, and obtain the text emotion classification results, audio emotion classification results and Image emotion classification results;

S4: Calculate the penalty factor corresponding to each modality based on the text emotion classification results, audio emotion classification results and picture emotion classification results; multiply the text features, audio features and picture features with the penalty factors corresponding to each modality respectively, Obtain the text weight reduction vector, audio weight reduction vector and picture weight reduction vector respectively;

S5: Input the text down-weighting vector, audio down-weighting vector and picture down-weighting vector into the preset multi-head attention layer to perform fusion and interaction of multi-modal features to obtain multi-modal emotional feature vectors;

S6: Input the multi-modal emotion feature vector into the preset multi-modal emotion classifier for the second emotion classification, and obtain the multi-modal fusion emotion classification result;

S7: Decompose the multi-modal emotional feature vector into several multi-modal emotional feature sub-vectors, and re-splice all multi-modal emotional feature sub-vectors in time series to obtain an emotional feature vector that integrates time series features;

S8: Input the emotional feature vector fused with time series features into the trained bidirectional RNN classifier for the third emotion classification, and obtain the time series context interactive emotion classification results;

S9: Conduct a hard vote on the text emotion classification results, audio emotion classification results, picture emotion classification results, multi-modal fusion emotion classification results and temporal context interactive emotion classification results, and use the emotion category with the most votes for each speaker as the The final emotion classification result completes multi-modal dialogue emotion recognition;

The multi-modal data in step S1 is obtained by: respectively extracting text data, audio data and picture data from the preset conversation video data of at least one speaker to obtain the multi-modal data;

The step S1 also includes preprocessing the obtained audio data and picture data. The specific method is:

Perform sampling rate adjustment, denoising and audio gain adjustment on the audio data in sequence, and use a sliding window to extract audio fragments to complete the preprocessing of the audio data;

Perform deduplication operations on the image data, specifically:

S1.1: Use the optical flow algorithm to perform optical flow calculations on two pictures of adjacent frames of the conversation video data, and obtain the optical flow calculation results;

S1.2: Calculate the change amplitude of the two pictures in adjacent frames based on the optical flow calculation results, and determine whether the change amplitude is greater than the preset threshold. If it is greater, the two pictures in the adjacent frames are retained in the deduplication result set. Execute step S1.3; otherwise, directly execute step S1.3;

S1.3: Starting from the first frame of the conversation video data, repeat steps S1.1~S1.2 to update the deduplication result set until the last frame of the conversation video data is traversed. The deduplication result set obtained in the last update is saved as the deduplication image data to complete the preprocessing of the image data;

The optical flow algorithm in step S1.1 is specifically any one of the Lucas-Kanade algorithm, the Farneback algorithm and the FlowNet algorithm;

The change amplitude in step S1.2 is specifically the displacement vector size or similarity measure of each pixel in the two pictures of adjacent frames; the similarity measure includes Euclidean distance and angle change;

In step S2, the preset text encoder is specifically a Transformer encoder, and the Transformer encoder is specifically any one of the BERT model, the MacBERT model, the RoBERTa model, and the ERNIE model;

The preset audio encoder is specifically the HuBERT model;

The preset picture encoder is specifically a ViT model;

The text emotion classifier, audio emotion classifier and picture emotion classifier preset in step S3, and the multi-modal emotion classifier preset in step S6 are all CRF classifiers;

In step S4, the specific method for calculating the penalty factor corresponding to each mode is:

According to the text emotion classification results, audio emotion classification results and picture emotion classification results, the type n of the emotion category is obtained, and the total number of occurrences of each emotion category is counted respectively, and the proportion of the i-th emotion category is calculated. ,Specifically:

in, is the number of occurrences of the i-th emotion category;

For the classification results of each modality, the proportions of all emotion categories contained in it are added up, and the summation result is used as the penalty factor for the corresponding modality;

The specific method of step S7 is:

Decompose the multi-modal emotion feature vector into several multi-modal emotion feature sub-vectors according to different speakers, and each multi-modal emotion feature sub-vector corresponds to each speaker one-to-one;

The multi-modal emotional feature subvectors include text downweighted subvectors, audio downweighted subvectors and picture downweighted subvectors;

Fuse the image downweighted subvector corresponding to each speaker with the multimodal emotion feature subvectors corresponding to all other speakers to obtain the fused multimodal emotion feature subvector corresponding to each speaker;

The fused multi-modal emotional feature sub-vectors corresponding to all speakers are re-spliced according to the speaking order of different speakers to obtain the emotional feature vector of fused temporal features;

The bidirectional RNN classifier in step S8 is specifically:

The bidirectional RNN classifier includes a forward RNN classifier and a backward RNN classifier with the same structure and arranged in parallel;

Both the forward RNN classifier and the backward RNN classifier include several sequentially connected RNN classification layers;

All RNN classification layers have the same structure, and the specific structure is any one of LSTM network, GRU network and PQRNN network.

In the specific implementation process, text data, audio data and picture data are first extracted from the preset conversation video data of at least one speaker to obtain multi-modal data; each modal data includes happiness, frustration, One or more emotional categories such as calm, happy, surprised, sad, disgusted, angry, and scared. Emotional categories can also be divided into positive and negative;

In order to solve the problem that the existing technology cannot identify multiple different emotions expressed in different modalities, this embodiment first designs a fine-grained emotion classification task for the data of the three modalities, and performs the first emotion classification, specifically as follows:

1) For text data, first extract the text data separately, and manually label emotional text fragments and emotional labels; for example, as shown in Figure 2, in the text "I feel very happy, even if you keep making me cry" There are two emotional segments, corresponding to the emotional labels of "happy (pos)" and "frustrated (neg)" respectively;

As shown in Figure 3, the Transformer encoder is then used to extract text features, and the CRF classifier is used to learn the dependencies between labels and emotion classification;

The emotional classification of text includes two levels of judgment: one is to judge which text belongs to the beginning (B), middle (I) or end (E) of the emotional text segment, or is not an emotional segment (O); the other is to judge Which emotion type (pos or neg) the emotion segment belongs to;

In this embodiment, the Transformer encoder is specifically any one of BERT, MacBERT, RoBERTa and ERNIE models;

To train a model using text data sets, you need to define a loss function and optimizer, and use the backpropagation algorithm to update the model parameters. During the training process, batch gradient descent or other optimization techniques are usually used; according to the model's Performance on the validation set, adjust hyperparameters, such as learning rate, batch size, number of Transformer layers, number of hidden units, etc.; you can use methods such as cross-validation or grid search for hyperparameter tuning;

Then use the trained model to perform the first emotion classification on the input text data, and obtain the text emotion classification results, specifically: text fragments containing emotional states and corresponding emotion categories;

Through the text emotion segment recognition and classification task, the model can focus more on identifying segments containing emotional information in the dialogue text. At the same time, when multiple emotions exist in the dialogue text, the model can identify more accurately;

2) For audio data, collect a data set containing audio data and corresponding emotional labels, and preprocess the audio signal to be recognized, such as sampling rate adjustment, denoising and audio gain adjustment, etc.; set the size and step of the sliding window. long, perform sliding window processing on the audio signal, starting from the starting position of the audio, and move the sliding window in sequence according to the set step size. Extract corresponding audio clips within each window;

The size of the sliding window determines the duration of each audio segment, while the step size determines the degree of overlap between adjacent windows;

Mark the extracted audio clips with corresponding emotion tags to ensure that each audio clip has an emotion label associated with it. The start of each audio clip is represented by the symbol "B", the content is represented by the symbol "I", and the end is represented by the symbol "E" indicates that non-emotional audio clips are represented by the symbol "O"; in addition, audio clips are also labeled according to emotional categories, such as "pos" indicating positive positive emotions and "neg" indicating negative negative emotions;

As shown in Figure 4, the preprocessed audio data set is data enhanced and then input into the HuBERT model + CRF classifier for training. The HuBERT model is used to extract contextual information of the input audio sequence, and the CRF classifier is used to learn between labels. The dependencies of ) to optimize model parameters, and use the backpropagation algorithm to calculate gradients and update model parameters;

This embodiment also sets an emotion probability threshold parameter. For each audio segment, it is determined whether the predicted emotion category or emotion score exceeds the set emotion probability threshold parameter. If it exceeds the threshold, it is determined that the time step is a segment containing emotion;

In addition, for time steps that are determined to contain emotions, continuous frame analysis also needs to be performed. This embodiment sets a threshold for continuous frames, indicating how many consecutive time steps are determined to contain emotional segments before the audio segment is finally Identified as an emotional piece;

Then use the trained HuBERT model + CRF classifier to perform the first emotion classification on the input audio data, and obtain the audio emotion classification results;

3) For picture data, the conversation video data is first divided into frames to extract feature information that expresses emotions. The feature information is mainly concentrated on human faces;

Afterwards, the image data is deduplicated, specifically as follows:

S1.1: Use the optical flow algorithm to perform optical flow calculations on two pictures of adjacent frames of the conversation video data to obtain the optical flow calculation results; the optical flow algorithm can estimate the pixel displacement between adjacent frames and capture inter-frame motion. Information, specifically any one of the Lucas-Kanade algorithm, Farneback algorithm and FlowNet algorithm;

S1.2: Calculate the change amplitude of two pictures in adjacent frames based on the optical flow calculation results. The change amplitude is the displacement vector size of each pixel or similarity measure (such as Euclidean distance and angle change);

Determine whether the change amplitude is greater than the preset threshold. If it is greater, retain the two pictures of the adjacent frames in the deduplication result set and execute step S1.3; otherwise, directly execute step S1.3;

S1.3: Starting from the first frame of the conversation video data, repeat steps S1.1~S1.2 to update the deduplication result set until the last frame of the conversation video data is traversed. The deduplication result set obtained in the last update is saved as the deduplication image data to complete the preprocessing of the image data;

Frames with large changes are retained, while frames with small changes are discarded. As shown in Figure 5, these frames cover the speaker's change from angry to happy. After deduplication, the frames representing "happy" and "angry" are retained. "Pictures in the emotion category;

Then use the preprocessed image data to train the model, and use the ViT model + CRF classifier to extract image features and classify. The ViT model divides the image into image blocks (or image patches), and then rearranges them into sequences to process the image; training Commonly used loss functions include smooth L1 loss for bounding box regression and cross-entropy loss for classification tasks. Model parameters are optimized by minimizing the loss function during training. Stochastic gradient descent (SGD) or other optimization algorithms can be used for parameter update;

Finally, the first emotion recognition of the image data is performed through the trained ViT model + CRF classifier, and the emotion classification result of the image data is obtained, that is, the emotion category corresponding to the face area;

After the first emotion classification, in order to avoid the influence of noise in multi-modal emotion classification, this embodiment also performs a second emotion classification, specifically as follows:

The penalty factor corresponding to each modality is calculated based on the text emotion classification results, audio emotion classification results and picture emotion classification results. The specific method is:

According to the text emotion classification results, audio emotion classification results and picture emotion classification results, the type n of the emotion category is obtained, and the total number of occurrences of each emotion category is counted respectively, and the proportion of the i-th emotion category is calculated. ,Specifically:

in, is the number of occurrences of the i-th emotion category;

For the classification results of each modality, the proportions of all emotion categories contained in it are added up, and the summation result is used as the penalty factor for the corresponding modality;

Multiply the text features, audio features and picture features by the penalty factors corresponding to each modality to obtain the text down-weighting vector, audio down-weighting vector and picture down-weighting vector respectively;

The text down-weighting vector, audio down-weighting vector and picture down-weighting vector are jointly input into the preset multi-head attention layer for fusion and interaction of multi-modal features to obtain multi-modal emotional feature vectors; through the multi-head attention mechanism, the model can Automatically focus on the parts of different modalities that have an important impact on emotion classification;

Input the multi-modal emotion feature vector into the preset multi-modal emotion classifier for the second emotion classification to obtain the multi-modal fusion emotion classification result;

After the second emotion classification, this embodiment also takes into account the mutual influence between different speakers in the dialogue, and also performs a third emotion classification, specifically as follows:

First, the multi-modal emotion feature vector is decomposed into several multi-modal emotion feature sub-vectors according to different speakers, and each multi-modal emotion feature sub-vector corresponds to each speaker one-to-one;

The multi-modal emotional feature subvectors include text downweighted subvectors, audio downweighted subvectors and picture downweighted subvectors;

As shown in Figure 6, assuming there are two speakers, speakers A and B have the following conversations in chronological order:

A: (Neutral) Dialogue 1;

B: (Neutral) Dialogue 2;

A: (Surprise) Dialogue 3;

B: (Sad) Dialogue 4;

B: (Fear, Sad) Dialogue 5;

A: (Anger) Dialogue 6;

The six sets of multimodal vectors in Figure 6 respectively correspond to the above six sentences of dialogue. When speaker A is speaking in the current round, speaker B is not speaking but has expressions. Therefore, this situation must be considered. For each time step, f _a is the multi-modal emotional feature subvector of speaker A, and the audio and text parts of the f _b vector spliced with it are empty, with only the reduced weight subvector of the picture modality. The picture modality here is selected from the previous one. Take the picture of speaker B (when the speaker is B, the same process is performed);

In Figure 6, the sliding window method is used to add each speaker's feature sequence to a fixed-length feature window, and the features within the window are spliced into a feature vector to add time series to each speaker's feature sequence. To use contextual interactive information to determine the emotion of the current conversation, it is necessary to integrate the features of the previous n rounds of dialogue into the features of the current dialogue through the RNN model to jointly determine the emotion. The features of the first n rounds serve for the emotional classification of the current dialogue; The fusion features of interlocutors A and B are repeatedly added to each time step to reflect the progress of the conversation and the mutual influence between interlocutors;

In addition, in Figure 6, speaker A and speaker B can also be the same person. In this case, it can be regarded as only one speaker talking to himself;

Fuse the image downweighted subvector corresponding to each speaker with the multimodal emotion feature subvectors corresponding to all other speakers to obtain the fused multimodal emotion feature subvector corresponding to each speaker;

The fused multi-modal emotional feature sub-vectors corresponding to all speakers are re-spliced according to the speaking order of different speakers to obtain the emotional feature vector of fused temporal features;

Input the emotional feature vector fused with temporal features into the trained bidirectional RNN classifier for the third emotion classification to obtain the temporal context interactive emotion classification results. The final emotion output is the emotion category of the last sentence of each speaker's dialogue;

As shown in Figure 6, the bidirectional RNN classifier includes a forward RNN classifier and a backward RNN classifier with the same structure and arranged in parallel; the forward RNN classifier and the backward RNN classifier each include several sequentially connected RNN classification layer; all RNN classification layers have the same structure, and the specific structure is an LSTM network;

At each time step, the forward RNN processes the input sequence from beginning to end, while the backward RNN processes it from end to beginning. The hidden layer states in both directions can be spliced at each time step to form a more comprehensive representation. Contains temporal context interaction information;

Finally, the results of the three emotion classifications were hard voted according to the majority rule, and the text emotion classification results, audio emotion classification results, picture emotion classification results, multi-modal fusion emotion classification results and temporal context interaction emotion classification results were jointly conducted. Hard voting uses the emotion category with the most votes for each speaker as the final emotion classification result to complete multi-modal dialogue emotion recognition; if there are multiple emotion results that appear most often, any one of them can be selected as the final result. Sentiment classification results;

This method avoids emotional interference by optimizing multi-modal interaction methods, while modeling historical conversations and interactions between speakers, mining the emotional characteristics contained in each modality in a more detailed way, and can enhance emotions. Classification accuracy and robustness.

The same or similar numbers correspond to the same or similar parts;

The terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limitations to the present application;

Obviously, the above-mentioned embodiments of the present invention are only examples to clearly illustrate the present invention, and are not intended to limit the implementation of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. An exhaustive list of all implementations is neither necessary nor possible. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims (10)

1. A multimode dialogue emotion recognition method based on multimode voting is characterized by comprising the following steps:

s1: acquiring multi-modal data generated by speaking at least 1 speaker; the multi-modal data includes text data, audio data, and picture data; the data of each mode comprises at least 1 emotion type to be identified;

S2: respectively inputting text data, audio data and picture data into a preset text encoder, an audio encoder and a picture encoder for feature extraction, and respectively obtaining text features, audio features and picture features;

s3: inputting the text features, the audio features and the picture features into a preset text emotion classifier, an audio emotion classifier and a picture emotion classifier respectively for first emotion classification, and obtaining a text emotion classification result, an audio emotion classification result and a picture emotion classification result respectively;

s4: respectively calculating penalty factors corresponding to each mode according to the text emotion classification result, the audio emotion classification result and the picture emotion classification result; multiplying the text feature, the audio feature and the picture feature with penalty factors corresponding to each mode respectively to obtain a text weight-reducing vector, an audio weight-reducing vector and a picture weight-reducing vector respectively;

s5: the text weight-reducing vector, the audio weight-reducing vector and the picture weight-reducing vector are input into a preset multi-head attention layer together to perform multi-mode feature fusion interaction, and a multi-mode emotion feature vector is obtained;

s6: inputting the multi-modal emotion feature vector into a preset multi-modal emotion classifier to carry out second emotion classification, and obtaining a multi-modal fusion emotion classification result;

S7: decomposing the multi-modal emotion feature vector into a plurality of multi-modal emotion feature sub-vectors, and re-splicing all the multi-modal emotion feature sub-vectors according to the time sequence to obtain an emotion feature vector fused with the time sequence features;

s8: inputting emotion feature vectors fused with time sequence features into a trained bidirectional RNN classifier to perform third emotion classification, and obtaining a time sequence context interaction emotion classification result;

s9: and carrying out hard voting on the text emotion classification result, the audio emotion classification result, the picture emotion classification result, the multi-modal fusion emotion classification result and the time sequence context interaction emotion classification result, and respectively taking the emotion type with the largest number of votes obtained by each speaker as the final emotion classification result to finish multi-modal dialogue emotion recognition.

2. The multimodal dialog emotion recognition method based on multimodal voting according to claim 1, wherein the multimodal data in step S1 is obtained by: and respectively extracting text data, audio data and picture data from preset dialogue video data of at least 1 speaker, and obtaining the multi-modal data.

3. The multimodal dialog emotion recognition method based on multimodal voting according to claim 2, wherein the step S1 further comprises preprocessing the obtained audio data and the obtained picture data, and the specific method is as follows:

Sequentially carrying out sampling rate adjustment, denoising and audio gain adjustment on the audio data, extracting audio fragments by utilizing a sliding window, and finishing preprocessing of the audio data;

and carrying out de-duplication operation on the picture data, wherein the de-duplication operation specifically comprises the following steps:

s1.1: performing optical flow calculation on 2 pictures of the adjacent frames of the dialogue video data by using an optical flow algorithm to obtain an optical flow calculation result;

s1.2: calculating the variation amplitude of 2 pictures of the adjacent frame according to the optical flow calculation result, judging whether the variation amplitude is larger than a preset threshold value, if so, reserving the 2 pictures of the adjacent frame to a de-duplication result set, and executing the step S1.3; otherwise, directly executing the step S1.3;

s1.3: and starting from the first frame of picture of the dialogue video data, repeating the steps S1.1-S1.2 to update the duplicate removal result set until the last frame of picture of the dialogue video data is traversed, and storing the duplicate removal result set obtained by the last update as duplicate removal picture data to finish preprocessing of the picture data.

4. The multimode dialogue emotion recognition method based on multimode voting according to claim 3, wherein the optical flow algorithm in the step S1.1 is specifically any one of Lucas-Kanade algorithm, farnebback algorithm and FlowNet algorithm.

5. The multimodal dialog emotion recognition method based on multimodal voting according to claim 4, wherein the variation amplitude in step S1.2 is specifically a displacement vector size or similarity measure of each pixel in 2 pictures of adjacent frames; the similarity measure includes euclidean distance and angle variation.

6. The multimodal dialog emotion recognition method based on multimodal voting according to claim 1 or 5, wherein in the step S2, the preset text encoder is specifically a transducer encoder, and the transducer encoder is specifically any one of a BERT model, a MacBERT model, a RoBERTa model and an ERNIE model;

the preset audio encoder is specifically a HuBERT model;

the preset picture encoder is specifically a ViT model.

7. The multimodal dialog emotion recognition method based on multimodal voting according to claim 6, wherein the text emotion classifier, the audio emotion classifier and the picture emotion classifier preset in step S3 are all CRF classifiers.

8. The multimodal dialog emotion recognition method based on multimodal voting according to claim 7, wherein in the step S4, the specific method for calculating the penalty factor corresponding to each modality is as follows:

Acquiring the category n of emotion categories according to the text emotion classification result, the audio emotion classification result and the picture emotion classification result, respectively counting the total occurrence times of each emotion category, and calculating the duty ratio of the ith emotion categoryThe method specifically comprises the following steps:

wherein,the number of occurrences of the ith emotion type;

and for the classification result of each mode, the duty ratios of all emotion categories contained in the classification result are added, and the addition result is used as a penalty factor of the corresponding mode.

9. The multimodal dialog emotion recognition method based on multimodal voting according to claim 8, wherein the specific method of step S7 is as follows:

decomposing the multi-modal emotion feature vector into a plurality of multi-modal emotion feature sub-vectors according to different speakers, wherein each multi-modal emotion feature sub-vector corresponds to each speaker one by one;

the multi-mode emotion feature sub-vector comprises a text weight-reducing sub-vector, an audio weight-reducing sub-vector and a picture weight-reducing sub-vector;

fusing the picture weight-reducing sub-vector corresponding to each speaker with the multi-modal emotion feature sub-vectors corresponding to all other speakers respectively to obtain fused multi-modal emotion feature sub-vectors corresponding to each speaker;

And re-splicing the fused multi-mode emotion feature sub-vectors corresponding to all the speakers according to the speaking sequence of different speakers to obtain emotion feature vectors fused with time sequence features.

10. The multimodal dialog emotion recognition method based on multimodal voting according to claim 9, wherein the bidirectional RNN classifier in step S8 specifically comprises:

the bidirectional RNN classifier comprises a forward RNN classifier and a backward RNN classifier which are identical in structure and are arranged in parallel;

the forward RNN classifier and the backward RNN classifier both comprise a plurality of RNN classification layers which are connected in sequence;

all RNN classification layers have the same structure, and the specific structure is any one of an LSTM network, a GRU network and a PQRNN network.