CN117711409A - Multifunctional voice interaction device based on deep learning - Google Patents

Abstract

The invention provides a multifunctional voice interaction device based on deep learning, belonging to the field of voice interaction. The device includes: a human-computer interaction module, used to provide a human-computer interaction interface for users, receive the voice and instructions input by the user, and display the voice Text and emotion categories are used to play multi-style speech; the speech recognition module is used to perform text recognition and emotion recognition on the voice input by the user based on the deep learning model, and the speech text and emotion categories are obtained; the speech synthesis module is used to perform text recognition and emotion recognition based on the speech synthesis model. , generate multi-style speech based on the speech input by the user. The present invention improves the flexibility of voice interaction and adds emotion recognition to voice recognition, so that words and emotions can be recognized simultaneously based on voice, making voice interaction closer to actual communication scenarios.

Description

A multifunctional voice interaction device based on deep learning

Technical field

The present invention relates to the field of voice interaction, and in particular to a multifunctional voice interaction device based on deep learning.

Background technique

Parks are an important part of the urban green space system, and their importance in cities has become increasingly prominent. With the rapid development of new generation information technologies such as 5G, Internet of Things, and big data, the management and service needs of parks have shown a trend of diversified development. It has become increasingly difficult for traditional parks to meet new needs and exposed many problems. Parks "Intelligence" has become an inevitable development trend. With the development of the new generation of information and communication technology, parks are realizing digital expression in aspects such as park service management and citizen intelligent interaction. As a city park that serves citizens, people-oriented and improving citizens' enjoyment and sense of participation in the park are the foundation and top priority for the development of smart parks. As intelligent interactive products that visitors can directly access, they are fully in line with the people-oriented core concept of smart parks. .

Regarding the intelligent interaction scenarios for citizens in smart parks, after research, it was found that the current interactive products have many shortcomings such as insufficient intelligence, insufficient main control computing power, and poor user interaction. Therefore, it is necessary to create an intelligent interactive device equipped with cutting-edge technology to provide citizens with entertainment and science popularization.

Contents of the invention

The purpose of the present invention is to provide a multifunctional voice interaction device based on deep learning, which can improve the flexibility of voice interaction and make voice interaction closer to actual communication scenarios.

To achieve the above objectives, the present invention provides a multifunctional voice interaction device based on deep learning, including:

The human-computer interaction module is used to provide users with a human-computer interaction interface, receive voice and instructions input by the user, display voice text and emotion categories, and play multi-style voices;

A speech recognition module, connected to the human-computer interaction module, is used to perform text recognition and emotion recognition on the speech input by the user based on a deep learning model to obtain speech text and emotion categories;

A speech synthesis module, connected to the human-computer interaction module, is used to generate multi-style speech according to the speech input by the user based on the speech synthesis model.

Optionally, the human-computer interaction module, the speech recognition module and the speech synthesis module are all built based on the JetsonNano SOC main control chip.

Optionally, the human-computer interaction module includes an HMI configuration screen, a microphone and a speaker.

Optionally, the speech recognition module includes:

The speech and text recognition sub-module is used to perform text recognition on the speech input by the user based on the text recognition model; the text recognition model is obtained by pre-training the Efficient Conformer model using the speech and text data set on the personal computer; the speech and text data The collection includes multiple voice samples and the text corresponding to each voice sample;

The voice emotion recognition sub-module is used to perform emotion recognition on the voice input by the user based on the emotion recognition model to determine the user's emotion category; the emotion recognition module uses the voice emotion data set on the personal computer in advance to analyze the voiceprint. The recognition model is trained; the voice emotion data set includes multiple voice samples and the emotion categories corresponding to each voice sample: the voiceprint recognition model includes a first fully connected layer, a reshaping layer, and a bidirectional LSTM layer that are connected in sequence. , Tanh activation layer, dropout layer, second fully connected layer, ReLU activation layer and third fully connected layer.

Optionally, the text recognition model includes a CNN front end, multiple LAC encoders and multiple LAC decoders connected in sequence;

The CNN front-end includes a first convolutional neural network, a second convolutional neural network and an embedded neural network connected in sequence;

Each LAC encoder includes a first low-rank feedforward neural network, a first multi-head attention layer, a third convolutional neural network and a second low-rank feedforward neural network connected in sequence;

Each LAC decoder consists of a masked multi-head attention layer, a first normalization layer, a second multi-head attention layer, a second normalization layer, a third low-rank feedforward neural network and a third normalization layer connected in sequence. One layer.

Optionally, the speech text data set is obtained by preprocessing the Chinese female voice database AISHELL-1 using a linear energy map, and using the ffmpeg tool to enhance noise, enhance speech speed, enhance volume, offset enhancement and spectrum enhancement. .

Optionally, the speech synthesis model is obtained by pre-training the fastspeech2 model using a multi-speaker data set and then converting it into a dynamic graph model using the Paddld tool.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: The multifunctional voice interaction device based on deep learning provided by the present invention integrates human-computer interaction, speech recognition and speech synthesis functions, improving the flexibility of voice interaction. , and by adding emotion recognition to speech recognition, text and emotions can be recognized simultaneously based on speech, making voice interaction closer to actual communication scenarios.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the drawings of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of the functional modules of the multifunctional voice interaction device based on deep learning provided by the present invention;

Figure 2 is a schematic diagram of the hardware composition of the multifunctional voice interaction device based on deep learning provided by the present invention;

Figure 3 is a schematic diagram of the HMI host computer configuration interface;

Figure 4 is a schematic diagram of the text recognition model;

Figure 5 is the training flow chart of the Efficient Conformer model;

Figure 6 is a flow chart of linear energy map processing;

Figure 7 shows the training results of the Efficient Conformer model;

Figure 8 is a schematic diagram of the voiceprint recognition model;

Figure 9 is a schematic diagram of the voiceprint recognition model training set loss;

Figure 10 shows the changes in the accuracy of the general performance index of the classification function of the voiceprint recognition model training set;

Figure 11 is another change diagram of the accuracy of the general performance indicator of the classification function of the voiceprint recognition model test set;

Figure 12 shows the confusion matrix of the voiceprint recognition model;

Figure 13 is the flow chart of the PaddleSpeech project implementation;

Figure 14 is the Mel spectrogram of real sounds in the AISHELL-3 data set;

Figure 15 is the Mel spectrum diagram of fastspeech2 synthesized sound.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

The purpose of the present invention is to provide a multifunctional voice interaction device based on deep learning, which adds deep learning technology to traditional voice interaction and is equipped with functions such as voice text and emotion recognition, and multi-style speech synthesis.

In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

As shown in Figure 1, the multifunctional voice interaction device based on deep learning provided by the present invention includes: a human-computer interaction module, a speech recognition module and a speech synthesis module.

Specifically, the human-computer interaction module, the speech recognition module and the speech synthesis module are all built based on the Jetson Nano SOC main control chip. As shown in Figure 2, the software of the present invention is based on the Jetson nano SOC main control chip and is implemented by installing the Ubuntu18.04 system. The hardware mainly includes human-computer interaction interface, Jetson Nano SOC main control chip, microphone array, external speakers and other sensors. Among them, the human-computer interaction interface is connected to the Jetson Nano SOC main control chip through UART communication, and the microphone array, external speakers and other sensors are connected to the Jetson Nano SOC main control chip through USB (Universal Serial Bus, serial bus).

The human-computer interaction module is used to provide users with a human-computer interaction interface, receive voice and instructions input by the user, display voice text and emotion categories, and play multiple styles of voices.

Specifically, the human-computer interaction module includes an HMI configuration screen, a microphone and a speaker. The interactive module is a bridge that realizes the connection of various functions. The present invention uses serial communication and cooperates with the HMI configuration screen to realize human-computer interaction between the user and the device. The human-computer interaction module is divided into two parts: HMI interactive interface design and UART communication. Serial port interaction is the basis for realizing voice interaction. The present invention uses serial port HMI as the host computer, designs and builds a configuration interface on it, uses UART to interact with Jetson Nano, and finally achieves the purpose of intelligent voice interaction. Figure 3 shows the HMI host computer configuration interface.

(1) HMI interactive interface design. The HMI interactive interface includes the location of each function button, the distribution of displayed information, etc. The HMI interactive interface includes the main interface, information display interface and virtual drawing interface. Each interface contains basic information display and various function buttons. The main interface is mainly used to switch between functional interfaces. The information display interface is used to display the current temperature and humidity, current time, prompt information and function buttons. Among them, the current time is implemented by the RTC (Real_Time Clock, real-time clock) acquisition function, and the HMI will automatically obtain the current time and display it on the screen. After clicking the function button, the interface will be adjusted.

(2) Communication is the basis of the human-computer interaction module and a necessary function to realize the interaction of multiple devices. UART communication is simple to implement, only two wires are needed to transmit data, and the transmission is stable. Writing a communication protocol is the basis for data transmission between two devices. The implementation of communication function is divided into HMI side and JetsonNano side.

Specifically, first define the function variable funcTX on the HMI side and associate funcTX with the button, that is, pressing the button will trigger funcTX. Use the custom prints function to send data to the JetsonNano using the serial port. After the HMI sends the data, JetsonNano needs to receive and process the data. In the terminal, use the serial library to receive serial port data for analysis and then use the subprocess library to run the corresponding instructions. First, reference the serial library, open the JetsonNano serial port, set the baud rate, stop bit and other parameters, so that it is in a waiting state; then create a functional class and define the operations corresponding to each protocol in the class, such as creating a MUSIC target detection class: MUSIC_010000 , the class name consists of functions and communication protocols.

Write shell script commands according to functional requirements and use the subprocess library to control Linux system devices. For example, after receiving a command to open a music player, run the MUSIC.sh file. This script will switch the system to the MUSIC directory and run the corresponding program. In addition, create a dictionary and add each operation to the dictionary. After receiving the data, the serial port retrieves the matching instructions in the dictionary to open the corresponding function.

The speech recognition module is connected to the human-computer interaction module. The speech recognition module is used to perform text recognition and emotion recognition on the speech input by the user based on the deep learning model to obtain the speech text and emotion categories.

In the present invention, the speech recognition module includes a speech character recognition sub-module and a speech emotion recognition sub-module. Speech and text recognition is the basis of voice interaction. Only by recognizing accurate text can you have a good voice interaction experience. Traditional speech recognition only has text recognition function and cannot achieve personalized or real voice interaction experience. The speech recognition module provided by the present invention adds speech emotion recognition on the basis of traditional speech and text recognition, which can not only recognize speech characters but also recognize Voice emotions. Both functions use Baidu's Paddle open source framework. Speech and text recognition uses the Efficient Conformer model, and speech emotion recognition uses the SER-TDNN model (voiceprint recognition model) invented based on ECAPA-TDNN.

(1) The speech and text recognition sub-module is used to perform text recognition on the speech input by the user based on the text recognition model.

The text recognition model is obtained by pre-training the EfficientConformer model on a personal computer using a speech text data set. The speech text data set includes multiple speech samples and the text corresponding to each speech sample. The speech text data set is obtained by preprocessing the Chinese female voice database AISHELL-1 using a linear energy map, and using the ffmpeg tool to enhance noise, speech speed, volume, offset enhancement and spectrum enhancement.

Specifically, as shown in Figure 4, the text recognition model includes a CNN front end, E LAC encoders and D LAC decoders connected in sequence.

The CNN front-end includes the first convolutional neural network, the second convolutional neural network and the embedded neural network connected in sequence. The CNN front-end is used to extract features from the input sound, including the frame number, frequency and other features of the voice, and convert the input acoustic features into a feature matrix using absolute position encoding. Among them, T is the sequence length after downsampling, and _de is the dimension of the embedding layer.

Each LAC encoder includes a first low-rank feedforward neural network, a first multi-head attention layer, a third convolutional neural network and a second low-rank feedforward neural network connected in sequence.

The LAC encoder introduces a linear attention mechanism based on the Conformer model, so that EfficientConformer can still maintain good recognition accuracy even when the model complexity is reduced.

The input data processing process of the i-th LAC encoder is:

X″ _i =X′ _i +Conv(X′ _i );

Among them, _Xi is the input data of the i-th LAC encoder, LFFN() is a low-rank feedforward neural network, is the output data of the first low-rank feedforward neural network, MHLSA() is the multi-head attention mechanism, X' _i is the output data of the first multi-head attention layer, Conv() is the convolution, and X″ _i is the third The output data of the convolutional neural network, Layernorm() is layer normalization, and Y _i is the output data of the i-th LAC encoder.

Each LAC decoder consists of a masked multi-head attention layer, a first normalization layer, a second multi-head attention layer, a second normalization layer, a third low-rank feedforward neural network and a third normalization layer connected in sequence. One layer. The third low-rank feedforward neural network in the LAC decoder is the same as the first and second low-rank feedforward neural networks in the LAC encoder. Through D LAC decoders, the speech representation learned by the model can be mapped to the final text result, and the features extracted from the speech signal are converted into corresponding text output.

This invention uses the PPASR project to transplant the Efficient Conformer model, and first performs model training and optimization on the PC (Personal Computer) side. The training process is shown in Figure 5. To use the PaddlePaddle deep learning framework, the main installed Python libraries include: Paddleaudio-1.0.2, scikit-learn-1.2.2, etc., and install CUDA (Compute Unified Device Architecture, unified computing device architecture) to enable GPU (GraphicsProcessing Unit, graphics processing machine) accelerates.

The speech text data set uses the open source Chinese female voice database AISHELL-1 and speech enhancement noise data. First, each data is classified, summarized and renamed according to predetermined requirements, and a linear energy diagram is used for data preprocessing. Linear energy map is an audio preprocessing method provided by PaddleSpeech. It mainly performs real-number fast Fourier transform on the audio to obtain the spectrogram of the real signal, and after scaling and other operations, obtains a list of data that can be directly processed by the model. The specific processing flow is shown in Figure 6. The audio is sequentially pre-emphasized and windowed, real fast Fourier transformed, absolute value squared, compressed, fast Fourier frequency calculated and logarithmic processed. After the data is preprocessed by the linear energy map, matrix data of different lengths are obtained, and the lengths need to be unified. The processing method is: after reading a batch of data, sort the data, find the longest audio and label, and then follow the max-length creates a zero tensor, and then fills all the original data into this tensor to obtain uniform length data with 0 at the end. In order to improve the accuracy of the text recognition model under limited data, the present invention introduces the ffmpeg tool based on the original project and adds five speech data enhancement methods: noise enhancement, speech rate enhancement, volume enhancement, offset enhancement, and spectrum enhancement. In the configuration file, modify the corresponding parameters to use the corresponding data enhancement method.

The training environment of the text recognition model is Intel i9-12900K processor I, the graphics card is NVIDIA GeForceRTX 3090, and the graphics card memory is 24G. The training settings are batch_size=128, num_workers=24, training is 186 rounds, and the training time is about 75 hours. The training results are shown in Figure 7. The text recognition model uses the CTC and LSM joint loss functions, with each weight being 0.5. In order to prevent overfitting, the text recognition model uses a preheating learning strategy, with a preheating learning rate of 0.00005, a number of preheating steps of 25000, and an initial learning rate of 0.001; the word error rate CER is a general performance indicator for speech recognition: CER = (insertion + deletion + replacement)/total number of words. Insertion, deletion and replacement are the parts that need to be adjusted after comparing the predicted text and the correct text. The best word error rate in 186 rounds is 0.052, and the recognition accuracy can reach 95%. Suitable for device use.

The text recognition model is a Windows system based on the X86 architecture, and the main control used in the present invention is based on the ARM64 structure. Therefore, it is first necessary to change the deep learning framework under the Windows architecture such as paddlepaddle to the ARM architecture. In addition, speech processing deep learning models will convert the data into a large number of discrete matrices based on time after data processing, so the computing power requirements of the equipment are extremely high.

Specifically, change the paddlepaddle framework to paddle inference for ARM devices, download the scikit-learn source code from GitHub, use the make tool to compile the ARM version, and configure other environments. Directly running the text recognition model will show a memory overflow. This is due to the low computing power of the device's main control and its inability to perform large-scale matrix operations. After replacing the beam search algorithm used in the original project with a greedy search algorithm that occupies less memory, the text recognition model ran successfully. It took 3 seconds to recognize about 30 words of speech, which met the requirements.

(2) The speech emotion recognition sub-module is used to perform emotion recognition on the voice input by the user based on the emotion recognition model to determine the user's emotion category. Specifically, the emotion categories include: angry, scared, happy, nervous, sad, and surprised.

The emotion recognition module is obtained by pre-training the voiceprint recognition model using the voice emotion data set on the personal computer. The voice emotion data set includes a plurality of voice samples and emotion categories corresponding to each voice sample.

The present invention improves on the basis of ECAPA-TDNN, obtains a voiceprint recognition model, and implements basic voice emotion recognition functions based on PaddlePaddle.

As shown in Figure 8, the voiceprint recognition model includes the first fully connected layer, the reshaping layer, the bidirectional LSTM layer, the Tanh activation layer, the discarding layer, the second fully connected layer, the ReLU activation layer and the third fully connected layer. .

The voiceprint recognition model mainly performs feature extraction and classification on the input data. After the data is input, it undergoes linear transformation through the first fully connected layer; the transformed data is reshaped into a three-dimensional shape by the reshaping layer to adapt to the input of the bidirectional LSTM layer. Requirements; the reshaped data is sent to the bidirectional LSTM layer for timing processing. Bidirectional LSTM will improve the fitting ability of the voiceprint recognition model; the output of the bidirectional LSTM layer is activated through the Tanh activation layer; after passing through the Tanh activation layer, the data is output Random discarding is performed through the discarding layer; the output data of the discarding layer enters the second fully connected layer for linear transformation; the transformed data undergoes nonlinear processing through the ReLU activation layer; after passing through the ReLU activation layer, the output data passes through the third fully connected layer. The final linear transformation; finally, the voiceprint recognition model outputs the prediction score for each emotion category.

Due to the poor quality of the open source voice emotion data sets, the present invention classifies and data-enhances three open source voice emotion data sets as the voice emotion data set of the voiceprint recognition module. The number of the enhanced voice emotion data sets and The quality has been greatly improved, meeting the training requirements of the voiceprint recognition model.

Specifically, data preprocessing is performed first, and the speech emotion data set is divided into a training set and a test set according to a certain proportion, and the files are traversed to form label information corresponding to the path and file name; a normalized model is generated for training. Training setting batch_size=128, training for 200 rounds. A log log will be generated during training. You can use the Paddle log viewing tool to view the training situation in real time. Figure 9 shows the loss of the training set. Using the cross-entropy loss function, training for 200 rounds, the optimal loss is 0.22; Figure 10 and Figure 11 are Changes in the accuracy of the general performance index of the classification function. The accuracy refers to the ratio of the number of samples correctly predicted by the voiceprint recognition model on the test set to the total number of samples. It is used to measure the classification accuracy of the voiceprint recognition model in the classification task. , accuracy = (number of correctly predicted samples)/(total number of samples). After 200 rounds of training, the accuracy of the training set can reach 0.87, and the accuracy of the test set can reach 0.90; Figure 12 is the confusion matrix of the voiceprint recognition model, showing the difference between the prediction results of the voiceprint recognition model on the test set and the real labels The corresponding relationship, as well as the accuracy and error of classification, can be seen from Figure 9 to Figure 12 that, except for the fear emotion, the accuracy of the other five emotions has reached more than 90%. Since the fear emotion data set is relatively small, the accuracy The rate is 43%. The training effect meets the requirements of the invention.

In the present invention, speech emotion recognition is also based on paddle, so it is enough to add the corresponding Python environment directly on the basis of the speech and text recognition environment. After the emotion recognition model is successfully deployed, the speech emotion can be successfully recognized after testing, which takes 2 seconds and satisfies this requirement. The need for invention.

The speech synthesis module is connected to the human-computer interaction module. The speech synthesis module is used to generate multi-style speech according to the speech input by the user based on the speech synthesis model. Specifically, the speech synthesis model is obtained by pre-training the fastspeech2 model using a multi-speaker data set and then converting it into a dynamic graph model using the Paddld tool.

In the present invention, the multi-style speech synthesis uses the fastspeech2 model based on the Paddle framework, the vocoder uses the Parallel WaveGAN pre-training model, and the AISHELL3 multi-speaker speech database is used for training to realize multi-style speech synthesis. fastSpeech2 is an end-to-end speech synthesis model published in 2020. Its core idea is to use an autoregressive Transformer model to generate acoustic features, and then convert these features into audio waveforms through a vocoder. It is the current mainstream TTS (Text-to- Speech, text-to-speech) model. This invention uses the PaddleSpeech project to implement the fastSpeech2 function. The speech synthesis model using fastspeech2 provides a variety of speaking styles, and you can choose a voice style according to your needs.

The present invention implements the PaddleSpeech project according to the process shown in Figure 13, and uses the AISHELL-3 multi-speaker data set for training. Since the data set is large, direct training consumes a lot of resources, so it is based on the pre-training model fastspeech2_aishell_ckpt_1.1.0 Make fine adjustments. Among them, the vocoder uses the ParallelWaveGAN pre-trained model pwg_aishell3_ckpt_0.5. When predicting, you need to specify the speaker ID according to the AISHLL-3 data set description to achieve multi-speaker speech synthesis. Figure 14 is the mel spectrogram of the real sound in the AISHELL-3 data set, and Figure 15 is the mel spectrogram of the fastspeech2 synthesized sound. It can be seen that the timbre of the synthesized sound is more realistic. In addition, the speech synthesis model provides more than 100 styles of speech, satisfying Requirements for multi-style speech synthesis.

The deployment of the speech synthesis model is the same as the text recognition model and emotion recognition model, and requires local compilation and model optimization in the ARM environment. In the speech synthesis module, the original fastspeech2 is a dynamic graph model. The advantage of dynamic graphs is that you can see the values of variables when building the network, which is easy to check. The disadvantage is that the forward operation is not easy to optimize because you do not know what to calculate next. , since a new calculation graph will be rebuilt every time it is run, it will also occupy a larger amount of memory. Static graphs can optimize the graph structure before running, such as constant folding, operator fusion, etc., to obtain faster results. Forward operation speed, therefore the present invention uses the Paddld tool set to convert the dynamic graph model of the project into a static graph model. After optimization, the speech synthesis model can run successfully, and it takes about 5 seconds to synthesize a speech of about 10 seconds.

In order to solve the problem of insufficient intelligence of traditional interactive devices, the present invention introduces deep learning and uses SOC master control to realize intelligent voice interaction. Adding emotion recognition to speech recognition can recognize text and emotions based on speech, making voice interaction closer to actual communication scenarios. The HMI host computer was made using the serial port screen, and the UART function was used to realize the host computer control device. The interaction was more intuitive and convenient, and it was easy to apply and promote in actual projects.

This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and the core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the present invention There will be changes in the specific implementation methods and application scope of the ideas. In summary, the contents of this description should not be construed as limitations of the present invention.

Claims (7)

1. A multifunctional voice interaction device based on deep learning, characterized in that the multifunctional voice interaction device based on deep learning includes: The human-computer interaction module is used to provide users with a human-computer interaction interface, receive voice and instructions input by the user, display voice text and emotion categories, and play multi-style voices; A speech recognition module, connected to the human-computer interaction module, is used to perform text recognition and emotion recognition on the speech input by the user based on a deep learning model to obtain speech text and emotion categories; A speech synthesis module, connected to the human-computer interaction module, is used to generate multi-style speech according to the speech input by the user based on the speech synthesis model. 2. The multifunctional voice interaction device based on deep learning according to claim 1, characterized in that the human-computer interaction module, the speech recognition module and the speech synthesis module are all constructed based on the Jetson Nano SOC main control chip. . 3. The multifunctional voice interaction device based on deep learning according to claim 1, wherein the human-computer interaction module includes an HMI configuration screen, a microphone and a speaker. 4. The multifunctional voice interaction device based on deep learning according to claim 1, characterized in that the voice recognition module includes: The speech and text recognition sub-module is used to perform text recognition on the speech input by the user based on the text recognition model; the text recognition model is obtained by pre-training the Efficient Conformer model using the speech and text data set on the personal computer; the speech and text data The collection includes multiple voice samples and the text corresponding to each voice sample; The voice emotion recognition sub-module is used to perform emotion recognition on the voice input by the user based on the emotion recognition model to determine the user's emotion category; the emotion recognition module uses the voice emotion data set on the personal computer in advance to analyze the voiceprint. The recognition model is trained; the voice emotion data set includes multiple voice samples and the emotion categories corresponding to each voice sample: the voiceprint recognition model includes a first fully connected layer, a reshaping layer, and a bidirectional LSTM layer that are connected in sequence. , Tanh activation layer, dropout layer, second fully connected layer, ReLU activation layer and third fully connected layer. 5. The multifunctional voice interaction device based on deep learning according to claim 4, wherein the text recognition model includes a CNN front end, multiple LAC encoders and multiple LAC decoders connected in sequence; The CNN front-end includes a first convolutional neural network, a second convolutional neural network and an embedded neural network connected in sequence; Each LAC encoder includes a first low-rank feedforward neural network, a first multi-head attention layer, a third convolutional neural network and a second low-rank feedforward neural network connected in sequence; Each LAC decoder consists of a masked multi-head attention layer, a first normalization layer, a second multi-head attention layer, a second normalization layer, a third low-rank feedforward neural network and a third normalization layer connected in sequence. One layer. 6. The multifunctional voice interaction device based on deep learning according to claim 4, characterized in that the speech text data set is preprocessed using a linear energy map to Chinese female voice database AISHELL-1, and the ffmpeg tool is used It is obtained by enhancing noise, enhancing speech rate, enhancing volume, excursion enhancement and spectrum enhancement. 7. The multifunctional voice interaction device based on deep learning according to claim 1, characterized in that the speech synthesis model is a fastspeech2 model trained using a multi-speaker data set in advance, and then converted into a dynamic graph using the Paddld tool. obtained by the model.