CN118692444A - Method, device, equipment, and medium for generating prompt audio - Google Patents

Abstract

The present application discloses a method, device, equipment, and medium for generating prompt audio, which relates to the field of automobile technology. The method is executed by a vehicle-mounted terminal of a first vehicle, and includes the following steps: collecting voice data of a first subject, the first subject is located in the first vehicle; analyzing the voice data to obtain emotional state information of the first subject, the emotional state information is used to indicate the emotional state of the first subject when expressing the voice data; generating corresponding prompt information based on the emotional state information, the prompt information is used to indicate providing emotional feedback to the first subject; converting the prompt information into prompt audio, and playing the prompt audio. It is possible to improve the accuracy of the recognition result by collecting the driver's voice data and analyzing the driver's current emotional state. Providing corresponding emotional feedback to the driver helps the driver to adjust his emotions in time, and improves the driver's driving experience and safety during driving.

Description

Method, device, equipment, and medium for generating prompt audio

Technical Field

The embodiments of the present application relate to the field of automobile technology, and in particular to a method, device, equipment, and medium for generating prompt audio.

Background Art

In the field of modern smart cars, the research and development of vehicle systems is moving towards a more personalized and intelligent direction. Smart cars can identify the driver's emotions in a timely manner and provide corresponding comfort measures, which can enhance the emotional connection between the driver and the vehicle, improve the overall driving experience and safety during driving.

In related technologies, image processing and machine learning algorithms are used to analyze the driver's facial expressions to identify emotions and provide personalized comfort measures accordingly.

However, changes in ambient light, the complexity of facial expressions, and the high requirements for real-time processing speed may affect the accuracy of emotion recognition and affect the vehicle's comforting effect on the driver.

Summary of the invention

The embodiment of the present application provides a method, device, equipment, and medium for generating a prompt audio, which can timely identify the emotional state of the driver and generate a prompt audio to provide emotional feedback to the driver. The technical solution is as follows:

In one aspect, a method for generating a prompt audio is provided, which is executed by an onboard terminal of a first vehicle, and the method includes:

collecting voice data of a first subject, the first subject being located in the first vehicle;

Analyzing the voice data to obtain emotional state information of the first subject, wherein the emotional state information is used to indicate an emotional state of the first subject when expressing the voice data;

generating corresponding prompt information based on the emotional state information, wherein the prompt information is used to instruct to provide emotional feedback to the first subject;

The prompt information is converted into a prompt audio, and the prompt audio is played.

On the other hand, a device for generating a prompt audio is provided, the device comprising:

a collection module, configured to collect voice data of a first subject, the first subject being located in the first vehicle;

an analysis module, configured to analyze the voice data to obtain the emotional state information of the first subject, wherein the emotional state information is used to indicate the emotional state of the first subject when expressing the voice data;

A generating module, configured to generate corresponding prompt information based on the emotional state information, wherein the prompt information is used to instruct to provide emotional feedback to the first subject;

The conversion module is used to convert the prompt information into a prompt audio and play the prompt audio.

In an optional embodiment, the analysis module is also used to input the voice data into a pre-trained deep learning model, perform sentiment analysis on the voice data through the deep learning model, obtain and output an analysis result, and the analysis result includes at least one candidate emotional state; collect driving information of the first vehicle, the driving information of the first vehicle includes vehicle operation information of the first vehicle and environmental information of the environment in which the first vehicle is located; based on the analysis result and the driving information, determine the emotional state information of the first subject from the candidate emotional states.

In an optional embodiment, the analysis module is also used to obtain an emotional state comparison table, which contains the correspondence between the driving information of the first vehicle and at least one emotional state; based on the driving information, determine a first correspondence matching the driving information from the emotional state comparison table; based on the emotional state indicated by the first correspondence, determine the emotional state information of the first subject from the candidate emotional states.

In an optional embodiment, before the analysis module, the device further includes:

A training module is used to obtain sample speech data, wherein the sample speech data is marked with an emotion tag, and the emotion tag is used to indicate the emotional state expressed by the sample speech data; based on the sample speech data, a pre-trained model is trained to obtain the pre-trained deep learning model.

In an optional embodiment, the training module is also used to model the time series of the sample speech data through the pre-trained model, determine the context information and time dependency of the sample speech data, and the time dependency refers to the characteristics of the sample speech data in the time series; output a predicted emotional state based on the context information and the time dependency of the sample speech data; and adjust the pre-trained model based on the training results and the emotional markers to obtain the pre-trained deep learning model.

In an optional embodiment, the generation module is also used to input the emotional state information into a pre-trained natural language model, identify and analyze the emotional state information through the natural language model, and determine the emotional type when the vehicle-mounted terminal provides feedback to the first subject; generate text prompt information that conforms to the emotional type, and the text prompt information contains emotional feedback content in text form.

In an optional embodiment, the generation module is also used to obtain historical text prompt information generated by the natural language model within a historical time period, and the historical text prompt information is used to indicate the emotional feedback content provided by the vehicle-mounted terminal to the first subject within the historical time period; based on the emotional type, information that meets the emotional type is filtered out from the historical text prompt information as the text prompt information.

On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a method for generating a prompt audio as described in any of the above-mentioned embodiments of the present application.

On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a method for generating a prompt audio as described in any of the above-mentioned embodiments of the present application.

On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating a prompt audio as described in any of the above embodiments.

The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

By collecting the driver's voice data and analyzing the driver's current emotional state, we can provide the driver with corresponding emotional feedback, help the driver adjust his emotions in time, and improve the driver's driving experience and safety during driving. Since voice is continuous, as the tone and volume in the voice data change continuously, it can better reflect the driver's true emotions. Compared with the related technology of collecting the driver's facial expressions to identify the driver's emotions, driver emotion recognition based on voice data can improve the accuracy of the recognition results and provide more appropriate emotional feedback.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of a system for generating a prompt audio according to an exemplary embodiment of the present application;

FIG2 is a schematic diagram of a prompt audio generation process provided by an exemplary embodiment of the present application;

FIG3 is a flow chart of a method for generating a prompt audio according to an exemplary embodiment of the present application;

FIG4 is a structural block diagram of a device for generating a prompt audio according to an exemplary embodiment of the present application;

FIG5 is a structural block diagram of a device for generating a prompt audio provided by another exemplary embodiment of the present application;

FIG. 6 is a structural block diagram of a computer device provided by an exemplary embodiment of the present application.

DETAILED DESCRIPTION

In order to make the objectives, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

It should be noted that the information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

It should be understood that, although the terms first, second, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

First, a brief introduction is given to the terms involved in the embodiments of the present application:

Recurrent Neural Network (RNN): is an artificial neural network suitable for processing sequence data. RNNs are able to process dynamic features in sequences, that is, they can remember previously input information and use this information to influence the current output. For example, RNNs can convert speech signals into text, learn the temporal features of speech signals, and predict the corresponding text sequence. By analyzing the pitch, volume, speed and other characteristics of speech, RNNs can also be used to identify the speaker's emotional state, such as happiness, sadness, anger, etc.

Loss function: Also known as cost function or objective function, it is a function that measures the difference between the model's predicted value and the actual value. In machine learning and deep learning, the loss function is a core part of the training process, used to guide the learning direction of the model, that is, to optimize the model parameters by minimizing the loss function.

Among them, there are various types of loss functions, such as the cross entropy loss function. In practice, the corresponding loss function can be selected to train the model according to the application scenario.

Optimizer: In machine learning, it is an algorithm used to adjust model parameters in order to minimize the loss function. The optimizer iteratively updates the model parameters to optimize the model's performance on the training data.

In the modern automotive industry, the innovation of in-vehicle systems is constantly moving towards personalization and intelligence. Smart cars use advanced technology to identify the driver's emotional state and provide customized comfort measures to enhance the driver's driving experience and driving safety.

When a car recognizes a driver's emotions, it usually relies on image processing and machine learning algorithms to collect facial images of the driver and analyze the driver's facial expressions to identify his or her emotional state. For example, if the system recognizes that the driver is nervous or anxious, it may provide some soothing measures, such as adjusting the air conditioning temperature or playing soft natural sounds.

However, the above methods face a series of challenges in practical applications. Changes in ambient light may affect the accuracy of facial expression recognition. Facial features may be difficult to capture under strong or weak light conditions, resulting in misjudgment by the emotion recognition system. In addition, people express emotions in a variety of ways, and sometimes subtle changes in expression may be difficult for the algorithm to accurately capture.

The present application provides a method for generating a prompt audio, which can analyze the driver's emotional state based on the driver's voice data to improve the accuracy of emotion recognition results. It also generates corresponding prompt audio to provide emotional feedback to the driver, thereby avoiding accidents caused by the driver's emotional state during the process of controlling the vehicle.

Secondly, the prompt audio generation system involved in the embodiment of the present application is explained. For schematic illustration, please refer to Figure 1. The implementation environment involves a first vehicle 100 and a server 120. The first vehicle 100 and the server 120 are connected via a communication network 140.

The first vehicle 100 is equipped with an onboard terminal and a voice collection component, wherein the voice collection component is used to collect the driver's voice data when the driver authorizes it to be turned on, and the driver's voice data includes the driver's voice. After collecting the voice data, the onboard terminal sends the voice data to the server 120 through the communication network 140, and the server 120 analyzes and processes the voice data to identify the driver's emotional state reflected by the voice data, and generates a prompt audio correspondingly and returns it to the onboard terminal, wherein the prompt audio is used to provide emotional feedback to the driver, for example, to comfort the driver and calm the driver down.

The server 120 includes a deep learning model, which inputs the voice data into the deep learning model and outputs corresponding emotional state information, where the emotional state information is used to indicate the emotional state of the driver.

The server 120 generates corresponding prompt information based on the emotional state information, synthesizes the prompt information into prompt audio, and returns it to the vehicle terminal.

The vehicle terminal plays prompt audio in a timely manner to provide emotional feedback to the driver. For example, when the emotional state information indicates that the driver is in a sad state, the prompt audio is used to comfort the driver.

In some embodiments, the above process can be regarded as multiple processing links, which are jointly executed by the vehicle terminal and the server.

Optionally, as shown in FIG. 2 , the following links are included: a voice data collection link 210 , an emotion recognition link 220 , an emotion analysis link 230 , an audio synthesis link 240 and an audio output link 250 .

Voice data collection link 210: used to collect the driver's voice data inside the car or during driving, which can be achieved through a car microphone or other voice collection equipment. In some embodiments, this link also includes a pre-processing step of the voice data, such as removing environmental noise, determining valid voice segments, etc., and the valid voice segment refers to the voice segment containing the driver's voice.

Emotion recognition link 220: It realizes the recognition and classification of the driver's voice emotional state, and usually adopts deep learning models for emotion recognition, such as recurrent neural networks or convolutional neural networks, etc. The input of this link is voice data, and the output is the corresponding emotional state information, such as happiness, frustration, anxiety, etc.

Emotional analysis section 230: further analyzes and processes the driver's emotional state information identified by the emotion recognition section 220. In some embodiments, this section combines other information (such as vehicle speed, traffic conditions in the vehicle's environment, etc.) to comprehensively analyze the driver's emotional state.

Audio synthesis link 240: Based on the results of emotion recognition and analysis, select appropriate voice comfort content and use speech synthesis technology to generate corresponding prompt audio. This link can convert text into natural and fluent speech to provide specific voice feedback such as comfort, encouragement, and reminder.

The audio output link 250 is responsible for outputting the generated speech synthesis content to the driver, which can be output through the vehicle audio system or the headphones worn by the driver to achieve emotional comfort and guidance functions for the driver.

It is worth noting that the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

In some embodiments, the above server can also be implemented as a node in a blockchain system.

In combination with the above-mentioned noun introduction and application scenarios, the method for generating prompt audio provided by the present application is described. The method can be executed by a server or a vehicle-mounted terminal, or can be executed by a server and a vehicle-mounted terminal together. In the embodiment of the present application, the method is described by taking the vehicle-mounted terminal of the first vehicle as an example, as shown in FIG3 , which is a flow chart of the method for generating prompt audio provided by an exemplary embodiment of the present application. The method comprises the following steps.

Step 310: Collect voice data of the first subject.

The first subject is located in the first vehicle, and the first subject may be a driver or an occupant in the first vehicle.

Optionally, the first vehicle is equipped with a component capable of collecting audio data inside the vehicle, wherein when a subject inside the first vehicle makes a sound, the audio data includes voice data. For example, the first vehicle is equipped with a microphone array composed of multiple microphones, which can capture sounds inside the first vehicle and determine the source of the sounds based on sound localization technology.

The voice collection component may be turned on in the following ways, including but not limited to: (1) the first subject operates the switch of the voice collection component to turn it on; (2) the first subject uses other terminals or devices with which the voice collection component is bound to turn it on; (3) the first subject turns it on through voice commands, for example, the voice collection component has a voice control function.

Voice data refers to data obtained by the voice collection component after collecting voice signals inside the vehicle when the voice collection component function is turned on by the subject in the first vehicle. When the voice collection component is not turned on, the voice signals in the first vehicle cannot be collected.

In some embodiments, the voice collection component remains in an on state continuously, and when there is a subject in the first vehicle, the voice collection component will actively play a prompt voice to solicit voice collection permission from the subject in the first vehicle.

For example, there is a sensor in the first vehicle, and the sensor can detect whether there is a first subject in the first vehicle. When it is detected that the subject has entered the first vehicle, the vehicle-mounted terminal automatically controls the voice collection component to play a prompt voice: "Do you want to enable voice collection permission?" If the subject in the vehicle agrees to enable voice collection permission through voice commands or other means, the voice collection component continues to remain on; if the subject in the vehicle refuses to enable voice collection permission through voice commands or other means, the voice collection component is automatically closed.

Optionally, the first subject is a driver as an example for explanation. The voice data of the first subject includes the human voice of the first subject, that is, the language sound produced by the vocal cords of the first subject, such as speaking, reading aloud, singing, etc. The voice data can reflect the characteristics of the first subject when speaking, such as intonation, volume, speed and continuity.

Optionally, the voice data also includes other sounds emitted by the first subject, such as laughter, crying, sighing, coughing, etc. These sounds can reflect the physical health and emotional state of the first subject.

In some embodiments, the voice data collected by the voice collection component may include, in addition to the sound emitted by the first subject, sounds emitted by other sources, such as ambient noise in the environment in which the first vehicle is located, sounds emitted by electronic equipment inside the first vehicle when in operation, etc.

When performing speech analysis, such as sentiment analysis, noise removal helps to more accurately capture the key features of the first subject in the speech data. In order to highlight the sound directly from the first subject in the speech data and improve the accuracy of subsequent recognition and analysis of the speech data, the vehicle-mounted terminal performs noise reduction processing on the speech data after the speech collection component completes the collection, so as to remove the noise in the environment and determine the effective speech segments in the speech data.

Step 320: Analyze the voice data to obtain the emotional state information of the first subject.

The emotional state information is used to indicate the emotional state of the first subject when expressing the voice data.

For example, the voice data is data collected when the first subject sings, and the emotional state information obtained after analyzing the voice data is used to indicate the emotional state of the first subject when singing, such as excitement, sadness, excitement, joy, etc.

Optionally, the speech data is input into a pre-trained deep learning model, and sentiment analysis is performed on the speech data through the deep learning model to obtain and output analysis results, which include at least one candidate emotional state.

Exemplarily, the deep learning model refers to a recurrent neural network model, the structure of which includes an input layer, a hidden layer, and an output layer.

Among them, the input layer is the first layer of the neural network to receive raw data. After the voice data is input into the recurrent neural network model, the voice data first enters the input layer, and the input layer maps the voice data to corresponding numerical features so that the recurrent neural network can process the data. The hidden layer is used to extract features from the input data, combine simple features into more complex features, help the neural network understand the input data, learn the importance of each feature in the input data, adjust the neural network to achieve the purpose of optimizing performance, and pass the processed data to the next layer. The output layer converts the information transmitted by the hidden layer into the final prediction result of the model.

That is, after the deep learning model processes the speech data through at least the above three structural layers, the output prediction data is the analysis result.

Exemplarily, the voice data is taken as data collected when the first subject makes a voice call.

The content of the first subject's call is collected after the first subject authorizes the voice collection component, as follows: "The traffic is too congested. I should have been home in 20 minutes, but now it seems that I will be at least two hours late."

Among them, the first subject reflected in the voice data has a high volume, a high pitch, and a fast speaking speed. Then, the voice data is input into a pre-trained deep learning model, which converts the call content in the voice data into text content, and performs natural language processing on it to analyze the emotional tendency of the text content. The pitch reflected in the voice data is used to identify the high-pitched changes of the first subject, and the volume reflected in the voice data is used to evaluate whether the first subject is emotional. Combined with the text content, the emotional state of the first subject is analyzed to obtain at least one candidate emotion.

The above analysis shows that the volume of the first subject has increased significantly, the speaking speed has increased, the tone has become sharper and the intonation has fluctuated greatly. At the same time, the text content contains words that express dissatisfaction and anxiety.

From the text content, it can be seen that the first subject is stuck in a traffic jam and is anxious about the possibility of returning home late. Exemplarily, the analysis results output by the deep learning model include the following candidate emotional states: anxiety, anger, irritation, fatigue, and impatience.

Optionally, driving information of the first vehicle is collected, where the driving information of the first vehicle includes vehicle operation information of the first vehicle and environmental information of an environment in which the first vehicle is located.

Based on the analysis result and the driving information, emotional state information of the first subject is determined from the candidate emotional states.

For example, the environment information indicates that the road where the first vehicle is located is relatively congested, there are a large number of vehicles on the road, and the following distance between the first vehicle and other vehicles is relatively close. The vehicle operation information indicates that the first vehicle is in a closed state for 5 minutes.

Then, the above driving information indicates that the road section where the first vehicle is located remains stationary for a short period of time, and the first vehicle is in a closed state in order to save vehicle energy consumption elements. Therefore, the analysis result and the driving information both indicate that the emotional state of the first subject is impatience and anger.

Exemplarily, an emotional state comparison table is obtained, wherein the emotional state comparison table contains a correspondence between driving information of the first vehicle and at least one emotional state.

Based on the driving information, a first corresponding relationship matching the driving information is determined from the emotion state comparison table.

Based on the emotional state indicated by the first corresponding relationship, emotional state information of the first subject is determined from the candidate emotional states.

Schematically, as shown in Table 1 below, Table 1 is a schematic diagram of an emotional state comparison table, which is used to represent the correspondence between driving information and emotional states, wherein the driving information is used to indicate the situations/road conditions encountered by the driver when driving the first vehicle.

Table 1

For example, if the current driving information indicates that the first vehicle remains stationary for 5 minutes, it means that the road condition at this time is "traffic stagnation", and if the first subject's voice tone is determined to be rising during the analysis of the voice data, the first subject's emotional state information is very likely to be "outburst of anger". Therefore, based on the driving information, the first corresponding relationship matching the driving information is determined from the emotional state comparison table as follows: traffic stagnation-outburst of anger.

Among them, at least one candidate emotional state includes the following emotional states: anxiety, anger, irritability, fatigue, and impatience. Based on the first corresponding relationship, "anger" is screened out from the candidate emotional states and determined as the emotional state information of the first subject.

The types of deep learning models include, but are not limited to, recurrent neural networks and convolutional neural networks. Taking recurrent neural networks as an example, the training process of the model is as follows: sample speech data is obtained, the sample speech data is annotated with emotion tags, and the emotion tags are used to indicate the emotional state expressed by the sample speech data. The pre-trained model is trained based on the sample speech data to obtain a pre-trained deep learning model.

For example, the sample speech data includes multiple sub-data, each sub-data corresponds to a different emotion tag, including the first sub-data.

For example, the first sub-data is the content of a conversation when a user encounters an elevator failure and asks for help from a staff member via the telephone in the elevator. The emotion tags marked in the first sub-data are: nervousness and fear.

It is worth noting that, since the emotional state of human beings is not singular in most cases, each sub-data can correspond to at least one emotional tag, so that the deep learning model can learn human emotions more comprehensively and improve the accuracy of analysis and recognition. When training the model based on the first sub-data, multiple sub-data that contain the same emotional tag can be combined to specifically train the model's ability to recognize the emotion indicated by the emotional tag. For example, sub-data 1 contains tags A, B, and C, and sub-data 2 contains tags A and D. Sub-data 1 and sub-data 2 are used as input data to train the model's ability to recognize emotional tag A. Different emotional tags can also be marked on the same sub-data, and the model can be trained to recognize the emotion indicated by one of the emotional tags by relying on only one sub-data at a time. This embodiment does not limit this.

Optionally, the time series of the sample speech data is modeled by a pre-trained model to determine the context information and time dependency of the sample speech data, where the time dependency refers to the characteristics of the sample speech data in the time series.

The predicted emotional state is output based on the contextual information and time dependency of the sample speech data.

The pre-trained model is adjusted based on the training results and sentiment labels to obtain a pre-trained deep learning model.

Exemplarily, sample speech data with emotion tags are prepared in advance, including speech samples of different emotional states, and a recurrent neural network model is constructed to capture the temporal dependency in the sample speech data through the ability of sequence learning.

Among them, the temporal dependency of speech data refers to the interrelationship of speech signals in time series, which is reflected in multiple levels, such as: (1) Phoneme dependency: Phonemes in speech (such as consonants and vowels) usually do not exist in isolation. The pronunciation of a phoneme is often affected by the previous and next phonemes. This phenomenon is called liaison or assimilation; (2) Phrase and sentence dependency: At the phrase or sentence level, speech features such as intonation, rhythm, and intensity are affected by sentence structure and semantic content, showing temporal coherence and variability; (3) Emotional expression: The emotional state in speech, such as happiness, sadness, anger, etc., is expressed through speech volume, pitch, speaking speed and other features over a period of time, showing temporal continuity; (4) Long-term dependency: In longer speech passages, such as speeches or conversations, certain features of speech may remain consistent or change gradually throughout the passage, showing long-term temporal dependency.

The structure of the recurrent neural network includes an input layer, a hidden layer, and an output layer. During the training process, the recurrent neural network models the time series of speech data to help the model capture contextual information and long-term dependencies in the sample speech data.

The recurrent neural network model is trained using sample speech data labeled with emotional tags, and the appropriate loss function (such as the cross entropy loss function) and optimizer (such as the Adam optimizer) are used for parameter optimization. The model parameters are continuously updated through the back-propagation algorithm to minimize the value of the loss function. Finally, the trained model is evaluated through the test set to determine the classification performance of the model on the emotion classification task. The test set also contains multiple sample speech data with known emotional states/tags in advance.

Step 330: Generate corresponding prompt information based on the emotional state information.

The prompt information is used to instruct to provide emotional feedback to the first subject.

Exemplarily, the types of prompt information include, but are not limited to: text prompt information, voice prompt information, animation prompt information, picture prompt information, etc. This embodiment takes the prompt information as the prompt information of this article as an example for description.

The emotional state information is input into a pre-trained natural language model, and the emotional state information is identified and analyzed through the natural language model to determine the emotional type when the vehicle-mounted terminal provides feedback to the first subject.

Generate text prompt information that matches the emotion type, and the text prompt information contains emotional feedback content in text form.

Among them, Natural Language Model (NLM) is a computational model used to process, understand and generate human language (i.e. natural language). These models are designed to simulate the way humans understand and use language, enabling computers to perform various language-related tasks. Natural language models can understand the structure and meaning of text content, perform sentiment analysis on text content, identify the emotional tendencies in text content, such as positive, negative or neutral, and generate natural and fluent language text as feedback.

In some embodiments, the text prompt information may be pre-prepared text content, and there is a correspondence between these text contents and different emotional state information. For example, when the emotional state information indicates that the first subject is in a depressed or sad state, the text prompt information corresponds to text content that can give comfort and encouragement to the first subject.

Alternatively, the text prompt information can also be determined from the text prompt information generated by the natural language model in the historical time period. Directly using the prompt information that has been generated in the historical time period according to the emotional state information can improve the efficiency of the natural language model in generating feedback content.

Optionally, historical text prompt information generated by a natural language model within a historical time period is obtained, and the historical text prompt information is used to indicate emotional feedback content provided by the vehicle-mounted terminal to the first subject within the historical time period.

Based on the emotion type, the information that matches the emotion type is filtered out from the historical text prompt information as the prompt information of this article.

In some embodiments, the vehicle-mounted terminal has network connection capability and can access and obtain multimedia content that has been made public on the Internet platform. The emotional state information is input into a pre-trained natural language model, and the emotional state information is identified and analyzed by the natural language model. After determining the emotional type when the vehicle-mounted terminal provides feedback to the first subject, the vehicle-mounted terminal automatically indexes the Internet platform based on the feedback emotional type, collects stories, poems, scripts, news and other text content that matches the feedback emotional type, and obtains text prompt information based on the text content. For example, after summarizing the text content, text prompt information is obtained; or, after extracting keywords from the text content, text prompt information is automatically generated based on the extracted keywords, wherein keywords can be words that appear more frequently in the text content and words that summarize the text content.

In some embodiments, a social account of the first subject is logged into the on-board terminal of the first vehicle. The social account is an account associated with the information system, service or application of the first vehicle. The first subject can call the services and functions inside the first vehicle by logging into the social account.

Optionally, there is an association between the social account of the first subject and other social accounts. For example, the other social accounts are accounts logged in by the first subject on the Internet application platform, or the other social accounts are accounts logged in by the second subject on the Internet application platform or other vehicles. The other social accounts and the social account of the first subject are friend accounts of each other.

Taking the example of the second subject logging in to a friend account, after the deep learning model outputs the emotional status information of the first subject, the emotional status information is sent to the friend account. The second subject edits the prompt information based on the emotional status information and feeds it back to the first subject's social account through the friend account. The in-vehicle terminal receives the prompt information fed back by the second subject.

For example, the emotional state information indicates that the first subject is in a depressed state, and the prompt information fed back by the second subject through the social account is an audio with comforting emotional characteristics, which contains the voice of the second subject.

In some embodiments, after the deep learning model outputs the emotional state information of the first subject, the vehicle terminal generates adjustment suggestions for the first vehicle as prompt information based on the collected vehicle operation information. The vehicle operation information also includes the air temperature and humidity in the first vehicle, the working status of various vehicle-mounted devices in the first vehicle, such as the brightness of the interior lighting equipment, whether the vehicle air conditioner is turned on, whether the window is open, etc.

For example, the emotional state information indicates that the first subject is in an angry state due to traffic congestion, and the vehicle operation information indicates that the air temperature inside the first vehicle is 30°C, the vehicle air conditioner is off, and the windows are open. Then the vehicle terminal automatically generates the following suggestions: Close the windows and turn on the air conditioner in the car. Based on the above suggestions, the prompt information is as follows: The road is congested and the noise outside the car is loud, please close the windows, and the temperature inside the vehicle is high, please turn on the air conditioner in the car.

Step 340: convert the prompt information into prompt audio, and play the prompt audio.

Among them, the prompt information is used as an example to illustrate that the process of converting text content into audio content is usually called text-to-speech (TTS) conversion, which mainly includes the following steps: (1) Text preprocessing: segment the text; (2) Voice selection: select a pre-recorded voice or a synthesized voice, which can be male or female, or have different age characteristics; (3) Phoneme synthesis: decompose the text into phonemes or phonetic symbols; (4) Rhythm generation: determine the tone, rhythm, intensity, etc. of the sentence; (5) Speech synthesis: use the TTS engine to convert phonemes into audio signals;

(6) Audio output: Output the processed audio to a common audio format.

In some embodiments, the emotional state of the first subject may change in a short period of time. Therefore, after the prompt audio is generated, the emotional changes of the first subject can be reflected when the voice data of the first subject is collected in real time according to the voice collection component and emotional analysis is performed. If the emotional state information of the first subject at the current moment still conforms to the generated prompt audio, the prompt audio can be played directly. If the emotional state information of the first subject at the current moment has changed, the corresponding prompt audio can be regenerated in combination with the changes in the emotional state information of the first subject at the current moment and at historical moments.

In some embodiments, when there are multiple subjects in the first vehicle, the voice data collected by the voice collection component is likely to contain conversation content between the multiple subjects in the vehicle. Depending on the content of the conversation, it is also possible to generate and play prompt audio suitable for providing emotional feedback to multiple subjects by performing sentiment analysis on the voice data.

For example, after the pre-trained deep learning model analyzes the voice data and determines that there are two parties arguing in the first vehicle, the prompt audio played by the car audio should be neutral, peaceful, and helpful in easing the tense atmosphere. For example, a calm reminder audio: "Try to take a deep breath and calm down before continuing the conversation"; an audio that emphasizes safety: "Safe driving is the most important thing. If your current emotions affect your driving, please consider stopping and taking a rest first."

It is worth noting that when providing emotional feedback for different situations, it is necessary to select appropriate audio based on the specific scenario. In addition to generating corresponding audio based on the collected voice data after analysis, it is also possible to achieve the effect of comforting the subject inside the vehicle by playing locally stored music. For example, playing music with a gentle rhythm can help the subject inside the vehicle calm down and stay calm.

In summary, the method for generating prompt audio provided by the present application can provide corresponding emotional feedback to the driver by collecting the driver's voice data and analyzing the driver's current emotional state, thereby helping the driver to adjust his emotions in a timely manner and improving the driver's driving experience and safety during driving. Since voice is continuous, as the tone, volume, etc. in the voice data continue to change, it can better reflect the driver's true emotions. Compared with the method of collecting the driver's facial expressions to identify the driver's emotions in related technologies, driver emotion recognition based on voice data can improve the accuracy of the recognition results and provide more appropriate emotional feedback.

FIG4 is a structural block diagram of a device for generating a prompt audio according to an exemplary embodiment of the present application. As shown in FIG4 , the device includes the following parts.

A collection module 410 is used to collect voice data of a first subject, where the first subject is located in the first vehicle;

An analysis module 420, configured to analyze the voice data to obtain the emotional state information of the first subject, wherein the emotional state information is used to indicate the emotional state of the first subject when expressing the voice data;

A generating module 430, configured to generate corresponding prompt information based on the emotional state information, wherein the prompt information is used to instruct to provide emotional feedback to the first subject;

The conversion module 440 is used to convert the prompt information into a prompt audio and play the prompt audio.

In an optional embodiment, the analysis module 420 is also used to input the voice data into a pre-trained deep learning model, perform sentiment analysis on the voice data through the deep learning model, obtain and output an analysis result, and the analysis result includes at least one candidate emotional state; collect driving information of the first vehicle, the driving information of the first vehicle includes vehicle operation information of the first vehicle and environmental information of the environment in which the first vehicle is located; based on the analysis result and the driving information, determine the emotional state information of the first subject from the candidate emotional states.

In an optional embodiment, the analysis module 420 is also used to obtain an emotional state comparison table, which contains the correspondence between the driving information of the first vehicle and at least one emotional state; based on the driving information, determine a first correspondence matching the driving information from the emotional state comparison table; based on the emotional state indicated by the first correspondence, determine the emotional state information of the first subject from the candidate emotional states.

In an optional embodiment, before the analysis module 420, as shown in FIG5 , the device further includes:

The training module 450 is used to obtain sample speech data, where the sample speech data is marked with an emotion tag, and the emotion tag is used to indicate the emotional state expressed by the sample speech data; and train the pre-trained model based on the sample speech data to obtain the pre-trained deep learning model.

In an optional embodiment, the training module 450 is also used to model the time series of the sample speech data through the pre-trained model, determine the context information and time dependency of the sample speech data, and the time dependency refers to the characteristics of the sample speech data in the time series; output a predicted emotional state based on the context information and the time dependency of the sample speech data; and adjust the pre-trained model based on the training results and the emotional markers to obtain the pre-trained deep learning model.

In an optional embodiment, the generation module 430 is also used to input the emotional state information into a pre-trained natural language model, identify and analyze the emotional state information through the natural language model, and determine the emotional type when the vehicle-mounted terminal provides feedback to the first subject; generate text prompt information that conforms to the emotional type, and the text prompt information contains emotional feedback content in text form.

In an optional embodiment, the generation module 430 is also used to obtain historical text prompt information generated by the natural language model within a historical time period, and the historical text prompt information is used to indicate the emotional feedback content provided by the vehicle-mounted terminal to the first subject within the historical time period; based on the emotional type, information that meets the emotional type is filtered out from the historical text prompt information as the text prompt information.

In summary, the device for generating prompt audio provided by the present application can collect the driver's voice data, analyze the driver's current emotional state, provide the driver with corresponding emotional feedback, help the driver adjust his emotions in time, and improve the driver's driving experience and safety during driving. Since voice is continuous, as the tone, volume, etc. in the voice data change continuously, it can better reflect the driver's true emotions. Compared with the method of collecting the driver's facial expressions to identify the driver's emotions in the related technology, the driver's emotion recognition based on voice data can improve the accuracy of the recognition results, and thus provide more appropriate emotional feedback.

It should be noted that the device for generating prompt audio provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for generating prompt audio provided in the above embodiment and the method for generating prompt audio belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

FIG6 shows a block diagram of a computer device 600 provided by an exemplary embodiment of the present application. The computer device 600 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a laptop computer or a desktop computer. The computer device 600 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

Typically, the computer device 600 includes a processor 601 and a memory 602 .

The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the method for generating the prompt audio provided in the method embodiment of the present application.

In some embodiments, the computer device 600 further includes some other components 603, and the types and quantities of the other components 603 may be selected based on the functional requirements of the computer device 600. Those skilled in the art will appreciate that the structure shown in FIG. 6 does not limit the computer device 600, and may include more or fewer components than shown in the figure, or combine some components, or adopt different component arrangements.

Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for generating prompt audio as described in any of the above embodiments of the present application.

An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a method for generating a prompt audio as described in any of the above embodiments of the present application.

The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method for generating the prompt audio described in any of the above embodiments.

A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims (10)

1. A method of generating alert audio, performed by an in-vehicle terminal of a first vehicle, the method comprising:

collecting voice data of a first main body, wherein the first main body is positioned in the first vehicle;

Analyzing the voice data to obtain emotion state information of the first main body, wherein the emotion state information is used for indicating the emotion state of the first main body when the voice data is expressed;

Generating corresponding prompt information based on the emotion state information, wherein the prompt information is used for indicating to provide emotion feedback for the first main body;

and converting the prompt information into prompt audio and playing the prompt audio.

2. The method of claim 1, wherein analyzing the voice data to obtain the emotional state information of the first subject comprises:

inputting the voice data into a pre-trained deep learning model, carrying out emotion analysis on the voice data through the deep learning model to obtain and output an analysis result, wherein the analysis result comprises at least one candidate emotion state;

collecting driving information of the first vehicle, wherein the driving information of the first vehicle comprises vehicle running information of the first vehicle and environment information of the environment where the first vehicle is located;

and determining emotion state information of the first subject from the candidate emotion states based on the analysis result and the driving information.

3. The method of claim 2, wherein the determining emotional state information of the first subject from the candidate emotional states based on the analysis results and the driving information comprises:

acquiring an emotion state comparison table, wherein the emotion state comparison table comprises a corresponding relation between driving information of the first vehicle and at least one emotion state;

Determining a first corresponding relation matched with the driving information from the emotion state comparison table based on the driving information;

and determining the emotion state information of the first subject from the candidate emotion states based on the emotion states indicated by the first correspondence.

4. The method of claim 2, wherein before inputting the speech data into the pre-trained deep learning model, further comprising:

Acquiring sample voice data, wherein the sample voice data is marked with emotion marks, and the emotion marks are used for indicating emotion states expressed by the sample voice data;

training the pre-training model based on the sample voice data to obtain the pre-trained deep learning model.

5. The method of claim 4, wherein training the pre-trained model based on the sample speech data results in the pre-trained deep learning model, comprising:

modeling a time sequence of the sample voice data through the pre-training model, and determining context information and time dependency relationship of the sample voice data, wherein the time dependency relationship refers to characteristics of the sample voice data on the time sequence;

Outputting to obtain a predicted emotion state based on the context information and the time dependency relationship of the sample voice data;

And adjusting the pre-training model based on the training result and the emotion marks to obtain the pre-trained deep learning model.

6. The method of claim 1, wherein the generating the corresponding hint information based on the emotional state information comprises:

Inputting the emotion state information into a pre-trained natural language model, and identifying and analyzing the emotion state information through the natural language model to determine the emotion type when the vehicle-mounted terminal feeds back the first main body;

And generating text prompt information conforming to the emotion type, wherein the text prompt information comprises emotion feedback content in a text form.

7. The method according to claim 6, wherein the inputting the emotion state information into a pre-trained natural language model, performing recognition analysis on the emotion state information through the natural language model, and determining the emotion type when the vehicle-mounted terminal feeds back the first subject, further comprises:

Acquiring historical text prompt information generated by the natural language model in a historical time period, wherein the historical text prompt information is used for indicating emotion feedback content provided by the vehicle-mounted terminal to the first main body in the historical time period;

And screening information conforming to the emotion type from the historical text prompt information based on the emotion type to serve as the text prompt information.

8. A device for generating alert audio, the device comprising:

the system comprises an acquisition module, a first vehicle and a second vehicle, wherein the acquisition module is used for acquiring voice data of a first main body, and the first main body is positioned in the first vehicle;

the analysis module is used for analyzing the voice data to obtain emotion state information of the first main body, wherein the emotion state information is used for indicating the emotion state of the first main body when the first main body expresses the voice data;

The generation module is used for generating corresponding prompt information based on the emotion state information, and the prompt information is used for indicating to provide emotion feedback for the first main body;

The conversion module is used for converting the prompt information into prompt audio and playing the prompt audio.

9. A computer device comprising a processor and a memory, wherein the memory stores at least one program, the at least one program being loaded and executed by the processor to implement a method of generating alert audio according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored therein at least one program that is loaded and executed by a processor to implement the method of generating alert audio according to any one of claims 1 to 7.