JP2025071574A - Vehicle Dialogue System - Google Patents

Abstract

To provide a vehicle dialogue system that can give occupants more natural dialogue experiences.SOLUTION: A vehicle dialogue system (1) using generative AI (10) includes: a voice input unit (2) that inputs the voice spoken by an occupant; a voice recognition unit (41) that converts the voice input by the voice input unit into first text data; a feeling estimation unit (45) that inputs at least one of the voice and the first text data to estimate the occupant's feelings and generates second text data representing an estimation result; a control unit (42) that inputs the first text data and the second text data, as input information (S1), to the generative AI (10); a voice synthesis unit (43) that converts response information (S2) from the generative AI (10) into voice; and a voice output unit (5) that outputs the voice converted by the voice synthesis unit (43).SELECTED DRAWING: Figure 1

Description

The present invention relates to a dialogue system for vehicles.

A voice dialogue system for automobiles has been proposed in which, when a user speaks to the system, the system selects an appropriate answer from among pre-registered dialogue scenarios and responds by converting the answer into voice (Patent Document 1).

This type of voice dialogue system selects an appropriate answer from a pre-recorded dialogue scenario even if the user's utterance is somewhat vague. However, this type of voice dialogue system outputs the answer selected from the dialogue scenario in the same tone of voice regardless of the situation, which can make the driver feel uncomfortable with the answer.

JP 2018-54790 A

The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a dialogue system for a vehicle that can provide a more natural dialogue experience for occupants.

In order to achieve the above object, the vehicle dialogue system according to the present invention has the following features.
A vehicle dialogue system using a generation AI that is mounted on a vehicle and that outputs response information made up of text data when input information made up of text data is input,
a voice input unit for inputting a voice uttered by a passenger;
a voice recognition unit that converts the voice input by the voice input unit into first text data;
a feeling estimation unit that estimates a feeling of the occupant by inputting at least one of the voice and the first text data and generates second text data representing an estimation result;
a control unit that inputs the first text data and the second text data as the input information to the generation AI;
A voice synthesis unit that converts the response information from the generation AI into voice;
a voice output unit that outputs the voice converted by the voice synthesis unit,
Dialogue system for vehicles.

The present invention provides a dialogue system for a vehicle that can provide a more natural dialogue experience for the driver or passengers.

The present invention has been briefly described above. The details of the present invention will become clearer by reading the following description of the embodiment of the invention (hereinafter referred to as "embodiment") with reference to the attached drawings.

FIG. 1 is a block diagram showing a vehicle dialogue system according to a first embodiment. FIG. 2 is a flowchart showing a processing procedure of the microcomputer shown in FIG. FIG. 3 is a state transition diagram of an agent. FIG. 4 is a diagram for explaining the states of an agent. FIG. 5 is a block diagram showing a vehicle dialogue system according to the second embodiment.

Specific embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 1 is a block diagram showing an embodiment of a vehicle dialogue system according to the present invention. In the following description, a case where the driver speaks is taken as an example, but the present invention can also be applied to a case where any occupant in the vehicle, such as a passenger, speaks.

The vehicle dialogue system 1 of this embodiment is a system that is mounted on a vehicle and communicates with the driver using a generation AI (Artificial Intelligence) 10. The generation AI 10 is configured, for example, from ChatGPT, and when input information S1 consisting of text data is input, response information S2 consisting of text data is output.

The vehicle dialogue system 1 includes a microphone 2 as a voice input unit, a communication module 3, a microcomputer 4 (hereinafter abbreviated as "microcomputer 4") that functions as an agent, a speaker 5 as a voice output unit, and a display unit 7. The microphone 2 inputs voice spoken by the driver to the microcomputer 4. The communication module 3 is for communicating with the generation AI 10 via an Internet communication network (not shown), and is composed of circuits, antennas, etc. for connecting to the Internet communication network.

The microcomputer 4 has a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory), and a CPU (Central Processing Unit) that operates according to a program stored in the memory, and is responsible for controlling the entire vehicle dialogue system 1.

The microcomputer 4 has a voice recognition unit 41, a voice dialogue unit 42 as a control unit that performs input control for the generation AI 10, control of connected devices, output control for these devices, etc., a voice synthesis unit 43, and an emotion estimation unit 45. The voice recognition unit 41 converts the voice signal input by the microphone 2 into text data (first text data) and inputs it to the voice dialogue unit 42 and the emotion estimation unit 45. The voice dialogue unit 42 adds text data (second text data) corresponding to the driver's emotion determined by the emotion estimation unit 45, text data (third text data) indicating the state of the driver or the vehicle driven by the driver when the driver spoke, or an instruction corresponding to the state, to the text data converted by the voice recognition unit 41, and inputs it to the generation AI 10 as input information S1.

Vehicle information S3 is also input to the voice dialogue unit 42 from multiple sensors mounted on the vehicle. Possible sensors include an illuminance sensor that measures the illuminance outside the vehicle, a temperature sensor that measures the outside air temperature, a GPS (Global Positioning System) that detects the vehicle position, a seating sensor that detects whether or not a person is sitting in a seat, and a speed sensor that detects the vehicle speed. The vehicle information S3 input from the sensor may be not only a measurement value measured by the sensor, but also a moving average or weighted average of the measurement values, or a value calculated using the measurement value, the moving average, or the weighted average of the measurement values.

The voice dialogue unit 42 also receives input of human information S4, which is the detection result from a driver monitor that detects the driver's state (facial expression, whether the driver is dozing, driving absent-mindedly, or not) based on an image of the driver's face. The voice dialogue unit 42 may also receive input of warning information, which is the detection result from an abnormality detection unit that detects abnormalities in the vehicle (engine abnormality, oil pressure abnormality, water temperature abnormality, charging abnormality, etc.) and turns on a warning lamp.

The voice dialogue unit 42 is also connected to vehicle equipment 11 mounted on the vehicle and can control the vehicle equipment 11. Examples of the vehicle equipment 11 include an air conditioner, a display such as a head-up display, and audio equipment mounted on the vehicle.

The voice dialogue unit 42 inputs the response information S2 from the generation AI 10 and outputs the input response information S2 to the voice synthesis unit 43. The voice synthesis unit 43 converts the response information S2 into a voice signal and outputs it to the speaker 5. The speaker 5 outputs the voice converted by the voice synthesis unit 43. The display unit 7 is composed of a means having a screen that can display information to the driver, such as a monitor installed near the driver's seat or a head-up display (HUD). When voice is output by the speaker 5, the microcontroller 4 displays a character linked to the voice on the screen.

The emotion estimation unit 45 estimates the driver's emotion in real time based on the voice signal input from the microphone 2 and the text data input from the voice recognition unit 41, generates text data indicating the estimation result, and inputs it to the voice dialogue unit 42.

The emotion estimation unit 45 extracts features such as the tone, strength, speech rate, and spectral characteristics of the driver's voice from the waveform of the voice signal input from the microphone 2. The emotion estimation unit 45 also extracts expressions and phrases that indicate emotions from the text data input from the voice recognition unit 41 and converts them into features that represent emotions. These features and emotion categories are stored in the emotion estimation unit 45 as a correspondence map. The emotion categories are divided into, for example, "happiness," "anger," "sadness," "surprise," and "neutral."

As an example, the correspondence map is a map in which features are associated with each category using initial values in advance. Therefore, when the emotion estimation unit 45 extracts features, it refers to the correspondence map to determine the driver's current emotion and inputs it to the voice dialogue unit 42 as text data.

The emotion estimation unit 45 may also compare the features extracted from the voice signal with the text data input from the voice recognition unit 41, and correct the features associated with each emotion category in the correspondence map.

The emotion estimation unit 45 outputs the driver's emotion determined by the correspondence map to the voice dialogue unit 42 as text data. The voice dialogue unit 42 adds text data corresponding to the emotion estimated by the emotion estimation unit 45 to the input information S1 input to the generation AI 10. For example, if the emotion estimation unit 45 determines that the driver's emotion is "anger," the voice dialogue unit 42 adds the instruction "use words that will calm the anger." This allows the generation AI 10 to generate an answer that calms the anger or uses gentle language.

The emotion estimation unit 45 may be configured to continuously retain the determined emotion of the driver. In this case, it is preferable that the duration for which the emotion is retained can be set to be indefinite or for a limited period such as a predetermined time. In addition, when emotions are retained in this manner, it is preferable that the microcomputer 4 is configured to reset the emotion retained in the emotion estimation unit 45 by input from the occupant or when the vehicle situation satisfies a predetermined condition.

Next, the operation of the vehicle dialogue system 1 configured as described above will be explained with reference to the flowchart shown in FIG. 2. When the microcomputer 4 detects that the ignition is on (IG-ON) (Sp1), it transmits text data indicating the state of the driver and the vehicle to the generation AI 10 (Sp2). An example of this text data indicating the state of the driver and the vehicle could be "The person you are speaking to is about to get in the car. Please keep the conversation in a way that takes into account the fact that the person is in a car." This allows the generation AI 10 to understand that the person you are speaking to (the driver) is in the car, and from then on the dialogue content will be based on the assumption that the person is in the car.

In addition, in Sp2, the microcontroller 4 may transmit text data of information on the vehicle equipment 11 that can be controlled. An example of this text data could be, "We can suggest adjusting the temperature and airflow of the air conditioner, adjusting the brightness of the head-up display, and controlling the audio, depending on the conversation with the person you are talking to." This allows the generation AI 10 to understand that the driver can adjust the temperature and airflow of the air conditioner, adjust the brightness of the head-up display, and control the audio in the vehicle he is riding in.

In addition, in Sp2, the microcontroller 4 may determine the number of people riding in the vehicle based on the detection results from the seating sensor, and transmit the determined number of people to the generation AI 10. An example of this text data could be "Only the driver is riding in the vehicle" or "There are two people riding in the vehicle, including the driver." This allows the generation AI 10 to understand whether there is only one driver or whether there are other people riding in the vehicle.

Next, the microcontroller 4 acquires the vehicle information S3 and person information S4 described above (Sp3). After that, when the driver speaks (Y in Sp4), the microcontroller 4 performs a voice recognition process to convert the voice spoken by the driver into text data, and estimates the driver's emotions based on the voice (Sp5). Next, the microcontroller 4 adds the driver's emotions estimated in Sp3, the state of the driver and vehicle at the time of speaking, or commands (text data) according to the state, to the text data converted by the voice recognition process, and transmits it to the generation AI 10 as input information S1 (Sp6).

In Sp6, the microcomputer 4 generates text data for prompt adjustment to obtain an answer that will make the driver feel positive, according to the driver's emotions estimated in Sp5. The microcomputer 4 may then add the generated text data to the input information S1 together with text data indicating the vehicle information S3 and person information S4 acquired in Sp3. The microcomputer 4 may also add text data indicating whether the vehicle is driving, stopped, or waiting at a traffic light as text data indicating the vehicle's status.

The microcomputer 4 also determines whether the driver's driving load is high when the driver speaks. Whether the driving load is high can be determined, for example, based on the speed from a speed sensor (vehicle information S3) or person information S4. It may also be determined based on pre-registered driver attributes (whether the driver is accustomed to driving or not, whether the driver is familiar with the vehicle's equipment or not). If the microcomputer 4 determines that the driving load is high when the driver speaks, it may assign text data to shorten the response information S2 as a command according to the state. An example of such text data may be, "Please summarize your answer."

Then, the microcomputer 4 receives response information S2 corresponding to the input information S1 sent in Sp7 (Sp7), converts the received response information S2 into voice, and outputs it from the speaker 5 (Sp8). Next, when the microcomputer 4 detects that the ignition is off (IG-OFF) (Y in Sp9), it ends the process. On the other hand, if the microcomputer 4 does not detect that the ignition is off (N in Sp9), it returns to Sp3 again.

Furthermore, after executing Sp3, if the driver has not spoken (N in Sp4), or if a predetermined proposal condition based on the information S3-S4 acquired by Sp3 is met (Sp10), the microcomputer 4 converts the proposal content into voice data and outputs it from the speaker 5 (Sp11), and then returns to Sp3. As a proposal condition for Sp11, for example, if the speed measured by the speed sensor (vehicle information S3) is fast, the proposal condition to reduce the speed is met, and a proposal to reduce the speed is output from the speaker 5.

When the above-mentioned control is executed, the microcomputer 4 may display a character on the vehicle's display, and may display the character as if it is speaking to the driver in sync with the audio output from the speaker 5. In this case, the microcomputer 4 may automatically change the appearance of the character, such as the facial expression, according to the driver's emotions, or the driver may change the character through operation.

When the microcontroller 4 automatically changes the character's facial expression, for example, if the driver's emotion is negative, such as "anger" or "sadness," as described above, the voice dialogue unit 42 adds instructions to the generation AI 10 to give an answer that will make the driver's emotion positive, and in accordance with this, the character's facial expression and movements are also displayed to make the driver's emotion positive, such as by expressing kindness or by moving slowly. Conversely, if the driver's emotion is positive, the character's facial expression also expresses happiness, thereby displaying empathy with the driver's positive emotion.

As another example, if the driver's emotions are negative, the microcomputer 4 may change the character to one that is most likely to induce positive emotions. To determine which character change is most effective for making the driver's emotions positive, for example, the voice dialogue unit 42 may accumulate data on the changes in the driver's emotions after changing the character. In other words, as the driver and the agent have repeated conversations, the character and its responses are optimized to suit the driver's preferences.

Fig. 3 is a state transition diagram of an agent, and Fig. 4 is a diagram for explaining the states of an agent.
When a character linked to the function of an agent is displayed on the screen of the display unit 7, the microcomputer 4 is first in a sleep state. In the sleep state, the character is made small so as not to obstruct the display of the basic screen or the display of operations.

When the wake-up word spoken by the driver is input to the microphone 2, the microcomputer 4 transitions to a wake-up state and starts functioning as an agent. In this state, the microcomputer 4 waits for the driver to speak. If the driver does not speak after waiting a specified time, the microcomputer transitions to a sleep state. If the driver speaks within the specified time, the microcomputer 4 transitions to a hearing state and performs voice recognition and analysis.

The microcontroller 4 generates input information S1 from text data generated from the driver's speech, the driver's emotions, and text data generated based on the driver and vehicle conditions, and is in a wait state while transmitting the input information S1 to the generation AI 10. Then, when response information S2 in the form of text data is received from the generation AI 10, the microcontroller 4 transitions to a speech state, outputs a voice representing the response information from the speaker 5, and displays a character that matches the voice.

In the speech state, if the driver starts speaking while the response information S2 is being output by voice, the microcomputer 4 executes the barge-in function to transition to the hearing state. Depending on the situation, the output of the response information that was interrupted by the execution of the barge-in function is resumed after the driver speaks, or is terminated as it is. For example, if the response information S2 is important information related to driving, the output of the remaining response information that was interrupted after the driver speaks may be resumed, and if the importance is low, it may be terminated as it is.

When the microcontroller 4 is in the speech state and the audio output of the response information S2 ends, it transitions to the wake-up mode and waits for the speaker to speak again.

When the screen of the display unit 7 doubles as a user operation screen, such as a touch panel, the driver may change the display position of the character by dragging or the like, or may enlarge or reduce the display size of the character using multiple fingers. Also, when a setting screen for the agent function, including changing the character, is displayed, the microcomputer 4 suspends the display of the character on the screen.

FIG. 5 is a block diagram showing a vehicle dialogue system according to the second embodiment.
In this embodiment, the same components as those in the first embodiment are given the same reference numerals and the description thereof will be omitted.

In the second embodiment, the emotion estimation unit 45 inputs the vehicle information S3 and the person information S4 and extracts features representing emotions. For example, when information indicating the accelerator opening, the brake pedal force, or the vehicle acceleration is greater than usual and the driver's facial expression corresponds to "anger," the emotion estimation unit 45 extracts this information as features that are strongly related to the driver's "anger." The emotion estimation unit 45 adds up the features based on the above-mentioned voice signal, the features obtained from the vehicle information S3, and the detection result of the driver's facial expression obtained from the person information S4 in a predetermined ratio, and determines the final emotion of the driver based on the correspondence map. Note that the emotion estimation unit 45 may input only one of the vehicle information S3 and the person information S4.

The present invention is not limited to the above-described embodiment, and can be modified, improved, etc. as appropriate. In addition, the material, shape, size, number, location, etc. of each component in the above-described embodiment are arbitrary as long as they can achieve the present invention, and are not limited.

For example, the emotion estimation unit 45 may estimate the driver's degree of fatigue in addition to classifying emotions, and input text data indicating the driver's degree of fatigue to the voice dialogue unit 42 in addition to or instead of determining the emotion. As an example, when the driver is fatigued, the voice dialogue unit 42 adds an instruction to "speak in gentle, conversational words" to the input information S1 input to the generation AI 10.

The microcomputer 4 may also change the answer or character depending on the mood of the driver and the passenger. For example, when the voices of both the driver and the passenger are input to the microphone 2, text data indicating the respective voice signals is input from the voice recognition unit 41 to the emotion estimation unit 45. When the emotion estimation unit 45 determines that the emotions of both the driver and the passenger are negative, the voice dialogue unit 42 adds an instruction such as "use words that mediate a fight" to the input information S1. When the emotions of the driver and the passenger obtained from the emotion estimation unit 45 are both positive and the driver and the passenger speak a lot, the voice dialogue unit 42 adds an instruction such as "use a cheerful atmosphere and keep words as short as possible" to the input information S1 so as not to disturb the conversation between the driver and the passenger. Conversely, when the emotions of both the driver and the passenger are positive but the driver and the passenger speak a little, the voice dialogue unit 42 adds an instruction such as "use a cheerful atmosphere and keep words as short as possible" to the input information S1.

The microcomputer 4 may also change the character or tone of voice depending on the driving scene of the vehicle. For example, even when the same character is displayed, the microcomputer 4 may change the costume or the character itself depending on the driving scene, such as summer and winter, daytime and nighttime, sunny and rainy, etc.

In addition, in the above embodiment, the emotion estimation unit 45 estimates emotions based on text data from the voice recognition unit 41, or based on the text data and the vehicle information S3. However, depending on the situation, emotions may be estimated based only on the vehicle information S3.

The emotion estimation unit 45 may also input human information S4, extract features indicating the driver's emotions from the human information S4, and use the extracted features as one of the parameters when determining the emotion. The emotion estimation unit 45 may also accumulate information on what emotions the driver is feeling and what character or tone of response would make the driver feel positive, based on features contained in the voice of the driver's utterances, such as further questions in response to the answer from the generation AI 10, and the amount of speech, and change the character settings according to the situation.

Here, the features of the vehicle dialogue system according to the embodiment of the present invention described above are briefly summarized and listed below in [1] to [4].

[1] A vehicle dialogue system (1) that uses a generation AI (10) that is mounted on a vehicle and that outputs response information (S2) made of text data when input information (S1) made of text data is input,
A voice input unit (2) for inputting a voice uttered by a passenger;
a voice recognition unit (41) for converting the voice input by the voice input unit into first text data;
an emotion estimation unit (45) that receives at least one of the voice and the first text data to estimate an emotion of the occupant and generate second text data representing an estimation result;
A control unit (42) that inputs the first text data and the second text data as the input information (S1) to the generation AI (10);
a voice synthesis unit (43) that converts the response information (S2) from the generation AI (10) into voice;
a voice output unit (5) that outputs the voice converted by the voice synthesis unit (43),
Dialogue system for vehicles.

According to the configuration [1] above, when obtaining output from the generation AI, it is possible to obtain an answer that takes into account the emotions of the occupant. Therefore, when presenting an answer to the occupant by voice, even if the answer has the same content, the tone and wording can be changed depending on the emotions of the occupant, so that the occupant can feel a sense of familiarity with the dialogue system. In addition, if the occupant's emotions are negative, an answer that takes these into account can be output, thereby preventing the occupant's emotions from worsening further.

[2] The control unit (42) generates additional information corresponding to the emotion indicated by the second text data, adds the additional information to the input information (S1), and inputs the additional information to the generation AI (10).
The vehicle dialogue system according to [1] above.

According to the configuration of [2] above, the vehicle dialogue system can specify to the generation AI the tone and wording of the response more specifically according to the occupant's emotions, so that an appropriate response can be provided that matches the occupant's emotions.

[3] The emotion estimation unit (45) selects one of emotions indicated by a plurality of categories classified in advance as an estimation result, inputs the selected result to the control unit (42) as second text data, and continuously holds or resets the selected result.
The vehicle dialogue system according to any one of claims 1 to 2.

According to the configuration of [3] above, the vehicle dialogue system categorizes the occupant's emotions in advance, which reduces the possibility of erroneously recognizing the occupant's emotions. In addition, the vehicle dialogue system can correct the features of the occupant's voice associated with each category, so that the occupant's emotions can be judged more accurately as the occupant speaks more.

[4] A display unit (7) having a screen for presenting image information to an occupant,
The control unit (42) changes the appearance or type of the character displayed on the screen.
The vehicle dialogue system according to any one of [1] to [3] above.

According to the configuration of [4] above, the vehicle dialogue system changes not only the voice output but also the character displayed according to the occupant's emotions, allowing the occupant to feel a sense of familiarity with the vehicle dialogue system both visually and aurally.

Reference Signs List 1 Vehicle dialogue system 2 Microphone 3 Communication module 4 Microcomputer 5 Speaker 7 Display unit 10 Generation AI
11 Vehicle equipment 41 Voice recognition unit 42 Voice dialogue unit (control unit)
43 Voice synthesis unit 45 Emotion estimation unit S1 Input information S2 Response information S3 Vehicle information S4 Person information

Claims (4)

A vehicle dialogue system using a generation AI that is mounted on a vehicle and that outputs response information made up of text data when input information made up of text data is input,
a voice input unit for inputting a voice uttered by a passenger;
a voice recognition unit that converts the voice input by the voice input unit into first text data;
a feeling estimation unit that estimates a feeling of the occupant by inputting at least one of the voice and the first text data and generates second text data representing an estimation result;
a control unit that inputs the first text data and the second text data as the input information to the generation AI;
A voice synthesis unit that converts the response information from the generation AI into voice;
a voice output unit that outputs the voice converted by the voice synthesis unit,
Dialogue system for vehicles. The control unit generates additional information corresponding to the emotion indicated by the second text data, adds the additional information to the input information, and inputs the additional information to the generation AI.
The vehicle dialogue system according to claim 1 . The emotion deduction unit selects one of emotions indicated by a plurality of categories classified in advance as an estimation result, inputs the selected result to the control unit as second text data, and continuously holds or resets the selected result.
The vehicle dialogue system according to claim 1 . A display unit having a screen for presenting image information to an occupant,
The control unit changes the appearance or type of the character displayed on the screen.
The vehicle dialogue system according to claim 1 .

Images