JPH09269889A - Dialogue device - Google Patents

Abstract

(57) [Abstract] [Problem] In a dialogue device between a user and a computer,
Dialogue is provided by integrating the input of user's voices and facial expressions from multiple input channels to determine the location of the right to speak, and controlling information such as messages output from the computer according to the location. To provide such a device that is capable of smoothly carrying. SOLUTION: A user's operation state is recognized by a recognition means 2 from a voice signal, a head movement, a gaze direction, facial expression input data 1 and time information by a time giving means 6, and the recognition result is a speech right control means. The location of the right to speak is determined by the computer or the user. The dialogue management means 4 smoothly carries the dialogue by generating a response from the computer side according to the location of the speaking right and transmitting it to the user via the output means 5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a dialogue device that enables a dialogue between a human and a computer, and more specifically, a device that realizes a smooth dialogue not only by voice but also by expressions other than voice. Regarding

[0002]

2. Description of the Related Art Conventionally, as an interface between a human being and a computer, the computer has mainly taken the initiative in asking questions, and the user has been passively answering the questions while proceeding with the work. Therefore, the order of the user's utterances is fixed, and no consideration is given to the easiness of the user's utterance and the naturalness of the dialogue. Therefore, Japanese Patent Laid-Open No. 6-1
Japanese Patent No. 10835 discloses an apparatus in which a user can speak by interrupting audio output from a computer. Further, Japanese Patent Application Laid-Open No. 62-40577 discloses a device capable of listening back in the middle of a computer utterance.

[0003]

When interacting with a computer, the naturalness of the interaction is lost unless the computer and the user speak in a timely manner. As a countermeasure, Japanese Patent Laid-Open No. 6-110835 discloses a method in which a user's utterance is detected and a computer utterance can be interrupted. Further, in Japanese Laid-Open Patent Publication No. 62-40577, "Eh, why? It is designed so that utterances such as "Suka" can be detected and reflected back. However, the atmosphere in which humans start or end talking does not appear only in the voice, but at the same time in the voice, facial expressions and gestures,
Alternatively, since they appear in a complementary manner, the countermeasures shown as the conventional example are not always satisfactory.
One of the reasons is that the human beings start or start the response from the computer after finishing the conversation, so that the timing of the response is delayed, and a smooth dialogue cannot be realized between the computer and the user.

The present invention has been made in view of the above-mentioned problems in the prior art. In a dialogue device between a human and a computer, a user's utterance begins or ends based on a plurality of signal features generated in response to a user's action. More quickly, determine whether the utterance initiative (speaking right) lies between the computer and the user, and determine the location of the utterance right. Further, according to the location of the speaking right, the audio information output from the computer side Or by controlling image information,
The problem to be solved is to provide a dialogue device that realizes a natural dialogue.

[0005]

According to a first aspect of the present invention, an input means for inputting a voice generated by a human or the like and a facial expression of the human or the like that can be visualized, and an audible voice and a visible signal to the human or the like are output. Dialogue having output means for performing the recognition, recognizing means for recognizing the operating state of the user based on the signal obtained by the input means, and processing / control means for processing the output of the recognizing means and controlling the output means. In the device, the recognizing means outputs the operation state of the user in a plurality of predetermined operation modes as a recognition result, and the processing / control means determines the dialogue situation from the recognition result and determines the whereabouts of the right to speak. Is performed, and the output means is controlled to perform an output corresponding to the result of the processing, the recognition results are integrated, the dialogue situation is judged, and the utterance end / start is started earlier. That is, whether the speaking right is on the computer or the user, and depending on the location of the speaking right, control the output of audio or visual signals to the user to smooth the dialogue. It is possible to carry to.

According to a second aspect of the present invention, in the first aspect of the present invention, output generating means for generating an output corresponding to the result of the processing in the processing / control means is provided, and the situation of the dialogue and the whereabouts of the right to speak are provided. It is also possible to output the audio and visual signals generated by the output generating means in response to the output, and generate a message prompting a dialogue in accordance with the determined whereabouts of the speaking right and outputting the message from the computer side to the user. And

According to a third aspect of the present invention, in the first or second aspect of the present invention, the start and end times of the user's operating state recognized by the recognizing means are added, and the time management of the operation in the processing / controlling means is performed. It is possible to carry the dialogue more smoothly by providing the time giving means used for the above and operating it based on the time information (temporal correlation).

According to a fourth aspect of the invention, in any one of the first to third aspects of the invention, the plurality of predetermined operation modes are the volume of voice, the height of voice, the direction of the line of sight, and the vertical swing of the head. To recognize the operating status of each mode,
The location of the speaking right is determined based on the volume of the user's voice (voice power), the height of the voice (voice pitch), the direction of the line of sight, and the vertical swing of the head (nodding), and the error is less likely to occur. It is possible to operate.

According to a fifth aspect of the present invention, in any one of the first to fourth aspects, the plurality of predetermined modes are the volume of voice, the pitch of voice, and the direction of the line of sight, and the operating state of each of the modes is set. Recognize and determine the user's emotion from the recognition result to redetermine the location of the user's speaking right to either the user or the device, and determine the emotion to make the user difficult to reply. As a result, the computer avoids a response that helps the user speak, or avoids the user's confusion, and enables a more satisfying conversation.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a first interactive device of the present invention.
It is a block diagram showing an embodiment. The dialogue apparatus of the present embodiment includes a plurality of channels of recognition means 2 for recognizing input data 1 including voice signals, head movements, gaze direction, and facial expression time information. The time giving means 6 for outputting information and the speech right control means 3 for integrating the recognition results output in parallel from the respective recognition means 2-1, 2-2 ... There is. The speech right control means 3 is connected to a history storage means 7 for holding a history of the location of the speech right, and a dialogue management means 4 for advancing a dialogue based on the location of the speech right recognized by the speech right control means 3. Therefore, the dialogue management means 4 is connected to the output means 5 for outputting output data. In addition, each recognition means 2-
.. have a recognition algorithm corresponding to the recognition data, and are configured to obtain the start time and end time of the recognition result from the time giving means 6.

Next, the operation of this embodiment will be described. As input data, it is possible to capture multiple signals generated in response to a user's action or motion. For example, the volume of voice obtained from a camera, microphone, motion sensor, or electrocardiograph. (Hereinafter referred to as "voice power"), voice pitch (hereinafter referred to as "voice pitch"), speech speed, pause (between) length, facial expression, face orientation, mouth size and shape, eyes The size, blinking, gaze direction, gesture, hand gesture, head movement, and heart rate are recognized. As the output means, for example, a speaker, a display (visible display means), or a tactile device can be used.
In the following, as input data, voice power, voice pitch,
The embodiment of the present invention will be described as a computer having a line-of-sight direction and a head swing, a voice output means for outputting a synthetic voice as an output means, and an output means for displaying a pseudo human being by CG. .

The voice power of the input data 1-1 is A / D converted, and the voice power level is recognized for each predetermined processing unit (frame: 1 frame is 1/30 seconds). It is detected and sent to the speech right control means 3. The voice pitch of the input data 1-2 is A / D converted, and the voice pitch level is recognized for each predetermined processing unit (frame: 1 frame is 1/30 seconds).
A predetermined level of "rising pitch" is detected and sent to the speech right control means 3. The direction of the line of sight of the input data 1-3 is determined by eye contact (the user is looking at the CG pseudo human on the display) for each predetermined processing unit (frame: 1/30 second for one frame). The recognized “eye contact” having a predetermined frame length is detected and sent to the speech right control unit 3. The head movement of the input data 1-4 is determined by a predetermined processing unit (frame: 1 frame is 1/30).
The vertical swing of the head is recognized every second), the "vertical swing of the head" having a predetermined frame length is detected, and the vertical swing of the head is transmitted to the speech right control unit 3.

The information transmitted from each of the recognition means 2-1, 2-2 ... Is information represented by a start time, an end time, a mode and a recognition result. The start time and the end time are values passed from the time giving means 6, and represent the start time and the end time of the input data for which the recognition result is obtained. In this embodiment, the mode represents a plurality of types of outputs simultaneously generated by the user, such as voice power, voice pitch, line-of-sight direction, and head movement. According to the recognition result, "power of voice", "pitch" of voice pitch, "eye contact" of gaze direction, and "vertical head swing" of head movement are obtained according to these modes. . It is also possible to add a facial expression that expresses feelings of laughter or confusion or a gesture as a mode.

In the utterance right control means 3, whether the user is going to speak (beginning of utterance) or is about to end utterance based on the input voice power, voice pitch, line of sight direction, and head movement of the user ( The end of the utterance is determined, and the location of the utterance right is determined based on the information passed from the history storage means 7, which of the computer and the user has the initiative (utterance right) of the current utterance together with the time information. When the right to speak is on the computer side, an “interruption” from the user is expected, and in order to smoothly proceed with the dialogue, the user's utterance should tell whether the user is trying to speak or has done so. In addition, it is desirable that the computer interrupts the utterance and prompts the user's utterance if the user is trying to utter, judging from the action appearing immediately before the utterance.

FIG. 2 is a diagram showing an output state of the recognizing means for explaining an example of processing in the utterance right control means 3 when the utterance right is located on the computer side. When the right to speak is on the computer side, actions that occur from the user include "interruption" as well as "aizuchi" for computer utterances. In the present embodiment, the judgment of "interruption" or "aizuchi" is made from time T1 when the speaking right is on the computer side, T1 + WT (for example, 0.
5 seconds) according to the Bayesian identification of "presence or absence of voice", "rising pitch" and "eye contact" of the user.
When it is determined by the Bayesian identification that the user is trying to speak (interrupt), that effect is indicated by the dialogue management means 4.
The dialogue management means 4 interrupts the utterance of the computer and makes the utterance "What's wrong?", Or the output means such as making the CG pseudo-human express the expression "what's wrong?" Do through 5. on the other hand,
If it is determined to be an affair, the dialogue management means 4 to that effect.
The dialogue management means 4 uses the output means 5 to continue the utterance as it is. It should be noted that here, with a certain length of utterance, an example of data in which a rising pitch can be seen is an utterance called so-called additional word or unnecessary word such as "eh", "yes?", "Ha". Point to. If these reactions are not seen during the system utterance, the message "interruption from the user / no gap" is transmitted to the dialogue management means 4, and the dialogue management means 4 outputs the output means 5 so as to continue the utterance. Through.

The Bayes identification method will be described below. The Bayesian identification uses information from a multimodal dialogue database, which records the state of dialogue between human beings and the dialogue between a computer and a user in various modes. FIG. 3 is a diagram showing an example of data in the multimodal dialogue database. In order to perform Bayesian identification for identifying azure / interruption, the user's azure and interruption are extracted from the data of the user who is speaking the computer from the multimodal dialogue database, and the "voice" at that time, The existence of "rising pitch" and "eye contact" is checked,
The result of the survey is used as one learning data.

For example, as an example of interrupt data from a user, a "voice" of a certain length is seen at time T1 during computer utterance, and the voice is "rising pitch".
Then, at the same time, during the time T1 + WT, "eye contact"
In the case of FIG. 2 (A) in which is observed, learning data of W 1 1 1 is obtained. The leading W means an interrupt, and the following 1 and 0 are 1 if "voice", "rising pitch" and "eye contact" are present, and 0 if they are not present. Further, as an example of the data of the azuchi, a certain length of "speech" was seen at time T1 during computer utterance, but it was not "rising pitch" and "eye contact" between time T1 + WT. In the case of FIG. 2 (B) where no is seen, learning data of A 1 0 0 is obtained. A at the beginning means Aizuchi. A lot of such learning data is prepared, and as the recognition data, for example, "1 1 1" (a "speech" of a certain fixed length is seen, and the sound is an "increasing pitch", and at the same time, "eye contact""Wasseen") was given, "W 1 1 1" and "A
If the number of “1 1 1” is compared and the number of “W 1 11” is larger, then it is determined that the user is trying to interrupt, that is, speak, and “A 1 1”
If there are more "1" s, then it is determined that the user is making a tie. If the same number (including the case where both are 0) is “unknown”, the fact is notified to the dialogue management means 4, and the dialogue management means 4 performs the utterance as it is through the output means 5.

Next, an example of processing in the speech right control unit 3 when the speech right is on the user side will be described with reference to FIG. If the user has the right to speak, whether the user is about to finish speaking or not is determined not only by the user but also by the action that appears immediately before the end of the speech so that the response from the computer can be made in a timely manner. It is desirable to do. When the user has the right to speak, there is a pause (interval) in the middle of the utterance that is easily mistaken for the end of the user's utterance. In the present embodiment, in order to determine the end of the user's utterance, the user's “vertical swing” and “rise” that occur between the time T2 and T2 + TW (for example, 0.5 seconds) when the user has the right to speak. Pitch "," eye contact "
Bayes identification of "presence or absence of voice". When it is determined to be a pose based on the Bayesian identification, the dialogue management means 4 performs it through the output means 5 with a smiling CG face image while inserting the voice of the voice. If it is determined that the user is about to finish the utterance, "the user is about to finish the utterance" is transmitted to the dialogue management means 4, and the dialogue management means 4 is notified.
Performs the utterance of the computer through the output unit 5 so that the face image is lips (moving the mouth when uttering) together with the voice output. Further, the volume of voice and the speed of speech may be designated, and facial expressions and gestures may be added.

An example of Bayes identification for identifying a pause in the middle of user utterance and the end of utterance will be described. In this case, the pose data of the user and the data of the end of the utterance are extracted from the multimodal dialogue database, and the vertical swing of the head, the ascending pitch, the eye contact, the The presence or absence of "voice" is checked, and the result of the check is used as one learning data. For example, as an example of the “end” data of the user's utterance, “vertical head swing” occurs at the time T2 during the user's utterance, and from time T2 to T.
"Eye contact" can be seen between 2 + WT and "voice"
In the case where is not observed (FIG. 3 (A)), the learning data of F 1 0 1 0 is obtained. The leading F means the end of the utterance, and the next 1 and 0 are 1 if the respective states exist, and 0 if they do not exist.

As another data example of the "end" of the user's utterance, utterances of "vertical head swing" and "rising pitch" are seen at time T2 during the user's utterance, and from time T2 to T2 +.
If “eye contact” is seen during WT and no “voice” is seen (FIG. 3 (B)), learning data of F 1 1 1 0 is obtained. On the other hand, as a data example of the “pause” of the user's utterance, “vertical head swing” occurs at time T2 during the user's utterance, and from time T2 to T2 + W.
When neither "eye contact" nor "voice" is seen during T (FIG. 3C), learning data of P 1 0 0 0 is obtained. The first P means a pause of utterance, and the next 1 and 0 are 1 if the respective states exist, and 0 if they do not exist.

A large amount of such learning data is prepared, and the recognition data is, for example, "1 0 1 0".
(A "vertical head swing" occurs at time T2 when the user is speaking,
Moreover, when the "eye contact" was seen and the "voice" was not seen) from the time T2 to T2 + WT), "F 1 0 1 0" and "P1" in the learning data were given.
The numbers of "0 1 0" are compared. As a result, "F
If there are more "1 0 1 0", then it is determined that the user is about to finish speaking, and "P 1 0
If there are more "10", then it is determined to be a pose. If the number is the same (including the case where both are 0), it is "unknown", so that is notified to the dialogue management means 4, and the dialogue management means 4 performs the output means 5 so as to wait for the user's utterance.

"Rise pitch" and "vertical swing of the head"
Occurs immediately before the end of the utterance, and thus Bayes identification for predicting the end of the utterance may be first performed from the "rise pitch" and the "vertical head swing" generated by the user. In this case, when the end of the utterance is predicted by Bayes identification, the dialog management means 4 is informed that “the user's utterance may end”. However, since the user's utterance may not be the end but may be a simple pose, the dialogue management means 4 performs it with the smiley CG face image with the smiling CG face image through the output means 5 and from the computer side. Prepare to speak. Then, "presence or absence of voice" and "eye contact" generated by the user for a certain period of time are examined to perform Bayesian identification of the end / utterance of the utterance. If it is determined by Bayes identification that the user's utterance is the end, "the user is about to end the utterance" is transmitted to the dialogue management unit 4, and the dialogue management unit 4 outputs the utterance of the computer along with the voice output and the face image. The output means 5 is used so that the user can speak. Human beings express their intentions in facial expressions, gestures, etc. before starting or ending talking. Therefore, by recognizing the human motion that appears before or after such utterance, the location of the utterance right can be determined more quickly, and the dialogue between the human and the computer can be advanced in a timely and smooth manner. become able to.

Next, a second embodiment of the dialogue apparatus of the present invention will be described with reference to the accompanying drawings. When the user has the right to speak, in addition to the case where the user is speaking, it may be expected that the speech on the computer side ends and the right to speak is transferred to the user. In the present embodiment, as described above, it is determined whether or not the user is in trouble when there is no user utterance even though the computer has finished speaking and the user has been given the speaking right. In such a situation, when there is no user's utterance, there are two cases: when the user cannot hear the utterance on the computer side, "I am in trouble" because I do not understand the content, and when I am "thinking" about the content of the response. Conceivable.

The configuration of this embodiment is as shown in FIG.
The emotion determination means 8 for determining whether or not the user is having difficulty answering is added to the configuration of the first embodiment. The operation of determining the emotion of the user in this embodiment will be described below. In the right to speak control unit 3, first, the location of the right to speak is determined based on the information passed from the history storage unit 7 together with the time information. Here, the information from each recognition means 2-1, 2-2, ... During the time information T3 to T3 + WT (for example, 2 seconds) when the speech right is transferred from the computer to the user is sent to the emotion determination means 8.

FIG. 6 is a diagram showing an output state of the recognition means for explaining an example of processing in the emotion determination means 8.
In the emotion determination means 8, the user is in trouble, or
It is "eye contact" that determines whether you are thinking,
Based on Bayesian identification of "presence or absence of voice" and "rising pitch". If it is determined that the user is “in trouble” by the Bayesian identification, the message “the user is in trouble” is transmitted to the dialogue management means 4. Then, the dialogue management means 4 causes the computer to output, for example, a utterance "What's wrong" or a CG face image with the expression "what's wrong" through the output means 5. On the other hand, when it is determined that the user is “thinking” by the Bayes identification, the fact that “the user is thinking” is transmitted to the dialogue management means 4. Then
The dialogue management unit 4 causes the computer to output a CG face image with a bowed head, for example, through the output unit 5 in order to wait for a response from the user.

Next, an example of Bayes identification for identifying whether a person is in trouble or thinking. In this case, from the multimodal dialogue database, the time information T3 to T3 + WT at which the computer uttered is finished.
For the data when the user is in trouble or thinking during, the presence or absence of "eye contact", "voice" and "rising pitch" is checked, and the result of the investigation is used as one learning data. . For example, as an example of data in which the user is "in trouble", there is a certain length of "voice" between "T3" and "T3 + W", and the voice is "up pitch" ( Figure 6
In (A)), the learning data of K 1 1 1 is obtained. The leading K means troubled, and the next 1 and 0 are 1 if the respective states exist, and 0 if they do not exist. As another example of data in which the user is “in trouble,” “eye contact” was seen, but “voice” was not seen (FIG. 6 (B)), it was K 1 0 0. Learning data is obtained. On the other hand, as an example of the data “thinking” by the user, when neither “eye contact” nor “voice” is seen (FIG. 6C), learning data C 0 0 0 is obtained. The leading C means thinking.

A large amount of such learning data is prepared, and as recognition data, for example, "1 1 1"("eyecontact", there is "voice" of a certain length, and that voice is "rising pitch" Was given), the numbers of “K 1 11” and “C 1 1 1” in the learning data were compared, and if the number of “K 1 1 1” was larger,
At that time, the user is judged to be "in trouble" and "C
If there are more "1 1 1", then it is determined that the user is thinking. If the same number (including the case where both are 0) is “unknown”, the fact is transmitted to the dialogue management means 4, and the dialogue management means 4 utters, for example, “what happened” or A CG face image with the expression "What happened" is output through the output means 5.

In this way, when the user has no utterance even after the computer utterance has been finished and the utterance right has been handed to the user, it is judged whether or not the user has a problem in replying. For example, the speaking right can be returned to the computer, and if the user thinks, it can be controlled to wait for the user's response while giving the speaking right to the user. Further, by understanding the user's emotions in this way, it is possible to avoid a situation in which the user is having difficulty answering, and realize a smooth dialogue with the computer without disturbing the user's thinking. Although the computer responds "what's wrong" here, of course, computer utterance may be repeated or the utterance content may be changed to speak again to the user.

[0029]

【The invention's effect】

Effect of claim 1: It is possible to determine whether the computer or the user has the speaking right based on the characteristics of a plurality of signals such as audio and video generated in response to the user's action (operating state), and the speaking right. It is possible to control the output of audio and visual signals to the user according to the location of the user. Therefore, it is possible to provide a dialogue device capable of smoothly carrying the dialogue.

Effect of claim 2: In addition to the effect of claim 1, a message for prompting a dialogue is generated from the computer side according to the location of the right to speak and is output to the user, so that the dialogue is smoother. It is possible to proceed.

Effect of claim 3: In addition to the effect of claim 1 or 2, by further comprising time adding means for outputting time information, time of a plurality of signal features generated corresponding to user's action It is possible to capture the real correlation. As a result, it is possible to determine the location of the speaking right with less error.

Effect of claim 4 In addition to the effects of claims 1 to 3, the volume of the user's voice (voice power), the pitch of the voice (voice pitch), the direction of the line of sight, and the vertical swing of the head (nod). The location of the right to speak is determined from this). As a result, it is possible to determine the location of the speaking right with less error.

Effect of claim 5: In addition to the effects of claims 1 to 4, the user's voice volume (voice power), voice pitch (voice pitch), and line-of-sight direction are recognized, and the result is obtained. By determining the user's emotion from the computer, the computer can make a response that helps the user to speak, avoid the confusion of the user, and obtain a more satisfying interactive device.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of an embodiment of a dialogue apparatus of the present invention.

FIG. 2 is a diagram showing an output state of a recognition unit for explaining an example of processing in a speech right control unit when the location of the speech right is on the system (computer) side in an embodiment of the present invention. .

FIG. 3 is a diagram for explaining a Bayes identification method and is a diagram showing data in a multimodal dialogue database.

FIG. 4 is a diagram showing an example of processing in a speech right control unit when the location of the speech right is on the user side in an embodiment of the present invention.

FIG. 5 is a block diagram showing a schematic configuration of an embodiment of a dialogue apparatus of the present invention.

FIG. 6 is a diagram showing an output state of the recognition means for explaining an example of processing of the emotion determination means in the embodiment of the present invention.

[Explanation of symbols]

1, 1-1, 1-2 ... Input data, 2, 2-1, 2-2 ... Recognition means, 3 ... Speech right control means, 4 ... Dialogue management means, 5, 5-
1, 5-2 ... Output means, 6 ... Time assigning means, 7 ... History storing means, 8 ... Emotion determining means

Claims (5)

[Claims] 1. An input unit for inputting a voice generated by a human or the like and a facial expression of the human or the like that can be visualized, an output unit for outputting an audible voice and a visible signal to the human or the like, and the input unit. In a dialogue device having a recognition means for recognizing a user's operation state based on the signal, and a processing / control means for processing the output of the recognition means and controlling the output means, the recognition means has a plurality of predetermined values. The operation state of the user in the operation mode is output as a recognition result, and the processing / control means determines the situation of the dialogue from the recognition result, performs processing for determining the whereabouts of the speech right, and responds to the result of the processing. An interactive device characterized in that the output means is controlled to perform the output. 2. An output generation means for generating an output corresponding to a result of the processing in the processing / control means is provided, and a voice generated by the output generation means according to the situation of the dialogue and the whereabouts of the speech right. The interactive device according to claim 1, wherein a visual signal is output. 3. The time addition means for giving the start and end times of the operation state of the user recognized by the recognition means, and the time addition means used for time management of the operation in the processing / control means. The dialogue apparatus according to claim 1 or 2. 4. The plurality of predetermined operation modes are the volume of voice, the height of voice, the direction of the line of sight and the vertical swing of the head, and the operation state of each of the modes is recognized. Item 5. The dialog device according to any one of items 1 to 3. 5. The plurality of predetermined modes are set to a voice volume,
The height of the voice and the direction of the line of sight are used, the operating states of the respective modes are recognized, the emotion of the user is judged from the recognition result, and the location of the speaking right of the user is re-determined as either the user or the device concerned. The dialogue apparatus according to any one of claims 1 to 4, characterized in that.