JP2024026341A - Equipment and programs etc. - Google Patents

Abstract

To provide a device and a program for improving the convenience for a user or another device in communicating with the device.SOLUTION: A robot 1 has a function of outputting voice and a function of communicating with a user. The robot 1 interacts with the user using an interaction engine, converts the content of the previous speech of the user into character string data simultaneously with the interaction, and displays the character string data on a touch panel unit 7 serving as a display unit. The user can visually confirm what the user has spoken, thereby contributing to correction or directionality of subsequent communication.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The present invention relates to a device, a program, and the like having a function of, for example, communicating.

Patent Document 1 discloses a technology related to an interactive communication robot.

Japanese Patent Application Publication No. 2011-000681

However, the problem with conventional communication robots is that they do not have sufficient capabilities. Therefore, it is an object of the present invention to provide a device, a program, etc., which have better capabilities than the conventional ones.
The purpose of the invention of the present application is not limited to this, and we intend to acquire rights through divisional applications, amendments, etc. for structures that aim to obtain effects from the parts of the structure disclosed in this specification, drawings, etc. . For example, the present specification discloses a problem in which the phrase ``can be done'' is replaced with ``the problem is.'' Each of the problems is described as independent, and we intend to obtain rights for the structure to solve these problems independently through divisional applications, amendments, etc. Even if the problem is understood implicitly from the description, the applicant has the intention to claim a part of the structure described in the specification in an amendment or divisional application. . Also, issues that combine these independent issues are also disclosed.

(1) A device having a function of communicating by outputting output information to at least one of a user or another device, the device having a function of controlling generation of the output information for the communication; It is preferable to include a function of controlling the timing of outputting the output information for the communication.
In this way, at least one of the user and the other device can obtain output information whose generation is controlled or output information whose timing is controlled. It is possible to provide a device that is superior to conventional ones.
The device may control the generation of the output information for communication to generate output information corresponding to communication, for example. In this way, it is especially convenient for the user or other equipment to communicate with the device. Communication may be performed by outputting any kind of output information, but it is particularly preferable to store history information of past communications and perform communication based on that history information as well. Further, it is preferable to perform this based on history information of communication with a plurality of different users or other devices. In particular, the output information may be output from one output means, but it is preferable to have a configuration in which a plurality of different output means can be used, and one of these output means is selected to perform the output. As the output means, for example, audio output means, display means, communication means, etc. may be used. Communication may be performed using audio, visual, or other five senses, such as touch.

It is particularly preferable that the "device" has a function of outputting the output information in voice. In this way, it is possible to control a device that operates by inputting voice, for example. The device may in particular be provided with an interface for communication and may be provided with decision means for carrying out the communication.
Note that the portion including the configuration of the "apparatus" may be constructed from a plurality of casings, but is particularly preferably constructed from one casing.
Further, the device may be, for example, a system having a function of accessing a network via wire or wireless. In particular, for example, a smartphone, a tablet terminal, a smart speaker, a smart camera, etc. may be used. Further, although the appearance is not limited, it is particularly preferable to use a device that looks like it can communicate with others. Especially good for robots. For example, it is particularly preferable to use a robot that imitates a person or an animal, or an anthropomorphic form other than these.

The device is preferably provided with an input-side interface for communication, and may be an interface with an input device such as a keyboard, for example, optical character recognition (OCR) that reads characters and converts them into data. It would be good to have an audio interface as the side interface. As for audio data, it is preferable to have a function of acquiring audio data based on an audio signal converted into an electrical signal using a microphone, for example.
It is preferable to provide an audio interface as the output side interface, for example, a speaker device, earphones, etc. It is preferable to have a visual interface as the output side interface, for example, it is preferable to have a display that can change the display contents, and for example, it is preferable to have a display device such as a liquid crystal display (LCD), a plasma display (PDP), an organic EL display, a cathode ray tube, etc. Good. Furthermore, it is preferable to provide output in the form of printed matter, for example. In particular, it is preferable to provide an interface on the output side that actually generates movement. It is preferable to include an actuator as a device that actually generates movement. For example, it is preferable to include a motor or the like. In particular, the device may be a robot equipped with members that actually generate movement. In particular, the device preferably outputs output information as a movement of a member that actually generates movement. In particular, it is preferable to provide an audio interface, a visual interface, and an interface that actually generates movement as an output interface.
The "user" is, for example, a person who can handle the device, and may be one person, but preferably more than one person.
The "other device" may be the same or different in appearance, function, etc., from the specific one of the above devices. Other devices may not have the function of outputting audio. It is good to have an audio output function. Further, other devices do not need to have a voice input function, but are preferably equipped with a voice input function.
Other devices may be devices that cannot access the network, but are preferably devices that can access the network. In particular, it is preferable to use a device that can access the Internet.
The output information may be configured to cause the output means to output a certain amount. A "certain output" is, for example, a notification to the outside. Even if it is not an obvious form of ``notification,'' just the fact that there is a change as a result of it can be interpreted as ``notification.''"A certain output" does not necessarily have to be for the purpose of notification. For example, the output of sound or light with or without any information is good, such as a change in some physical quantity, movement of an object, etc.

(2) The device may have a display function that changes the display mode on the display unit depending on the voice communication.

When communicating by voice, the display mode on the display unit is changed in relation to the voice communication, so it is possible to communicate by changing the voice to a visual display, increasing convenience when communicating.
The "display section" is preferably a device that changes the display mode on the display section according to the voice communication, such as a liquid crystal display (LCD), a plasma display (PDP), an organic EL display, a cathode ray tube, etc. A display device like this is good. In particular, it is preferable to include a display section in the device.
"Display in such a way that the display mode on the display section changes depending on the communication by voice" refers to the case where the voice of the user or other device is displayed on the display section and its mode is changed, and the voice of the device itself is also displayed. It is possible to have only one of them displayed in the section and change its aspect, but it is especially good to have both.In this case, it is possible to display either one or both, but only one is displayed It is preferable to have a configuration that has both a state in which the screen is displayed and a state in which both are displayed and can be switched. For example, an example in which only one side is displayed is the case where the face screen S1 is displayed in the robot 1 of the first embodiment below, and an example in which both are displayed is the chat screen in the robot 1 in the first embodiment below. This is the mode when S2 is displayed.
As for the display mode, for example, the screen may be changed in various ways in relation to the audio, for example, it is preferable to perform an animation in which an object displayed on the display screen moves according to the audio. For example, it may be possible to change the image as the sound is output, or to superimpose another image on top of the image as the sound changes. Also, for example, audio data may be converted into character data and displayed. It is preferable that the display of the character data changes from moment to moment according to changes in the voice.
The communication by voice is preferably controlled by a function that controls the generation of the output information or a function that controls the timing of outputting the output information for the communication, and in particular, the communication controls the generation of the output information. It is preferable that the control is performed by a function that controls the function and the timing of outputting the output information for the communication.
Further, the function of changing the display mode on the display unit may be controlled by a function of controlling the generation of the output information or a function of controlling the timing of outputting the output information for the communication, and in particular, the function of controlling the output of the output information for the communication. It is preferable that the control is performed by a function of controlling the generation of the output information and a function of controlling the timing of outputting the output information for the communication.
Similarly, in (3) and later, the configuration for outputting from the device is preferably controlled by a function that controls the generation of the output information or a function that controls the timing of outputting the output information for the communication. In particular, the communication may be controlled by a function that controls the generation of the output information and a function that controls the timing of outputting the output information for the communication.

(3) It is preferable that the change in the display mode on the display unit is based only on utterances from at least one of the user and the other device.

The device changes the display mode on the display based on the utterances from the user or other devices, allowing the user or other devices to visually see whether their utterances are being recognized by the device, improving communication. It will be more convenient when planning. In addition, the special communication through different types of human interfaces feels fresh and interesting.
The utterance may be made directly from the user or from another device, or may be made indirectly by converting the utterance into text data, for example. Alternatively, the utterance may be converted into some corresponding information, such as another sound or a visualized pattern, and the display mode may be changed on the display unit based on the converted information.
Since (3) is "based on the utterances of at least one of the user and other devices," the device itself may have an audio communication function, for example.
The configuration based on the utterance of at least one of the user or the other device may include, for example, converting the voice into a character string using a voice recognition function and interpreting the content of the utterance of at least one of the user or the other device. It is preferable to have a configuration for specifying.

(4) The display mode on the display section is changed alternately with audio output from the device.
Communication is not one-sided, and it is possible to communicate in a stable manner. "Alternating" may mean, for example, a configuration in which communication between the device side and the user or other device side basically proceeds in an interactive manner, and it is preferably a configuration in which only one side speaks unilaterally.

(5) The display mode may include character information obtained by converting utterances from at least one of the user and the other device.

In this way, users and other devices will be able to concretely understand how their own utterances are being recognized by the device from the displayed text information, and will be able to see whether or not they are communicating correctly. It can be determined from the content of text information. You can also visually confirm what you have said. Also, if you misunderstand something, you can say it again or rephrase it in a different way to guide correct communication.
For example, in the case of Japanese, "character information" is characters based on the utterances of the user or other devices, such as regular kanji, hiragana, and katakana, and if the utterances constitute a sentence, Sentences should contain a mixture of kanji, hiragana, katakana, foreign language, etc. When a foreign language is spoken, such as English or Chinese, it is preferable to display those characters.
For example, it may be equipped with a voice recognition function and a voice output function, and display on a display unit the contents of the utterance of at least one of the user or the other device, which is recognized by the voice recognition function and converted into text information. It is particularly preferable to have a function of generating response character information based on the content, converting the response character information into voice information using a voice synthesis function, and outputting the voice information.

(6) It is preferable that the entire contents of the characters into which the utterance is converted from the start to the end of the utterance are simultaneously displayed on the display unit.
Since the entire utterance of a certain length from the user or other device is received by the device, correct communication from the device in response to that utterance can be expected. In addition, users can instantly visually check what they have said, which contributes to correcting and directing future communication.
The time from the start to the end of speech should be set taking into account the amount of time a person can speak in one breath. The end of speech may be, for example, when the volume of sound continues for a predetermined period of time at which it is considered that there is no speech. Speech may be started, for example, on the condition that the volume of the sound exceeds a predetermined level. Speech may be started, for example, on the condition that a sudden change in the level of the sound level is greater than or equal to a predetermined level is detected.

(7) The device may have a function of displaying a dialogue history of voice communication between the device and at least one of the user and the other device as text information on the display unit.

It is easy to see how the device interacted with the user or other equipment, increasing the convenience of voice communication.
It is preferable to display the conversation history in such a way that it is possible to see which party has had the conversation, for example. For this purpose, for example, it is preferable to provide a speech bubble to distinguish which utterance the text information is based on. It is good to be able to check past conversations by scrolling on the screen. In order to indicate that it is an interactive format, it is preferable to display different avatar characters on the device side and on the user or other device side. It is good if the date and time of the utterance are displayed at the same time, as this will give the user an opportunity to recall the past from the dialogue history.

(8) When displaying the dialogue history as text information, the device may display a message to that effect on the display unit if there is a change in the user or the other device and the device's dialogue target changes. It would be good to have a function to do this.

If the object of interaction with the device changes, the content of the interaction will also change. By displaying that the subject of dialogue has changed, for example, when viewing past dialogue history, the reader can see that the content of the dialogue has changed before and after that, allowing them to see the break in the content of the dialogue. Become.
The device may have a function of detecting a change in the user or the other device. For the detection of replacement, it is preferable to have a function to detect from changes in the characteristics of the voice, but it is also preferable to have a function to use a camera to acquire and detect the status of surrounding people or equipment. It is good to have a configuration that does this.

(9) Preferably, the device has a function of displaying a voice recognition status of at least one of the user and the other device on a display unit of the device using a voice recognition function.

For example, when a user or another device is speaking, by making the user understand that the user is definitely listening, it is possible to make the user understand that the conversation is occurring smoothly.
For example, as a visual function, depending on the degree of voice recognition, the display unit can change the color of the display screen or objects displayed on the display screen, for example, or display different objects depending on the voice recognition status. It is preferable to display a quantitative display according to the degree of speech recognition, for example, if the speech recognition is well recognized, a high numerical value is displayed. As a function through auditory sense, for example, it is good to make the sound louder or quieter depending on the degree of voice recognition, and for example, it is good to change the tone color.
In particular, it is preferable to have a configuration in which an anthropomorphic display is displayed on the display section and the facial expression thereof is changed.

(10) A function is provided for performing the communication by performing the voice output using the result of recognizing voice and converting it into a character string, and the result of recognizing voice and converting it into a character string is It is preferable to provide a function of outputting the output content corresponding to the character string as a voice when there is a part that matches the character string stored in the storage means that stores the correspondence between the character string and the output content.

If a character string converted by recognizing voice matches a character string stored in the voice storage means, if the necessary voice output can be performed only within the device, it will be possible to quickly respond to utterances from the user or other devices. I can comply. Additionally, since there is no connection to an external server, connection costs can be reduced.
For example, it is preferable to prepare a large number of built-in scenarios as character strings stored in the storage means. Built-in scenarios are scheduled interactions, such as a simple greeting such as "How are you" from the device in response to a "Good morning" from the user, or a conversation to perform a certain process, for example. Scenarios such as ``Open the settings screen'' (user), ``Are you sure?'' (device), ``Yes'' (user), and ``Okay, let's open the settings screen'' (device) are good examples. As a storage means, for example, a ROM or SSD inside the computer, an external SD card, a microSD card, a CD-ROM, etc. are preferable.

Here, "the case where there is a part that matches the character string" refers to the case where there is a match with a completely stored character string, and the case where the regular expression may be different in some part. A regular expression is a language processing method that expresses a set of character strings as a single string. For example, when you say "××× louder××", you can say "Yupibo, turn up the volume" or "hey, please turn up the volume." This is a case where even if there are different parts, such as "Please turn up the volume," or "Please turn up the volume," if the main parts are the same, the expression is interpreted as "Turn up the volume.""Outputting the output content corresponding to the character string" means, for example, outputting a voice that says "Yes, I will increase the volume" in response to the character string "increase the volume". Good output. Furthermore, the device may perform certain processing following such audio output. For example, after uttering "Yes, turn up the volume," the device may actually increase the volume of its own utterances in subsequent interactions.
The process of recognizing voice and converting it into a character string may be performed by the device, but it is also possible to send voice data to a voice recognition server connected to a network and convert the character string converted by the voice recognition server. It may also be done by receiving the information. It is desirable to have both, and it is good to have a function to decide which result to use depending on the communication situation.

(11) If the result of recognizing speech and converting it into a character string does not match the character string stored in the storage means that previously stored the correspondence between the resulting character string and the output content, the dialog It is preferable to have a function of connecting to a server equipped with an engine and outputting audio data.

When there is no part of the character string obtained by recognizing and converting the voice that matches the character string stored in the voice storage means, a connection is made to a server equipped with a dialogue engine, thereby reducing the cost for connection.
The server may be, for example, a device that is connected using an Internet line and has computer functions as a storage unit and a control unit. The present invention preferably includes a dialogue engine. In the case of an external server, connection can be made using, for example, an ID, password, or electronic authentication. A cloud server is recommended as the external server. Preferably, the server includes a speech recognition engine, and the speech recognition engine is capable of converting speech into character string data. The converted character string data may be sent to the device using an internet line.
The function to connect to a server equipped with a dialogue engine and output voice data is, for example, to send a voice-recognized character string to the dialogue engine, and receive a character string containing the dialogue content corresponding to the character string from the dialogue engine. , it is preferable to convert the character string of the dialogue content into audio data using a speech synthesis function.
The conversion of character strings into voice data may be performed by a speech synthesis engine provided in the device, but it is also possible to send the character strings to a voice recognition server that converts character strings into voice data, and to convert the character strings from the voice recognition server. It is preferable to perform this by receiving audio data corresponding to the character string.

(12) Even if the result of recognizing speech and converting it into a character string has a part that matches a character string stored in a storage means that stores in advance the correspondence between the resulting character string and the output content, It is preferable to provide a function that connects to a server equipped with a speech recognition engine and outputs speech data when conditions are met.

If there is a part of the character string that is converted by recognizing the voice that matches the character string stored in the storage device, outputting a fixed voice that the user can predict will also reduce the user's desire to interact. Therefore, connecting to an external server in this way allows for more human-like dialogue.
For example, when a user utters "Hello," and the device recognizes it, the original scenario is a built-in scenario that prompts a conversation like "Hello, how are you doing?" It is better to request voice data for "Hello" from an external server without using a scenario, and request creation of response data for that "Hello" using the external server's dialogue engine. The certain condition may be, for example, once every few times or random timing.

(13) A function for detecting when the voice is interrupted and becomes silent after voice recognition, a storage means for storing voice data from voice recognition to the silence state, and a function for storing the voice data stored in the memory means. It is preferable to have a function of connecting to a server equipped with a voice recognition engine and outputting voice data when a silent state occurs.

There is often silence in dialogue. However, even in a silent state, if the device remains connected to a server equipped with an external speech recognition engine, unnecessary costs will be incurred. Therefore, by storing the audio data from speech recognition to silence in advance in a storage means and sending that audio data at the timing of silence rather than in real time, it is possible to cut out the silent portion. Costs can be reduced.

(14) The device preferably has a recording function, and a function of connecting to a server equipped with a voice recognition engine and outputting voice data by detecting voice at a predetermined sound pressure level.
If the device is always connected to a server equipped with an external speech recognition engine, unnecessary costs will be incurred. As a result, the connection is not made unless the dialogue is silent or close to silent, and the external server is connected only when a necessary dialogue starts, thereby reducing connection costs.
A server equipped with a speech recognition engine may receive recorded data of a predetermined period in the past that has already been recorded by the device, and return a character string corresponding to the recorded data, but in particular, it may receive recorded data of a predetermined period in the past, which has already been recorded by the device, and return a character string corresponding to the recorded data. It is best to use a type that receives audio data and returns a character string. Although there are many cases where a pay-as-you-go charge is made in the form of a number of yen per hour for receiving audio data, it is possible to significantly reduce costs.

(15) An example of a dialogue selected from the dialogue data stored in the storage means for the user when the server is busy when connecting to a server equipped with a speech recognition engine and outputting audio data. It is preferable to provide a function to output audio.

When a user is busy, it is normal to make a notification to that effect, but such notifications are sudden and unrelated to the conversation, for example, and can potentially disrupt the conversation. . Therefore, instead of notifying you that you are busy, for example, you can use a call to say "Can you come again?" or a transitional utterance such as "Huh, I see." It becomes possible to connect to the engine and continue appropriate dialogue, and the dialogue does not become unnatural.

(16) Provided with a function to output a dialogue example selected from the voice data stored in the storage means without connecting to a server equipped with a voice recognition engine when it is determined that the recognized utterance of the user is too long. It is better to do this.

If the user's utterance is too long, the speech recognition engine may misrecognize it. As a result, you may receive an irrelevant answer. Therefore, when a sentence exceeds a certain level, in order to eliminate such a possibility and restart the conversation, say things like "Yeah," "Seriously?" or "Is it true?" It is best to select and output audible dialogue examples that can be taken in any way, allowing the user to continue the dialogue appropriately.

(17) In the communication through dialogue, when the voice of the device is missed, the device may re-output the previous voice based on a certain utterance of the user.
For example, if you cannot hear or forget to hear what the device said just before, such as "Say it again" or "Speak again," you can use this call to listen to the words spoken by the device just before. Can make words speak. This makes it possible to continue the conversation that was occurring just before without interruption.

(18) When the voice cannot be recognized, it is preferable that the device outputs a voice to prompt the user to speak again.
This makes it possible to continue the conversation that was occurring just before without interruption.

(19) When the recognized voice content satisfies a certain condition, a certain display may be displayed on the display unit.
For example, when an utterance containing a predetermined word is made and the speech is recognized, the display section is made to display "a certain display" corresponding to the word. The "predetermined word" may be, for example, the user's birthday, the name of the user's child, the nickname of the device, the name of the company, a specific advertising catchphrase, or the like. It is preferable to have a function of presetting a correspondence between a predetermined word and a certain display. As a result, it is possible to communicate not only by simple dialogue but also by visual inspection, increasing the number of modes of communication with the device, and increasing the convenience of communication.

(20) The device preferably has a function of moving the casing or the part connected to the casing, and when the recognized voice content satisfies a certain condition, the casing or the part connected to the casing may make a certain movement. .
For example, when an utterance containing a predetermined word is made and the speech is recognized, the casing or a part connected to the casing is made to make a certain movement, such as a gesture, corresponding to the word. This allows communication to include not only simple dialogue but also the movement of the device, increasing the number of modes of communication with the device, and increasing the convenience of communication. It is especially good when combined with the above “certain indication”.

(21) The device includes an eye that is a part that the user can recognize as an eye, a user position recognition function that recognizes the user's position, and a function that moves the eye, and the position recognition function serves as the communication. It is preferable to have a function of moving the eyes so as to face the direction of the user's position recognized in the above.

When the eyes, which can be recognized as eyes, face the direction of the user's position, the user can get a pseudo-sensation of talking to a person, increasing the desire to communicate with the device, and increasing the value of using the device. will improve.
The ``eyes, which are the parts that the user can recognize as eyes'' may be eyes as objects displayed on a display screen, or may be eyes that actually operate mechanically rather than in such a virtual image. The device itself may also be controlled to face in the direction of the user's position in synchronization with the eyes. Only the device for directing the eyes in the direction of the user may be controlled to direct in the direction of the user's position.

(22) The device preferably has a face recognition function that recognizes the user's face.
Since it is possible to identify the faces of individual people, it becomes possible to communicate in a way that is tailored to each person's individuality. For example, by associating each person's recognized face with their name, it is possible to call the person whose face has been recognized during a conversation by that name. Furthermore, it is particularly preferable to have a configuration in which a dialogue is performed specifically for a person whose face has been recognized based on past dialogue history.

(23) The device preferably includes a display unit, and has a function of displaying the recognition status of the user's face on the display unit using the face recognition function.
In this way, the user can understand the state of recognition of his or her own face by the device by looking at the display section. In particular, it is a good idea to display whether face recognition has been completed or whether it has not yet been recognized as a person's face as a recognition status.In this way, the user can see if recognition has not yet been completed and the device recognizes the face as much as possible. You can cooperate by keeping your face still to make it easier.

(24) The position recognition function determines the position of the sound source, including the triangle, based on three microphones placed at the vertices of the triangle and the difference in arrival time of sound from the sound source to each of the three microphones. a sound source direction identification unit that identifies a sound source direction from a position projected onto a plane including the triangle along a direction perpendicular to the plane to a reference position located inside an area surrounded by the triangle on the plane; It would be nice if it were a function.
This allows the three microphones to specify the direction of the sound source. If the direction of the sound source can be specified, the device can be directed in that direction when the user speaks, allowing the user to feel as if they are communicating through dialogue.

(25) The device preferably includes an infrared remote control signal output section, and the communication is preferably communication with the other device having an infrared remote control receiving function.
This makes it possible to easily communicate with the device via the infrared remote control signal output section of the device.
For example, an infrared remote control signal is output from the device to a receiving device equipped with an infrared remote control signal receiving section, such as a television, an audio device, an air conditioner device, etc. It becomes possible to execute control such as ON/OFF. In particular, it is preferable that the device is equipped with a voice interaction function, so that when a command phrase such as "turn on the TV" or "turn off the TV" is uttered, the device controls the infrared remote control signal output section based on the command.

(26) The device may be remotely controlled via the Internet from the other device.
The convenience of the device is increased because the device can be remotely controlled from other devices. For example, a smartphone or the like may be used as another device. The device may be equipped with a camera. The device may be equipped with a mechanism for changing the direction of the camera. It is preferable to access from another device and, for example, view videos from the camera on the device side or change the direction of the camera. This makes it possible to control the device without being near the device. Furthermore, it is preferable to access the system from a smartphone, turn on the monitoring function of the device, and report the movement of a person (object) to the smartphone by e-mail. Furthermore, it is preferable to assume that the device is always moving, for example to watch over a sick person or a cared person, and to notify the user if the person has not moved for a certain period of time or more. The other device may be, for example, a tablet terminal or a personal computer.

(27) Preferably, the device outputs the audio using text information transmitted from the other device via the Internet.
For example, it may be provided with a function to read out the character strings of received e-mails. By setting the device to receive e-mail from someone, the contents of the e-mail will be output audibly from the device, eliminating the need to visually check one's own terminal. Other devices may be, for example, smartphones, tablet terminals, personal computers, etc., but may also be other "devices."

(28) It is preferable that the time, time, or number of times the audio output is performed can be changed depending on the content of the text information.
For example, if you send an email that says ``I took some medicine,'' I want it to be sent at a set time. For example, if the subject lines match, the device will speak at a predetermined time, or, for example, you can have the device speak important information twice with a gap in time, so that a user near the device can misunderstand the contents of the email. It can be executed without any problem.

(29) The device preferably has a function of transmitting voice-recognized character information to the other device via the Internet.
By using the voice communication function of the device to convert voice into text and sending it as text data to another device, for example, if you want to send an e-mail, you can send it without manually inputting it into your own terminal.

(30) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function of receiving character strings output as , selecting the longest output character string, and having the user interact with the character string.
When the response is the longest, it feels like a conversation, the monotony of the conversation disappears, and the listener (user) can enjoy the conversation.

(31) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function of receiving a character string outputted as , select the output character string with a question mark at the end, and having the user interact with the character string.
When a question mark is added at the end of a word, the flow of the conversation begins to further answer that question, making it easier for the conversation to continue and for the listener (user) to enjoy the conversation.

(32) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function to receive a character string outputted as , combine it with an affirmative sentence, and then combine it with a question sentence to have a dialogue.
By arranging your responses in this way, your responses will appear as if you have put a lot of thought into your sentences, making users feel as if their utterances are being taken seriously, and making them want to continue the conversation. It becomes easier to continue the conversation, and the listener (user) can enjoy the conversation. In addition, it is possible to obtain an output scale and to request a response from the listener (user).

(33) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function to receive character strings output as , and have a conversation by combining them so that the character string that changed the topic is placed at the end.
By arranging the conversation in this way, changing the topic creates a conversation that invites the next utterance, making it easier to continue the conversation.

(34) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function to receive character strings outputted as , and to combine and interact so that a friendly output character string is placed first.
Arranging things in this way makes it easier for the listener (user) to be drawn into the conversation, making it easier to continue the conversation.

(35) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function to receive character strings output as , combine them in a random order, and interact with them.
This increases the variety of dialogue, so even if the listener (user) makes the same utterance, they will no longer receive exactly the same response, making it easier to continue the dialogue without getting bored with it.

(36) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. If a character string output as is received and there is an output character string that includes an emoticon among them, it is not used as an interaction target, and is displayed on the display unit together with the output character string that is an interaction target. It would be good to have a function that allows
Although emoticons cannot be output as audio, by deliberately displaying emoticons on the display and making them part of the dialogue along with the audio, it is possible to create an unusually interesting dialogue.

(37) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function of receiving character strings output as , selecting only one of the output character strings containing the same character string, and combining it with the other output character strings to interact with each other.
This is because if the same string of characters is repeated, the dialogue becomes tedious and the listener feels uncomfortable.

(38) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. It is preferable to have a function of receiving character strings output as , converting the endings of the output character strings using an ending conversion engine, and then combining and interacting with the endings of the output character strings.
This is good because the expressions are more approachable than the sentences produced by ordinary dialogue engines.

(39) Sending the voice-recognized character string as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and transmitting the output character string from the different servers. receives a character string output as , stores some of the output character strings in a storage means without using all of the output character strings, and retrieves them from the storage means in a subsequent dialogue to use them in the dialogue. It would be good to have a function that allows
By using it when voice recognition fails or when a response from an external server is slow to arrive, it allows the conversation to continue without interruption, contributing to natural dialogue.

(40) When using a server equipped with a speech recognition engine, it is recommended to use a mix of free servers and paid servers.
This allows, for example, especially heavy users of device interaction to save on server connection fees.

(41) Preferably, the other device is a smart speaker, and the device communicates with the smart speaker by outputting audio to the smart speaker.
Since smart speakers do not have a display, they can be used in combination with other devices for added convenience.
The smart speaker may be, for example, a speaker that has a wireless communication connection function and a voice operation assistant function. For example, GoogleHome, AmazonEcho, LINE Clova, etc. may be used. For example, the smart speaker may be equipped with functions that realize various functions and abilities (skills). By speaking to the smart speaker from a voice communication device, the smart speaker can perform its functions. When making a speech from a device, for example, the user utters a phrase that activates a smart speaker skill, the device recognizes the speech, saves it as string data, and at a certain timing synthesizes the string data into speech. It is recommended that the skill be executed by speaking to the smart speaker.

(42) Preferably, the other device is a smart speaker, and the device communicates with the smart speaker by outputting audio to the smart speaker.
Activate a smart speaker or activate a skill. Or make inquiries. It becomes possible to automatically execute smart speaker skills at a certain fixed timing or at a certain scheduled time without having to activate the smart speaker yourself.

(43) Preferably, the other device is a smart speaker, and the device has a translation function for translating the audio output of the smart speaker.
This is useful if you find it tedious to listen to the smart speaker's long, predetermined answers to questions, or if you want to quickly review the content.

(44) The audio output of the device preferably has a function of reading out web articles.
Even if you don't read the web article, you can listen to it just by interacting with the device.

(45) Preferably, the certain output is performed when execution of a predetermined processing unit of robotics process automation is completed.
With robotic process automation, it is difficult to understand the execution status of each processing unit, but by having the equipment produce a ``certain output'' according to the execution of the processing unit, the processing status becomes easier to understand and convenience increases.

(46) The certain output may be a notification operation.
It is known that a certain output has been made by the notification operation.
(47) When the computer that is executing robotics process automation enters a state of waiting for input from the user, a notification operation is performed.
This allows the user to be notified of the input waiting state and prompts the user to proceed with the next process.

(48) A client computer that performs robotics process automation, and a server computer that instructs the client computer to execute robotics process automation, and when the client computer receives an instruction from the server computer, performs a notification operation. It is a good idea to do so.
This makes it possible to notify the user that there has been an instruction from the server computer and prompt the user to proceed with the next process.

(49) Preferably, the output information is generated to perform an action pointing in the direction of a computer executing robotics process automation.
This allows the user to know on which computer the execution was performed, making it possible to prompt the user to proceed with the next process.
(50) The output information may be generated differently depending on the execution state of the robotics process automation.
This allows you to distinguish what kind of execution was performed.

(51) A program for causing a computer to implement the functions of the device according to any one of (1) to (50).
For example, a part written as "certain" such as "certain output" may be written as "predetermined".
The inventions shown in (1) to (50) above can be combined arbitrarily. For example, a configuration may be adopted in which at least a part of the configuration of at least one invention following (2) is added to all or part of the configuration of the invention shown in (1). In particular, it is preferable to create an invention in which at least a part of the structure of at least one invention after (2) is added to the invention shown in (1). Further, arbitrary configurations may be extracted from the inventions shown in (1) to (50) and the extracted configurations may be combined. The applicant of this application intends to acquire rights to inventions containing these structures. Furthermore, even if there is a description of "in the case of..." or "at the time of...", the description is not intended to be limited to those cases or times. We have also disclosed these cases and other configurations, and we intend to acquire the rights. Furthermore, the sections described in order are not limited to this order. It also discloses a configuration in which some parts have been deleted or the order has been changed, and we have the intention to acquire the rights.

When communicating with a user or other device, the device can generate output information in response to the communication. Therefore, it is more convenient for users or other devices to communicate with the device.
The effects of the invention of the present application are not limited to these, but the effects obtained from the parts of the structure disclosed in the specification and drawings are also disclosed, and the rights to the structure that achieves the effects have also been acquired through divisional applications, amendments, etc. have the intention to do so. For example, in this specification, a portion where it is written as “can be done” is a description that clearly indicates the effect to be achieved, and there are also portions that show an effect even if there is no description that “can be done.” Furthermore, even without such a description, there are effects that can be understood from the configuration.

1 is a front view of a robot according to a first embodiment of the present invention. FIG. 3 is a side view of the robot according to the first embodiment. FIG. 3 is a rear view of the robot according to the first embodiment. FIG. 2 is a block diagram illustrating the electrical configuration of the robot. An explanatory diagram capturing an example of a certain expression on a face screen displayed on a robot's face. An explanatory diagram capturing an example of a chat state with a chat screen displayed on the robot's face. FIG. 2 is an explanatory diagram illustrating a standby image with a face screen displayed on a robot's face as a background. FIG. 3 is an explanatory diagram illustrating a standby image with a chat screen displayed on the robot's face as a background. (a) to (d) are explanatory diagrams illustrating deformation patterns of eye objects displayed on the robot's face. (a) to (c) are explanatory diagrams illustrating how the content of the user's utterance gradually appears as a character string on the robot's face. FIG. 2 is an explanatory diagram illustrating a state in which an eye object is moving following a user's face on a face screen displayed on a robot's face. An explanatory diagram illustrating an example of a smartphone. An explanatory diagram illustrating the relationship between robot startup, wake-up mode, dialogue mode, and sleep mode. FIG. 7 is an explanatory diagram illustrating the relationship between a robot and a smart speaker in Embodiment 7. FIG. 9 is an explanatory diagram illustrating an example of a smartphone in Embodiment 9.

<Embodiment 1>
As shown in FIGS. 1 to 3, a robot 1, which is a communication robot that operates in response to human voices, has a fixed part 2 that is the lower body, and a movable part 3 that is the upper body that is placed on the fixed part 2. It is equipped with a casing. The movable part 3 includes a body part 4 disposed adjacent to the fixed part 2 and a head part 5 supported by the body part 4. The fixed part 2 is formed to have an upwardly open bowl-like appearance, and the body part 4 is formed in a cylindrical shape that is formed by an upper edge of the fixed part 2 and a continuous curve in the vertical direction. The robot 1 has the largest diameter at the connecting portion between the fixed portion 2 and the body portion 4, and has an outer shape that narrows in the vertical direction with the connecting portion as a boundary. The body part 4 has a cylindrical shape with a large notch in the front part having a semicircular shape. The head 5 is fitted into the upper part of the body 4 so as to be embedded therein. The body 4 rotates horizontally (in the direction of the arrow in FIG. 1) with respect to the fixed part 2, and the head 5 rotates vertically (in the direction of the arrow in FIG. 2) and in the left-right rotation direction (in the direction of the arrow in FIG. 3) with respect to the body 4. It rotates in two directions (in the direction of the arrow).

The head 5 as a whole has a truncated spherical shape, which is a portion of a spherical body (front part) cut off in one plane. The cut-shaped front portion appears in a circular shape and constitutes the face portion 6 of the robot 1. A rectangular portion formed on the surface of the face portion 6 is a touch panel portion 7 serving as a display portion that is a liquid crystal display (LCD) with a touch panel function. The contents displayed on the touch panel section 7 will be described later. A smoke panel is arranged in the area of the face part 6 around the touch panel part 7, so that the entire face part 6 has a uniform dark-colored background. Inside the head 5, an illuminance sensor 8 and a high-intensity white LED 9 are disposed at housing positions on the left and right sides above the face section 6, respectively. A lens 11 of a face recognition camera 10 is disposed at the upper center of the touch panel section 7 in the face section 6 .

Microphones 12 are disposed inside the body 4 at three positions at the same height, shifted by 120 degrees, at the front left and right positions of the body 4 and at the rear center position. A pair of left and right speaker devices 13 are disposed on the fixed portion 2 inside the fixed portion 2 . An opening 14 for outputting sound generated by the speaker device 13 is formed on the side of the speaker device 13. A power switch 15 and an up switch 16 and a down switch 17 for adjusting the volume of the speaker device 13 are disposed adjacent to the speaker opening 14, respectively. A USB OTG (On-The-Go) terminal 18, a DC 12V power jack 20, and a micro SD card socket (reader) 19 are provided at the rear of the fixed part 2.

The robot 1 is capable of connecting to a predetermined external cloud server using an Internet line. The cloud server includes a storage area as a storage means for storing data required by the robot 1, various engines for performing various processes required by the robot 1, and the like. Therefore, in a broad sense, the robot 1 can be interpreted as a device that includes software and other parts of these cloud servers.

Next, the electrical configuration of the robot 1 according to the first embodiment will be described based on the block diagram of FIG. 4.
The above touch panel section 7, illuminance sensor 8, high brightness white LED 9, face recognition camera 10, microphone 12, speaker device 13, terminal 18, and micro SD card socket 19 are connected to the controller MC as a control means. In addition, a wireless LAN device 21, a Doppler sensor 22, first to third motors 23 to 25, etc. are connected, respectively.

The touch panel section 7 has an input operation function for inputting information by touching its surface. The touch panel unit 7 can display different screens as shown in FIG. 5 or 6 in a natural dialogue mode to be described later. The controller MC causes the touch panel section 7 to display, as a first screen, a face screen S1 that controls the expression of the robot 1 as shown in FIG. 5, particularly changes in the area around the eyes, on the touch panel section 7. The face screen S1 is a screen displayed by default, and the eye object 27, cheek object 28, and elliptical area 29 of the robot 1 are displayed. The eye object 27 includes several deformation patterns of the eye object 27 as animation images (FIGS. 9(a) to 9(d)). Furthermore, the pupil object 27a moves from side to side as an animation image.

Further, the controller MC causes the touch panel section 7 to display a chat screen S2 as shown in FIG. 6 as a second screen. The chat screen S2 can be displayed on the touch panel section 7 instead of the face screen S1 by touching and sliding the touch panel section 7 while the face screen S1 is displayed. The display of the face screen S1 and the chat screen S2 can be switched between each other by a slide operation. Chat screen S2 will be described later.
Further, the touch panel section 7 can display the standby image of FIG. 7 or FIG. 8 on the touch panel section 7 in the standby mode (wake-up mode). The standby image will be described later.
The robot 1 is provided with a setting screen as a different image other than these, and it is possible to move to the setting screen from the chat screen S2. When the robot 1 is started for the first time, you can access it from the settings screen and, for example, register your ID/password to the server, register your Wi-Fi password, user registration (for example, name, age, gender, etc.), and facial recognition. , enter the necessary initial setting items such as setting the e-mail address to which the data will be transferred.

The illuminance sensor 8 recognizes the brightness of the environment in which the robot 1 is installed. The high-brightness white LED 9 is automatically turned on when the light intensity is insufficient for photographing by the face recognition camera 10 based on the value detected by the illuminance sensor 8.
The microphone 12 is an audio input means for acquiring the user's utterances during dialogue with the user, and at the same time, by using the three microphones 12 placed at the vertices of the triangle at the same time, the arrival time of sound between them can be adjusted. It is also a direction detection means that can identify the direction of the sound source based on the difference in the direction of the sound. The controller MC determines the arrival time difference from the phase difference between the electrical signals acquired by each microphone 12. The controller MC calculates the sound source angle with respect to the reference direction based on the arrival time difference. For example, a microphone specialized for dialogue with the user may be provided on the face part 6.
The speaker device 13 is a voice output means through which the robot 1 speaks (voice output) during dialogue with the user.
The microSD card socket 19 reads and rewrites data on the inserted microSD card.
The wireless LAN device 21 is a device for wirelessly connecting the robot 1, which is a Wi-Fi compatible device, to the Internet. In this embodiment, the international standard is IEEE802.11b.
The Doppler sensor 22 is a sensor that uses microwaves. It emits microwaves and compares the frequency of the reflected microwaves with the frequency of the emitted radio waves to determine whether an object (person) is moving. To detect. It takes advantage of the fact that the frequency of reflected waves changes when an object (person) is moving due to the Doppler effect. For example, this is a device used to detect abnormalities around the robot 1, such as the presence or absence of a suspicious person when the user is not present.
The first motor 23 is a servo motor for rotating the body portion 4 in the horizontal direction (in the direction of the arrow in FIG. 1) with respect to the fixed portion 2. The second motor 24 is a servo motor for rotating the head 5 in the vertical direction (in the direction of the arrow in FIG. 2) with respect to the body 4. The third motor 25 is a servo motor for rotating the head 5 in the left-right rotation direction (in the direction of the arrow in FIG. 3) with respect to the body 4. When the direction in which the user speaks is determined by the microphone 12, the controller MC controls the first motor 23 so that the face part 6 faces directly toward the user, and moves the torso part 4 (movable) relative to the fixed part 2. Rotate part 3).

The controller MC is composed of a well-known CPU, ROM, RAM, memory as a storage means such as SSD, a bus, a real-time clock (RTC), and the like. Various programs for executing various functions of the robot 1 are stored in the ROM of the controller MC.
Various programs include, for example, a dialogue program for controlling dialogue with the user via the microphone 12 and the speaker device 13, a face recognition program related to face recognition using the face recognition camera 10, a touch panel unit 7 and the first to first A display variation/gesture program for controlling the motors 23 to 25 of 3 to change the facial expressions and movements of the robot 1 during interaction with the robot 1, and a display variation/gesture program that controls the motors 23 to 25 of 3 to change the facial expressions and movements of the robot 1 during interaction with the robot 1. A screen display program that displays images on the touch panel section 7, a data transmission/reception program that processes camera images acquired on the robot 1 side, e-mails from smartphones, etc., with other computers and smartphones, and a data transmission/reception program that processes e-mails from smartphones, etc. A program for when the user is away from home for monitoring purposes, a GUI function/internet connection function, an OS for operation/operation such as process management, etc. are stored. Input/output data, etc. for dialogue and face recognition are temporarily stored in the RAM. Each program can perform multitasking functions, such as dialogue, facial recognition, and gestures, either in conjunction with other programs or independently.

A. Regarding the contents of operations during dialogue In the configuration described above, the controller MC controls communication through dialogue with the user by executing a dialogue program. Note that face recognition that becomes possible in synchronization with the start of dialogue will be described later in "B. Operation details during face recognition" below.
Here, the dialogue program is
1) Issue a request to the cloud server for the user's utterance data (voice data) acquired from the microphone 12, and respond with the user's utterance data (character string data) converted into text using the speech recognition engine on the server side. Subprogram 2) Built-in scenario subprogram that executes the built-in scenario dialogue based on the user's utterances (string data) 3) If the user's utterances do not correspond to the built-in scenario, the utterance data (string data) is sent back to the cloud server The utterance data transfer subprogram 4) issues a request to the robot 1 and causes the dialogue engine to create response data (character string data) for the robot 1 using the dialogue API (application programming interface). Character string data display subprogram to be displayed on the touch panel section 7 as a display section 5) Converts the response data (character string data) into voice data by a voice synthesis engine and outputs the voice as speech from the robot 1 side from the speaker device 13 6) A display mode/motion variation subprogram that changes the display mode on the touch panel section 7 and the motion of the robot 1 based on the character string data on the user side and the character string data on the robot 1 side;
Including etc.
Hereinafter, an example of the control contents of the controller MC mainly based on the dialogue program will be described together with the mutual relationship between the standby mode (wake-up mode) after startup, the natural dialogue mode, and the sleep mode. Their mutual relationship is as shown in FIG.
5 and 6 are screens in natural dialogue mode, and FIGS. 7 and 8 are screens in standby mode. In the sleep mode, these screens are darkened because the backlight of the touch panel section 7 is turned off.

1. Startup The robot 1 is started by turning on the power switch 15 (process M0 in FIG. 13). A boot program is executed in the controller MC, and then when the OS is started, the OS enters a state of waiting for a "command" from a user, that is, a wake-up state. In this initial standby mode, the standby screen shown in FIG. 7 is displayed.
The following describes the state after the initial settings are completed, that is, the ID and password of robot 1 are registered on the cloud server, user registration is completed, facial recognition of multiple users is performed, and the e-mail address of the smartphone is changed to robot 1. It will be registered in , etc. on subsequent startups.
“1. “Effects on startup”
In this way, upon startup, one screen (FIG. 7) selected from a plurality of standby screens is first displayed. In other words, a fixed screen will always be displayed at startup. Since robot 1's eyes are closed (suggesting that it cannot interact), the user can easily tell that it is in standby mode.

2. Standby mode (wake-up mode)
The standby mode is a state in which it is possible to interact with the robot 1 when there is a trigger to start the natural interaction mode. It is also the state to which a transition occurs when the dialogue ends in the natural dialogue mode. Also, if the natural dialogue mode is not started for a certain period of time, the device will enter sleep mode. The sleep mode is a state in which it is not possible to communicate with the robot 1 through dialogue.
Here, "natural dialogue" means that the user interacts with the synthesized voice of the device (robot 1) using the built-in scenario of the robot 1 or the dialogue engine (dialogue software) on the server. The natural dialogue mode is a state in which natural dialogue is possible.
There are a plurality of screens in the standby mode, and in this embodiment, two types are prepared, as shown in FIG. 7 and FIG. 8.
FIG. 7 is a standby screen that transitions from the screen in the natural dialogue mode of FIG. 5 (process M2 in FIG. 13). It is also a standby screen displayed at startup by process M0 in FIG. 13. In FIG. 7, the layer screen of the face screen S1 with the eyes (eye object 27) of the robot 1 closed is displayed thinly in the background of the screen of the clock layer on which the date, time, day of the week, and current time are displayed in large letters. .
FIG. 8 is a standby screen that transitions from the screen in the natural dialogue mode of FIG. 6 (process M2 in FIG. 13). In FIG. 8, the layer screen of the chat screen S2 is displayed thinly in the background of a clock layer on which the date and time, day of the week, and current time are displayed in large letters. In other words, it is in standby mode, but not in natural dialogue mode.
“2. Effects in standby mode (wake-up mode)
In this way, since different standby screens are prepared, when the natural dialogue mode is started from a certain standby screen, the user can interact on the screen that was accessed immediately, which is convenient. Furthermore, by displaying a screen specific to the standby mode, the user can easily know that the robot 1 is in the standby mode.

3. Starting and Stopping the Natural Dialogue Mode After being activated and in the standby mode or sleep mode, the controller MC processes the transition to the natural dialogue mode at the following multiple timings, that is, the mode transition trigger (Figure 13). processing M1, M5). The trigger below is an example. In the natural dialogue mode, the mode switches to the face recognition mode (face recognition becomes possible) as described in "B. Operation details during face recognition" below.

1-1) In the standby mode, when the controller MC recognizes an utterance (voice) such as "Hey, Yupibo" from the microphone 12 as a startup phrase within a certain period of time, it uses this as a trigger to enter the natural dialogue mode (Figure 13 Processing M1). Further, when it is determined that there is a touch operation on the touch panel section 7, this is also used as a trigger to set the natural dialogue mode (process M1 in FIG. 13).
1-2) In the sleep mode, the controller MC determines whether or not there is a touch operation on the touch panel section 7 on the standby screen at a predetermined timing. The touch operation on the touch panel section 7 differs depending on the display mode on the face section 6. On the standby screen of the face screen S1, a touch on the entire touch panel section 7 is performed, and on the standby screen of the chat screen S2, a touch on a dialogue start button object 36, which will be described later, is performed. will be started. In other words, it will start with different operations on different screens.
If it is determined that there has been a touch operation on the touch panel section 7, the controller MC is placed in standby mode (processing M3 in FIG. 13), and then if it is determined that there has been another touch, it is placed in natural dialogue mode (processing in FIG. 13). M1).
1-3) In the sleep mode, the controller MC determines whether there is a touch operation on the touch panel section 7, as in 1-2). If it is determined that there has been a touch operation, the controller MC is temporarily placed in standby mode (process M3 in FIG. 13). In this state, if an utterance (voice) such as "Hey, Yupibo" is recognized from the microphone 12 as a startup phrase within a certain period of time, the controller MC uses this as a trigger to enter the natural dialogue mode (process M1 in FIG. 13). .
2) In the sleep mode, the controller MC determines whether a predetermined time preset by the RTC has arrived, and at the timing when the predetermined time has arrived, the controller MC enters the natural dialogue mode (process M5 in FIG. 13).
3) In the sleep mode, the controller MC outputs random utterances (audio data) from the speaker device 13 at random time intervals, for example, using random numbers generated at a predetermined timing. In other words, as a kind of soliloquy, the robot 1 outputs a voice that invites dialogue, such as "Hey, hey, what are you doing?" or "I have free time," to set the robot 1 in a natural dialogue mode (process M5 in FIG. 13). Encourage users to speak.
4) In the sleep mode, the controller MC determines whether the object (person) is moving using the Doppler sensor 22, and enters the natural dialogue mode at the timing when it detects that the object (person) is moving (process M5 in FIG. 13). ).

5) In the sleep mode, when the controller MC detects a weather change such as a weather abnormality or an earthquake, it notifies the user and uses this as an opportunity to start a natural dialogue. The external cloud server uses an abnormal weather detection engine to acquire and store information on abnormal weather conditions, including weather abnormalities (e.g., heavy snow, typhoons, etc.), earthquakes, lightning strikes, etc., at regular time intervals based on certain criteria. The constant time may be the same timing, or the timing of acquisition may be changed depending on the weather conditions. In the first embodiment, abnormal weather information is distributed from the server to the device (controller MC) using a push-type distribution system, for example. When the controller MC acquires the information, it enters the natural dialogue mode (process M5 in FIG. 13).
6) If the controller MC does not detect any utterance from the user for a certain period of time in the state of being in the natural dialogue mode in each of 1) to 5) above, it changes to the standby mode (processing M2 in FIG. 13), and then After a certain period of time, the computer enters sleep mode (process M4 in FIG. 13). The length of these mode displacement times can be changed as appropriate, for example, using the terminal device or by speaking as a built-in scenario.

“3. Effects on starting and stopping natural dialogue mode”
By providing the start of many types of natural interaction modes in this way, you will be able to interact with robot 1 at various times, increasing the number of opportunities to interact with robot 1, and thereby increasing the opportunities to enjoy interacting with nature. As a result, the user will feel the benefits of owning the robot 1. Furthermore, when the interaction mode ends, the device first enters the standby mode and then enters the sleep mode, which reduces power costs.
Furthermore, since the screen jumps from the sleep mode to the standby mode and becomes the natural dialogue mode screen, the dialogue can be started immediately, so the dialogue can be started smoothly. Further, as long as the dialogue continues, the dialogue screen (FIGS. 5 and 7) is displayed, which motivates the user to engage in dialogue.

4. Built-in scenario dialogue in natural dialogue mode In natural dialogue mode, multiple dialogue processes are available: built-in scenario dialogue and normal dialogue using the server's dialogue engine.
The controller MC first determines whether the utterance data (character string data) based on the user's utterances matches the built-in scenario, and if it does not match the conversation using the dialogue engine via the cloud server (hereinafter referred to as normal). (conversation). From the user's perspective, it seems as if he is constantly interacting with the robot 1, but in reality there are multiple internal processes in the natural interaction mode.
The controller MC executes a process of comparing the user's utterance (character string data) created by the voice recognition engine on the cloud server side with the text data of the built-in scenario (script). In this embodiment, the text data of the built-in scenario is stored in memory. A built-in scenario may be added to the SD card. With an SD card, it is easy to increase the number of built-in scenarios one after another by rewriting.
When the controller MC recognizes the user's utterance, it determines whether the character string data matches the scheduled regular expression or non-regular expression, and if it matches, the voice synthesis engine converts the script corresponding to the character string data into voice data. , and the speaker device 13 outputs the voice as the utterance of the robot and 1.
Built-in scenarios include simple scenarios such as greetings such as ``Hello'' and ``It's nice weather today'' to encourage the user to speak, and scenarios to request processing based on the user's utterances. Many built-in scenarios such as scenarios are set (prepared). Tables 1 to 3 disclose examples of such built-in scenarios. Of course, in reality, many built-in scenarios are set in addition to these built-in scenarios.

If the dialogue is not performed according to the built-in scenario, the dialogue in the built-in scenario ends midway. Examples of cases in which dialogue is not performed according to the built-in scenario are as follows.
1) When it does not match the planned regular expression or non-regular expression This is a case where the built-in scenario does not match the regular expression or non-regular expression from the beginning or midway through. This also includes cases where the user's utterances could not be correctly captured due to poor diction. In this case, the controller MC determines that it is a normal conversation, immediately connects to the external cloud server, issues a request for speech data to the external cloud server, and sends the string data to the dialogue engine on the external cloud server. Create formatted response data. Then, the response data is converted into voice data by the voice synthesis engine, and the voice is output from the speaker device 13 to continue the conversation.

2) When the dialogue in the built-in scenario ends as planned For example, when you say to the user according to the scenario, for example, "Can I do XXX?", if you say "Yes" or Possible cases include a case where there is no dialogue because a positive utterance such as "Please" is made and the dialogue in the built-in scenario ends as planned, or a case where dialogue ends in the middle of the scenario. In this case, the device enters standby mode after a certain period of time.
3) When the user's utterance is negative regarding whether or not to proceed with a certain process When the user utters according to the scenario, for example, "Can I proceed with XXX?" , the built-in scenario also ends if there is a negative utterance such as ``No'' or ``It was a mistake'' instead of a positive utterance such as ``Yes'' or ``Please''. ) or similar to 2).
At the time of this negative utterance, the controller MC confirms whether or not it is okay to stop the process by asking, ``Are you sure?''. This makes it possible to respond to mistakes made by the user, changes of mind, etc. For example, in a power off scenario, if you ask the user, "Do you really need to turn off the power?" according to the scenario, and the user says "yes," then ""Do I really need to turn off the power?" is repeated multiple times (for example, three times in the embodiment), and if the answer is "Yes," the dialogue in the built-in scenario ends.

“Effect of using built-in scenario”
With built-in scenarios like this, all interactions do not need to be requested to an external server and can be handled within the device, reducing communication costs for connecting to the server, and reducing communication time and server-side Since no calculation time is required, the user no longer feels uncomfortable when the conversation is interrupted due to a delay in responding to the user's utterances. In addition, for example, by providing such a built-in scenario when executing a predetermined process, the user does not have to operate the touch panel unit 7 or access the robot 1 from another terminal to execute the process. It is user-friendly as it eliminates the need for dialogue and allows processes to be executed interactively.

5. Requests and responses in normal dialogue On the other hand, if the utterance data (character string data) is not a built-in scenario, the controller MC connects to the cloud server, sends the utterance data (character string data) to the server again, and receives the response data from the dialogue engine. request creation. If the user has been authenticated, the controller MC sends authentication information for each user (for example, ID and password) at the beginning of the speech data in the request. In that case, past dialogue information is taken into account to create response data. On the other hand, if the user cannot be authenticated, no authentication information is sent as the user has not been identified in the past, so past interaction information is not taken into consideration. When the cloud server receives a request, it uses the dialogue API (Application Programming Interface) to have the dialogue engine create response data (character string data) based on the utterance data (character string data), and sends it to robot 1 (controller MC). respond. If there is a history of past user interactions, response data is created that takes this information into account. The controller MC converts this response data into audio data and causes the speaker device 13 to output it as speech from the robot 1 side.
Similar to the built-in scenario above, if the user remains silent for a certain amount of time, the system will enter standby mode.
“5. Effects on requests and responses in normal dialogue”
Unlike the built-in scenario, by connecting to an external cloud server and having normal conversations, it is possible to quickly perform advanced dialogue analysis using a much larger amount of data than the built-in scenario, making it as if you were actually having a conversation with a person. High-level dialogue can be achieved.

6. Regarding the actions of robot 1 during dialogue, the controller MC performs the following actions when dialogue is being performed in natural dialogue mode, regardless of whether it is a built-in scenario or normal dialogue. ～d. Execute control of gestures such as
stomach. Gestures of the robot 1 in startup, natural dialogue mode, and standby mode The controller MC controls the first to third motors 23 to 25 of the robot 1 at the following various timings to change the posture of the robot 1. Below is an example.
1) At startup: If the face part 6 of the head 5 is not facing the front or if the head 5 is tilted, it is moved to the default position of the front.
2) When touching the screen: Same as 1) (to make the face recognition camera 10 face the user directly in face recognition)
3) When the trigger utterance "Hey Yupibo" occurs: Same as 1) (to make the face recognition camera 10 face the user directly in face recognition)
4) When detecting the voice direction: Turn the face part 6 of the head 5 in that direction.
5) As a special emotional utterance, for example, when the user is happy: The third motor 25 is controlled to rotate the head 5 in the left and right directions (clockwise and counterclockwise) with the face 6 facing the user.
6) As a special utterance, for example, when the user is sad: The second motor 24 is controlled so as to keep the head 5 nodding and still for a while, and then return it to the default position.
7) Emotions are expressed as utterances, such as when greeting utterances such as "good morning,""hello,""goodevening," and "hello" are generated: The second motor 24 is controlled to bow the head 5.
8) When utterances of simple positive communication terms such as "I see,""Isee,""Isee," and "Yeah!" occur as non-special emotional utterances: Nod the head 5. The second motor 24 is controlled as follows. In steps 6 to 8), it is preferable to change the speed and time of the second motor 24 so that sadness, bowing, and nodding are different.
9) When a simple negative communication term such as "No", "I can't", or "No" is uttered as a non-special emotional utterance: Rotate the movable part 3 left and right several times. The first motor 23 is controlled as follows.
10) In response to special emotional utterances, non-special emotional utterances, and various dialogues, various gestures may be made, such as rotating the head 5 in some direction to the left or right (clockwise and counterclockwise), back and forth, This may be combined with rotating the entire movable part 3 left and right, rotating it largely, or rotating it in a small nodding manner.
This gesture of the robot 1 may be combined with the following display on the touch panel section 7 (face section 6).
“6. Effect 1 on robot 1's behavior during dialogue
By having the robot 1 make these gestures, the user becomes familiar with the robot 1, enjoys dialogue with the robot 1, and at the same time learns the pleasure of actively interacting with the robot 1.

B. Change in display mode in the state of the face screen S1 1) When the user is speaking When the controller MC acquires the user's speech from the microphone 12 and recognizes it, a face as shown in FIG. 5 on the touch panel section 7 is displayed. On the screen S1, an animation is displayed in which the elliptical area 29 is displayed in blue and the area (that is, the size) of the area changes in accordance with the user's utterance volume. Specifically, when the volume of the user's speech increases, the controller MC expands the elliptical region 29 while maintaining the elliptical shape, and when the volume decreases, the controller MC contracts the elliptical region 29 while maintaining the elliptical shape. Also, the cheek object 28 is displayed in green.

Further, the controller MC causes the touch panel unit 7 to display the user's utterance data (character string data) responded from the cloud server in a predetermined manner.
For example, as shown in FIG. 5, when the touch panel unit 7 is on the face screen S1, when an utterance such as "Hello" is acquired from the user, the controller MC displays "Hello" on the layer screen of the face screen S1 based on this utterance. Display the string. In terms of order, this display starts before the robot 1's utterance, which is a response to the user's utterance.
As a display mode, for example, the face screen S1 is displayed from a transparent state to gradually become opaque as shown in FIGS. 10(a) to 10(b), and finally the face screen S1 is displayed as shown in FIG. 10(c). The face screen S1 is completely hidden. In other words, the character strings are displayed gradually. The state shown in Figure 10(c) in which only this character string is displayed brightly against a dark background is stopped and displayed for a very short period of time, and then the layer screen in which the character string is displayed in reverse is shown in Figure 10(c). 10(c)→FIG. 10(b)→FIG. 10(a), and the face screen S1 is returned to the default state. At this time, all the character strings in one utterance appear and disappear at the same time. This display mode is an example, and the display may be displayed in a different mode.
“6. Effects on robot 1's behavior during dialogue Part 2
This allows users to visually see the words they say on Robot 1, allowing them to check whether Robot 1 has heard them correctly, determine whether the dialogue is occurring correctly, and prevent strange and off-topic dialogue. You can guide the dialogue to avoid this. Also, if a conversation is off-topic, it's easy to get irritated, but by checking it, you can understand the reason, so you can change the way you speak and try the conversation again.
In addition, since the user's utterance data, both the targets of built-in scenarios and the targets of normal dialogue, are requested to be converted into string data to the cloud server, there is no need to take the time to perform the prerequisite processing to convert such characters into string data. Since creation of response data by the dialogue engine is requested for the first time after the column data, the user's utterances can be displayed on the touch panel section 7 at least before the robot 1's utterances based on the response data, and the order of the dialogue can be adjusted. There is no risk of making a mistake.

When utterances from the user are expressed as character strings, the length varies depending on the utterance and is therefore not the same. Furthermore, if the sentence is a "sentence" that has clauses instead of words, it may become quite long. Even when the utterance is such a long sentence, the controller MC adjusts the font size of the character string so that the contents of one utterance are all displayed on the touch panel section 7 at the same time. That is, the shorter one utterance is displayed in a larger font, and the longer one utterance is displayed in a relatively smaller font.
“6. Effects on robot 1's behavior during dialogue Part 3
As a result, no matter what the user says, it can be confirmed with a single visual glance, making it difficult to interrupt the conversation until the full text appears, and making it difficult to overlap the next user's utterance. Furthermore, since each utterance appears at the same time, the entire sentence can be understood at once, and even if the display time is short, the user can fully understand it. In addition, since the character string is spread over the entire touch panel section 7, each character can be displayed in a large size, making it easy for the user to see, and even if the display time is very short, it can be sufficiently confirmed.

2) When the robot 1 is speaking, the elliptical area 29 is displayed in red on the face screen S1 as shown in FIG. Display the animation that will be displayed. Specifically, when the output level from the speaker device 13 increases, the controller MC expands the elliptical area 29 while maintaining its elliptical shape, and when the volume decreases, it reduces the elliptical area 29 while maintaining its elliptical shape. Also, the cheek object 28 is displayed in light red.
“6. Effect 4 on robot 1's behavior during dialogue
In this way, the display mode on the face screen S1 changes depending on the alternating interaction between the user and the robot 1, and it is interesting that the interaction takes place not only during the actual interaction but also on the screen, making the conversation more lively. become.

C. Changes in Display Mode of Chat Screen S2 The display mode of the chat screen S2 of the touch panel unit 7 that accompanies dialogue will be described based on FIGS. 6 and 8. As described above, the display changes from the face screen S1 to the chat screen S2 depending on the user's operation.
First, the configuration of the chat screen S2 will be explained again.
As shown in FIG. 6, a user object 31 as an avatar character is displayed at the lower left position of the chat screen S2, and a robot object 32 is also arranged to face the user object 32 at the lower right position. A different user object 31 is prepared for each authenticated user recognized in a face recognition mode to be described later or for an unauthenticated user, and a different object is displayed depending on the user currently interacting. In the central region, balloon objects 33 in which the contents of the dialogue between the user side and the robot 1 side are arranged in character strings are displayed in order along the time axis. A dialogue stop button object 34 is displayed on the upper left side of the chat screen S2. A setting button object 35 is displayed on the upper right side of the chat screen S2.

On the chat screen S2, balloon objects 33 are displayed so as to be added every moment according to the dialogue between the user side and the robot 1 side. In the speech bubble object 33, the user's utterance contents converted into character string data and the utterance contents of the robot 1 also converted into character string data are displayed in a line along the time axis and can be displayed on the chat screen S2, The latest speech content is displayed in a new speech bubble object 33 at the bottom of the past speech bubble object 33 column in synchronization with the speech.
The direction of the speech bubble object 33 is indicated so that it can be determined whether the speech is from the user or the robot 1. Since it is not possible to display all of the dialogue history on the screen at once, the chat screen S2 has a screen configuration that can be scrolled in the vertical direction, so that the balloon object 31 can be displayed going back to the past. When not going back to the past, the balloon object 33 of the most recent dialogue is always displayed.
In Embodiment 1, if the dialogue is restarted after the dialogue ends and the standby mode is entered, and the user who is recognized again in the face recognition mode described later is changed, 33 columns of speech bubble objects are displayed. ``User change'' is displayed in the middle, and a different user object 31 is displayed depending on the user whose user object 31 has been recognized anew.

Furthermore, by touching the dialogue stop button object 34 on the chat screen S2, the user actively interrupts the dialogue and enters a standby mode. In this case, the standby screen of the chat screen S2 of FIG. 8 will be displayed instead of FIG. 6, but a dialogue start button object 36 will be displayed in place of the dialogue stop button object 34. To return to the natural dialogue mode, touch the dialogue start button object 36 to return to the chat screen S2 in FIG. 6.
“6. Effect 5 on robot 1's behavior during dialogue
In this way, the user object 31 of the user and the robot object 32 of the robot 1, which are in a dialogue relationship, are arranged so as to face each other, and the speech bubble objects 33 with which they have interacted are lined up between them, giving the chat feeling that they are actually having a dialogue. It can be received from screen S2.
You can also check your past chat history later, so you can use chat instead of a diary. Additionally, since you can see who has had what kind of conversations, you can also check data such as who in your family uses the service the most. Since the message "User replacement" is displayed, you can see that the dialogue is interrupted at that point, and there is no confusion when reading the past history.

D. Special gestures in normal dialogue During normal dialogue, when the cloud server determines that a specific word is included in the user's utterance data, it responds with a command to perform a special gesture along with character string data. Based on the command, the controller MC causes the user to perform the following specific actions based on the above screen display program, gesture program, and dialogue program. The following control is an example, and the control may be performed to perform other gestures, or if there are multiple commands while the user is speaking, the command may be controlled to perform the gestures consecutively or simultaneously. The following special movements may be performed separately or in combination. I. of "6. Behavior of robot 1 during dialogue" above. In place of the gesture of the robot 1 in , the following display unit may be displayed, or the following display unit displays may be combined as appropriate.

1) When a negative expression is uttered by the user in a normal conversation between people, the display on the touch panel section 7 is changed from the normal eyes in FIG. 9(a) to the display in FIG. ) to display a changing animation on the angry eyes object. In this embodiment, the eye object is stored in memory.
2) When the user utters an expression that would be fun in a conversation between normal people, the display on the touch panel section 7 is changed from the normal eyes of FIG. c) The smiling eye object is displayed in a changing animation. In this embodiment, the eye object is stored in memory.
3) When a user utters an expression that makes one feel sad during a conversation between normal people, the display on the touch panel section 7 is changed from the normal eyes of FIG. d) Display a changing animation of the sad-looking object. In this embodiment, the eye object is stored in memory.
4) When the user's child's name is spoken by the user, the second motor 24 is controlled so that the head 5 of the robot 1 performs a nodding gesture. In this embodiment, the eye gesture program is stored in memory.
5) When the name of the company that manufactures the robot 1 is uttered by the user, the head 5 of the robot 1 moves back and forth and left and right, and at the same time the body 4 repeatedly swings against the fixed part 2. The first to third motors 23 to 25 are controlled to perform gesture movements such as the following. At the same time, the text data of the company name is synthesized into speech, and the company name is repeatedly called out as voice from the speaker device 13. In this embodiment, the gesture program is stored in memory.
“6. Effect 6 on robot 1's behavior during dialogue
By performing these special actions, the user can expect unexpected actions from the robot 1 at the same time as the dialogue, and can actively enjoy the dialogue with the robot 1.

B. Regarding the operation details during face recognition 1. Starting and Stopping the Face Recognition Mode The controller MC recognizes and authenticates the user's face by running a face recognition program. The face recognition program recognizes the acquired image as a human face by recognizing the facial pattern, and also digitizes and stores the various positions of the recognized face to match the numerical data of the face registered in the past. Authentication is performed by determining the degree of The controller MC enters the face recognition mode in synchronization with the natural dialogue mode, and executes face recognition every time the mode shifts from the standby mode to the natural dialogue mode.

In the face recognition mode, the controller MC uses the face recognition camera 10 to recognize the user's face. in particular,
1) Activate the face recognition camera 10. The user's face image captured by the face recognition camera 10 is displayed on the touch panel unit 7 (face display mode). In other words, the user is prompted to look at his or her face on the touch panel section 7. This enables facial recognition processing, and by previewing it in this way, it can be determined whether the person has been authenticated in the past.
2) If the face recognition camera 10 cannot photograph the face in 1) and the face cannot be recognized within a certain period of time, the third motor 25 is driven to swing the head 5 up and down. In other words, the face recognition camera 10 is caused to scan in the vertical direction. Then, while scanning the face recognition camera 10 in the vertical direction, the first motor 23 is driven to rotate the face recognition camera 10 once around 360 degrees to perform a face recognition operation.
3) If the face can be recognized in 1) or 2), perform authentication. If the user is already registered, data of a specific authenticated user is used in the dialogue to enter the natural dialogue mode. If the user is not registered, the system recognizes the user as an unspecified person and enters the natural dialogue mode. The touch panel section 7 returns from the face display mode to either the previous face screen S1 (FIG. 5) or the chat screen S2 (FIG. 6).
4) If the face cannot be recognized in step 2), it is assumed that the person could not be recognized, and the face recognition mode and the natural dialogue mode are terminated to enter the standby mode. The touch panel unit 7 returns from the face display mode to either the standby screen of the face screen S1 (FIG. 7) or the standby screen of the chat screen S2 (FIG. 8), which is the previous standby mode.
“1. Effects on starting and stopping face recognition mode”
Since the basic idea of dialogue is to look at the other person's face while speaking, if the face cannot be recognized, the user will not be able to have the dialogue, so the user will be actively asked to recognize the face. Dialogue cannot occur unless you are face-to-face, so users can feel as if they are actually having a conversation.

2. Regarding the behavior of the robot 1 during face recognition 1) After face recognition, the controller MC causes the face recognition camera 10 to acquire images and constantly performs face pattern recognition at a fixed timing. Then, the user's face is recognized within the angle of view of the face recognition camera 10, and the default state is that the center position C of the two eyes of the user's face is at a predetermined position within the angle of view, for example, the origin at the center of the angle of view. position. When the center position C deviates from this default position, the controller MC displays an animation in which the pupil object 27a moves in either the left or right direction depending on the amount of deviation. Normally, the pupil object 27a appears entirely as an ellipse inside the white of the eye in the eye object 27, as shown in FIG. 9(a), but in a certain state when the user's face is moving, it is shown in FIG. 11. Thus, the eye object 27a is displayed as a partially hidden eye object 27, as if the user were looking in that direction.
If the user moves and the face comes out of the field of view of the face recognition camera 10 and the face cannot be recognized, the controller MC drives the first motor 23 to move the movable part 3 in the direction in which the center position C is shifted. Control is performed to rotate the entire body and direct the face recognition camera 10. The drive of the first motor 23 is stopped at the stage when the face is recognized. If the face is not recognized even after a certain degree of rotation, for example, the entire movable part 3 is rotated by 45 degrees, the controller MC stops driving the first motor 23 at that stage, and in this state, the face pattern recognition is performed at all times. Continue.
“2. "Effects on the behavior of robot 1 during face recognition"
This makes the user feel as if they are interacting with the robot 1 while being watched all the time, which makes the interaction more interesting.

2) Although the controller MC always executes facial pattern recognition with the face recognition camera 10, face recognition cannot be performed if the user is moving or is not within the field of view of the face recognition camera 10. Therefore, it is preferable to notify the user of the face recognition status as a change on the screen. In the first embodiment, the face recognition state is notified by a change in the color density of the sickle-shaped reflective object 27b, which represents the reflection on the pupil inside the pupil object 27a. In the first embodiment, the controller MC displays the face in very light blue when the face is not being recognized, darker blue when the face is being recognized, and dark blue when the face is being recognized normally.
“2. Effect 2 on robot 1's behavior during face recognition
As a result, the user can easily tell whether or not the robot 1 has recognized his or her face, so the user actively cooperates with the robot 1 to recognize his or her face, and the dialogue progresses smoothly.

C. Regarding operations when set to be away In the first embodiment, images can be transferred to the e-mail registered when set to be away in the away setting mode, that is, when set to be away. In the first embodiment, the absence setting mode is set and canceled on the setting screen that is displayed after the user touches the setting button object 35 on the chat screen S2 of the robot 1. In the following, processing based on the absentee setting program of the controller MC when the absentee setting mode is set will be described.
The controller MC performs the steps described in B. above in "2. Start and stop of natural dialogue mode" in the absence setting mode. In the standby mode in step 4), when it is determined by the Doppler sensor 22 that an object (person) is moving, the control is performed as follows.

1) If there is any moving object around the robot 1, the controller MC notifies the user of this state by e-mail. Controller MC writes the URL of the cloud server of robot 1 in an email to a terminal device, such as a smartphone, of a user (hereinafter referred to as an external user) who is not near robot 1 and whose email address is registered through the Internet line. and send it. Include expressions in the subject line of the e-mail or in the message that make it easy to understand the intent of the notification. For example, it may be a sentence such as ``It looks like someone is coming'' or an icon that indicates this.
“C. ``Effects of operations when set to be away''
As a result, by first receiving an e-mail, an external user can be notified and recognized that an object (person) is moving around a certain robot, and the external user can take countermeasures against this situation. You will be given the opportunity to take it.

2) The controller MC sends an e-mail to the external user and at the same time activates the face recognition camera 10 to acquire an image.
3) At the same time as sending an e-mail to an external user, the controller MC determines whether there is a triggering utterance within a certain period of time. For example, it is a greeting utterance such as "I'm home, Yupibo." Based on this utterance, a built-in scenario in the natural dialogue mode is started, prompting the user (in this case, the person near the robot 1 that uttered "I'm home") to perform facial recognition. When controller MC determines that the user is one of the registered users as a result of face authentication, it sends a second e-mail to the external user's terminal device. This e-mail provides information to external users that the people around the robot 1 are not suspicious. In other words, the second e-mail is to notify that the person is related to the person, such as a family member. In the e-mail, the name of the user registered based on the registration information is written as information in the subject line or in the message sent. Note that the trigger may be other than utterance, for example, by touching the touch panel section 7 and performing facial recognition to confirm that the user is a registered user.
“C. Regarding the operation when set to be away, effect part 2”
As a result, if a family member such as a child returns home while the user is away, the second e-mail will notify the user of this fact, so there is no need to take the trouble to check on the state of the home while the user is away via the smartphone.

4) In 1), the external user who received the e-mail enters the ID and password of Robot 1 to connect to the cloud server, and connects to the cloud server, especially if the second e-mail is not sent in 3). The camera image of the face recognition camera 10 provided by the cloud server can be viewed in real time on the browser. This is possible even if a second email is sent.
FIG. 12 shows an example of a user's smartphone 41, and after connecting to the cloud server, a predetermined camera image of the facial recognition camera 10 is displayed on its display screen 43, which also serves as a touch panel. Four operation icons 44a to 44d for remotely controlling the direction of the face recognition camera 10 are displayed within the camera image. By operating the operation icons 44a to 44d, the external user outputs a control command to the controller MC via the cloud server, and based on the control command, the first motor 23 or the third motor 25 is driven and controlled to control the robot 1. The head 5 and body 4 of the person can rotate to change the direction of the face recognition camera 10. Furthermore, by touching the recording button icon 45, recording can be started, and by touching it again, recording can be stopped.
Furthermore, the voice data uttered into the microphone (not shown) of the smartphone 41 is outputted from the speaker device 13 of the robot 1 via the cloud server, while the voice data uttered from the microphone 12 of the robot 1 is transmitted to the smartphone 41 via the cloud server. Audio is output from a speaker device (not shown). Therefore, the external user can interact with a user near the robot 1 using the smartphone 41 and the robot 1 while viewing the image of the face recognition camera 10.
“C. Regarding the operation when set to be away, effect 3”
This allows the external user to remotely control the direction of the face recognition camera 10 to check the surroundings of the robot 1, and for example, to check the safety situation at home when the user is away. In addition, even if family members such as children return home while you are away, you can actively contact them from outside using your smartphone in this way, contributing to good relationships with others including your family.

<Modification 1 of Embodiment 1>
Next, a first modification of the first embodiment will be described.
In the built-in scenario dialogue in the natural dialogue described above, if the user's speech is poor or other sounds are mixed in and the audio data acquired from the microphone 12 does not match the regular expression or non-regular expression of the built-in scenario, the controller The MC may output a voice such as "Please say it again" as an utterance to prompt the user to speak again, without immediately determining that it is a normal dialogue.
The controller MC makes such a prompting utterance come from the speaker device 13, and if the user makes a correct utterance according to the built-in scenario within a certain period of time, the controller MC processes the dialogue as a built-in scenario dialogue again. Make it. On the other hand, even in such a case, if the user's utterance does not match the built-in scenario with regular expressions or non-regular expressions, the system connects to an external cloud server.
In this way, the robot 1 can communicate only within the robot 1 without needlessly connecting to an external cloud server.

<Modification 2 of Embodiment 1>
Next, a second modification of the first embodiment will be described.
If the user misses the words (utterances) of robot 1 in the above natural dialogue, the user can repeat the previous utterance of robot 1 within a certain period of time by uttering an utterance requesting robot 1 to repeat the previous utterance. It may also be possible to make speech (voice output). This process is possible for both built-in scenario dialogue and natural dialogue.
The controller MC performs voice recognition to determine whether or not there is an utterance that would trigger a missed response, for example, an utterance such as "Say again" during the dialog mode. If it is determined that the user has requested to repeat the utterance after the robot 1 has uttered the utterance, the content that the robot 1 has uttered just before is output again as a voice. Then, the previous utterance is canceled and the second utterance is processed as the first utterance.
In this way, even if the user misses the conversation during the conversation, the conversation can be resumed without interruption.

<Modification 3 of Embodiment 1>
Next, a third modification of the first embodiment will be described.
In the above natural dialogue, if the user checks the user's utterance displayed on the touch panel unit 7 by the robot 1 and finds that the voice was incorrectly recognized, he/she can point it out within a certain period of time. It may be possible to modify the dialogue to be correct. This process is possible for both built-in scenario dialogue and natural dialogue.
During the dialogue mode, the controller MC receives utterances from the user that may serve as a trigger to point out that the voice recognition is incorrect, such as utterances such as "You're wrong" or "That's wrong. I'll say it again." Recognize voice to see if it was there. When the controller MC determines that the utterance occurred within a certain period of time after the user's utterance was displayed on the touch panel section 7, it outputs a voice prompting the user to speak again. For example, the content of the utterance "I'm sorry. Please say it again" is output as a voice.
and,
(1) In the case of a built-in scenario dialogue, the user's previous utterance is canceled, and the content uttered by the user is processed for speech recognition as the correct utterance.
(2) In normal dialogue, the dialogue contents such as "You're wrong" and "I'll say it again" are sent to an external cloud server as utterance data (voice data), and the user can utter them again, including such utterances. Request creation of response data by content.
In this way, even if the robot 1 incorrectly recognizes the user's utterance, it can correct the dialogue to be correct.

<Modification 4 of Embodiment 1>
For example, the following configuration, which is different from the configuration of the first embodiment, may be adopted.
(1) In the above setting, if the built-in scenario is not executed, a request is made to the cloud server and the conversation shifts to normal dialogue. In other words, if a built-in scenario was to be executed, it was set to be a built-in scenario, but I decided to make a request to the cloud server instead of responding locally under the condition that it corresponds to a built-in scenario. It's okay. The certain condition may be, for example, once every few times or at random timing.
This results in an unexpected dialogue with the robot 1, and allows for a more human-like dialogue that is less formal.
(2) In the first embodiment, the controller MC does not include a voice recognition engine, but connects to an external server equipped with a voice recognition engine to convert the user's utterances (voice data) into text. This reduces the burden on the robot 1.
However, the controller MC has a voice recognition engine in its memory, calls the voice recognition engine, and converts the user's utterance data (voice data) acquired from the microphone 12 into text using the voice recognition engine. Column data may also be created. In other words, the controller MC of the robot 1 may have the ability to convert the user's voice data into text data. This makes it possible to create character string data without using a speech recognition engine, which reduces internal processing time, for example.
(3) In the first embodiment, initial settings are made from the settings screen, but registration may also be made from outside via a cloud server using a terminal device such as a smartphone, for example. This makes setting easier and saves time, especially for people who are accustomed to using terminal devices.
(4) The first to third motors 23 to 25 may use drive means other than servo motors. Other drive means include, for example, other types of motors, hydraulic cylinders, etc.
(5) In the first embodiment, character string data is synthesized into speech within the robot 1. Using the internal speech synthesis engine in this way does not require too much data compared to directly exchanging voice data with the server, but the response data (character string data) obtained using the dialogue engine on the cloud server side It is also possible to synthesize the voice data and send the voice data to the robot 1 as a response.
(6) In the first embodiment, the configuration is such that the setting screen is accessed from the setting button object 35 on the chat screen S2, but the setting screen may be accessed by sliding the touch panel section 7. .

(7) Regarding "C. Special gestures in normal dialogue", even in the built-in scenario, if it is determined that a specific word is included in the user's utterance data, a special gesture may be executed. good. For example, when the controller MC determines that a specific word is included in the user's utterance data, the controller MC may control the user to make a special gesture in the same way as described above.
(8) If the shape of the robot 1 is different in "C. Special gestures in normal dialogue", the robot 1 may be controlled to make further different gestures. For example, if the robot 1 has arms and legs, the controller MC may control a driving means to move them.
(9) As the animation of the eye object 27 on the face screen S1, an animation that causes the eye to blink may be inserted from time to time. For example, the control is performed by inserting the closed eye object 27 in FIG. 7 from the state of the eye object 27 in FIG. 9(a). In this way, a realistic feeling as if the robot 1 is actually looking at you can be created, allowing you to enjoy the interaction with the robot 1 even more.
(10) In face recognition mode, an unregistered user was recognized as an unspecified person, but if you change from that state to the settings screen and authenticate and register as a new user. , convenient.
(11) Regarding "C. Operations when set to be away", it would be convenient to be able to set the away setting mode from a smartphone.
(12) In "C. Regarding operations when set to be away", if there is any moving object around the robot 1, the controller MC performs processing such as notifying the user of this state by e-mail. Conversely, a process may be provided to recognize that the object is moving at regular intervals, and to notify the user of this state by e-mail if there is no moving object within a fixed period of time.
For example, if there is a sick person or person to be cared for, by placing the robot 1 near the person, it is possible to watch over them on the assumption that they are constantly moving.

<Embodiment 2>
Next, a second embodiment will be described.
An infrared LED may be provided in place of the high-intensity white LED 9 of the robot 1 of the first embodiment, or in addition to the high-intensity white LED 9. At this time, if the module of the face recognition camera 10 is equipped with an infrared filter, it is removed. Since infrared LEDs are invisible to humans, if there is an intruder such as a burglary at night, there is a possibility that the high brightness white LED 9 will turn on and the intruder will be surprised and run away. On the other hand, if an infrared LED is used, it will be difficult to tell that the image is being taken, so the intruder will not run away, and it will therefore be possible to confirm and save images of the intrusion.

<Modification 1 of Embodiment 2>
Next, a first modification of the second embodiment will be described.
If the robot 1 is equipped with an infrared LED, the infrared LED may be used to control various indoor devices equipped with an infrared remote control signal receiving section. Examples of various devices include a television, an audio device, an air conditioner, and the like.
It is preferable that the robot 1 be installed in a place where it can directly see the infrared remote control signal receiving section of a device equipped with an infrared remote control signal receiving section. In the first modification of the second embodiment, the controller MC is equipped with a shape recognition program related to shape recognition using the face recognition camera 10. For example, in the case of a television, the controller MC is equipped with a shape recognition program for shape recognition using the face recognition camera 10. can be recognized as such. When the robot 1 makes an utterance such as "turn on the TV" as a trigger for outputting an infrared remote control signal to various devices of the user, the robot 1 controls the first motor 23 or the third motor based on the utterance. 25, the robot 1 is operated so that the face recognition camera 10 is rotated 360 degrees to photograph the surroundings while swinging the face 5 up and down, and the shape of the television is recognized by the shape recognition program.
When the controller MC determines that there is a television, it memorizes the direction and at the same time outputs an infrared remote control signal from the infrared LED in the direction of the object to execute controls such as ON/OFF to operate the television. When there is a trigger for the next TV, shape recognition is first performed in that direction. The infrared remote control signal can be used not only to control ON/OFF switching, but also to change the infrared frequency to respond to, for example, changing the channel of a television, or adjusting the temperature of an air conditioner. Become. Although such detailed control requires multiple types of infrared remote control signals with different frequencies, it is preferable to set the infrared frequency using the user's smartphone 41 in FIG. 12 via the server, for example.
Further, instead of searching for the direction using the shape recognition program, the directions of various devices may be obtained by operating the face recognition camera 10 via the smartphone 41 and changing its direction, and the directions may be registered.

<Embodiment 3>
Next, Embodiment 3 will be described.
In Embodiment 3, control will be described that focuses on reducing running costs associated with connections such as the use of dialogue API when using a server equipped with a voice recognition engine.
Furthermore, the dialogue program of the third embodiment includes a subprogram that can detect a silent state of the voice obtained from the microphone 12. The dialogue program also includes a recording/output subprogram for acquiring audio data of the user's speech from the microphone 12, recording it once, and outputting it to the server.
The controller MC does not connect to the server immediately when there is an utterance, but first records the audio data of the user's utterance, and connects to the server only after detecting a silence state in the recorded audio data of the user's utterance. The recorded voice data is outputted and the response data is generated by the dialogue engine. In this way, it is not always connected to the server, and there is no need to connect to the server for a long time including silent time, so the silent connection time can be cut.
One process of the speech recognition engine is dedicated to the user while the user is speaking. In other words, one process takes up a very long amount of time for a computer, such as several seconds, and as a result, the cost burden for the user using the speech recognition engine increases. By doing so, the number of processes required per unit user can be reduced, contributing to cost reduction for users.

<Modification 1 of Embodiment 3>
Next, a first modification of the third embodiment will be described.
Modification 1 of Embodiment 3 will also explain control focused on reducing connection running costs when using a server equipped with a voice recognition engine. If there is a time lag before the user starts speaking, or if the user is waiting for the user to speak but the user does not speak and the connection with the server ends due to a timeout, the speech recognition server consumes the process. This incurs costs for the user.
The dialogue program for the robot 1 of the first modification of the third embodiment includes a recording/output subprogram for acquiring speech data from the microphone 12, recording it once, and outputting it to the server. The dialogue program also includes a subprogram that detects and outputs the sound pressure level of recorded audio data.
The controller MC connects to the server when the voice data constantly recorded in a state of natural dialogue has a certain sound pressure or more, and reproduces the data being recorded. In other words, it ignores small utterances that are silent or cannot be recognized, connects to the server and outputs voice data only when there is an utterance that can be interacted with, and has the server create response data using a voice recognition engine. Make it.
This makes it possible to eliminate wasted connection time waiting for speech.

<Embodiment 4>
Next, Embodiment 4 will be described.
Speech recognition servers are expensive, so it may not be possible to prepare sufficient resources in advance, and if server resources are scarce, the server may be busy when the device tries to connect to the speech recognition server. There is.
In the normal dialogue, utterance data (voice data) is sent to the server and a request is made to create response data. The server responds by creating response data converted into string data based on the utterance data. However, if the device is busy, the response data may not be sent and an error may occur. An error message will be returned from the server.
When the controller MC of the robot 1 receives an error message from the server, it outputs a voice reply that allows the user to continue the dialogue from a built-in scenario without making the user speak to the effect that there is a server connection error. For example, it would be a good idea to wait for the server to become available by giving vague replies such as ``Say again'', ``Yeah, yeah'', or ``What is that?'', or giving some kind of support.

<Modification 1 of Embodiment 4>
Next, a first modification of the fourth embodiment will be described.
If the user's utterance is too long, the speech recognition engine may misrecognize it. Therefore, if it is determined that the recognized user's utterance is too long, it is possible to provide a function to output a dialogue example selected from the voice data stored in the storage means without connecting to a server equipped with a voice recognition engine. good.
When the controller MC of robot 1 determines that the length of the user's utterance data exceeds a certain level, the controller MC of the robot 1 responds with questions such as "Yes", "Seriously?", or "Really?" from the built-in scenario without connecting to the server. It outputs voice-like utterances that can be taken in any way in a dialogue. The utterance data may be voice data as it is, or may be data that has been converted into character string data by the controller MC or the server.
This prevents irrelevant words from being answered, and allows the user to redirect the conversation and encourage the user to engage in dialogue again.

<Embodiment 5>
Next, Embodiment 5 will be described.
In Embodiment 5, a case will be described in which a plurality of speech recognition engines are used in combination.
There are two types of speech recognition engines: a local speech recognition engine (that is, when processing is performed within the device without connecting to an internet server), and a cloud speech recognition engine that connects to the internet server and responds with dialogue data created in response to a request. be. There are multiple types of speech recognition engines, both local and cloud-based, and some are free and some are paid. Therefore, when using servers equipped with these different speech recognition engines, use a mix of free servers and paid servers.
In the cloud server to which the robot 1 is connected using the Internet line, when the robot 1 issues a request for speech data in the dialog mode and causes the speech recognition engine to create response data, the cloud server responds as follows, for example. It is good to do.
(1) Access the paid dialogue API until a certain time A set per month.
(2) Use a mixture of paid APIs and free speech recognition engines from a certain time A to a certain time B set per month. For example, a paid speech recognition engine server is used for the first few consecutive recognitions, and a free speech recognition engine server is used for subsequent consecutive recognitions.
(3) If the time exceeds a certain time B set per month, only the free speech recognition engine will be used. This is just one example, and for example, a setting may be made in which only the free speech recognition engine is used immediately after a certain time A set per month has been exceeded.
In this way, it is possible to have a conversation without exceeding the paid range. A cloud server connected to the robot 1 processes requests issued by the robot 1 by calculating paid and free hours based on a set time per month based on a program that executes such processing.
The robot 1 itself executes such processing and instructs the cloud server to use the paid conversation API or the voice recognition engine of the free server when issuing a request. Good too.
<Embodiment 5-1>
The same applies to the case where a plurality of speech recognition engines of the fifth embodiment are used in combination, and the case where a plurality of dialogue engines are used in combination.

<Embodiment 6>
Next, Embodiment 6 will be described.
If only one fixed dialogue engine is used, the responses will become a fixed pattern and the user may get bored with the conversation with the robot 1. Therefore, in Embodiment 6, in order to solve this problem, a process for arranging the output results of a plurality of dialogue engines so that the conversation does not get boring will be explained.
As the dialogue engine, not only a cloud dialogue engine but also a local dialogue engine within the robot 1 may be used.
This process may be performed on the robot 1 side to which a plurality of response data have been sent, or may be performed on the cloud server side when response data from multiple servers equipped with dialogue engines are acquired.
(1) Output the result of the engine that gave the longest response among the chat dialogue engines.
When the longest response is used, it feels like a conversation, which eliminates the monotony of the conversation and allows the user to enjoy the conversation.
A. For example, when a user says, "I'm hungry," the three engines a to c will respond with "a engine: Do you often snack?", "b engine: Haven't you eaten?" and " C engine: If the answer is "Eat something", use the A engine and output the response data.
B. For example, when a user says, "Today's weather is sunny," the three engines a to c will respond, "A engine: I'll go to the sky right now and check,""B engine: It looks like it's going to be sunny."?""c engine: Whether it's sunny or rainy sometimes determines your mood that day, right?", the c engine is employed to output the response data.

(2) Among the chat dialogue engines, those with a "?" at the end are brought to the end and output. At this time, if there are multiple answers marked with "?", they are output consecutively.
When a question mark is added at the end of a word, the flow of the conversation begins to further answer that question, making it easier for the conversation to continue and allowing the user to enjoy the dialogue.
For example, the above (1) A. The option outputs ``Do you often snack? Do you not eat rice?'' In addition, the above (1) B. If the option is ``Is it clear?'' is output.
(3) After combining affirmative sentences, interrogative sentences are combined and output.
By arranging your responses in this way, your responses will appear as if you have put a lot of thought into your sentences, making users feel as if their utterances are being taken seriously, and making them want to continue the conversation. It becomes easier to continue the conversation and the user can enjoy the conversation. In addition, it is possible to obtain an output scale and to request a response from a person.
For example, the above (1) B. If response data such as ``I'm going to go to the sky right now and check it. Whether it's sunny or rainy sometimes determines the mood of the day, right? Does it look like it's sunny?'' is output.

(4) Results from engines that change topics more frequently than others are brought and output after the results from other engines.
By arranging it in this way, changing the topic will create a conversation that invites the next utterance, making it easier to continue the conversation.
A. For example, when a user says ``Domo Domo'', the answers from the three engines a to c are ``A engine: It's okay'', ``B engine: That's right'', and ``C engine: I watch baseball. If so, then the c engine data is placed at the end and it is output as ``Yes, that's right. Do you watch baseball?''
B. For example, when a user says, "It's hard to find something," the three engines a to c will respond, "A engine: That's right,""B engine: That's true,""C engine: Family." How many people do you have in your family?'', the c-engine data is placed at the end and it outputs, ``That's right. How many people does your family have?''
(5) Output the engine that gives a friendlier response than the others first, and then output the responses from the other engines together.
By arranging things in this way, it becomes easier for the user to be drawn into the dialogue, making it easier to continue the dialogue. Whether a word is friendly or not can be determined by ranking the words relative to each other.
A. For example, the above (4)B. In the case of , the result of the b engine is output first, and it says, ``Yes, that's right. How many people do you have in your family?''
B. For example, if the user says, "I see," and the answers from two engines a to b are "A engine: Oh, that's a nice comment" and "B engine: Hmm," then the b engine It outputs the data at the end, saying, ``Hohoho, that's a nice compliment.''

(6) Combining the processes in (1) to (5) arbitrarily. This increases the variety of dialogue, so even if the user makes the same utterance, the exact same response will not be returned, and the dialogue This makes it easier to continue the conversation without getting bored.
For example, when a user says, "What is old age?", the answers from three engines a to c are "A engine: Please wait a moment,""B engine: I don't dislike support," and "C engine." ：Are you in good health now?", use the friendliest b-engine first, then combine it with an affirmative sentence, then a question sentence, and say, "I don't hate support, please wait a moment.I'm in good health now. Are you here?” is output.

(7) In the case of an engine for text output, parentheses and emoticons may return, so if these return, audio output is suppressed and output. And they will be displayed on the screen.
Although emoticons cannot be output as audio, by deliberately displaying emoticons on the display and making them part of the dialogue along with the audio, it is possible to create an unusually interesting dialogue.
A. For example, when a user says ``I'm hungry,'' the three engines a to c will respond with ``a engine: (deliberately ignored),'' ``b engine: Hello, how are you?'' and ``c engine: Hello. ”, only engine a does not output audio, but displays it on the touch panel unit 7 (display screen).
B. For example, when a user says, "It's already there," the three engines a to c will respond, "A engine: Do you do this often?" B engine: (´・ω・｀)" c Engine: Are you a night owl?'', only b is displayed on the touch panel section 7 (display screen) without being outputted as a sound.
(8) Output one of the responses that include the same character string.
This is because if the same string of characters is repeated, the dialogue becomes tedious and the listener feels uncomfortable.
For example, when a user utters "Chinese food," the three engines a to c will respond, "A engine: That's nice,""B engine: Yes, it's Chinese food.""C engine: I eat Chinese food." Are you going to China?'', the b engine and c engine have the character string ``Chinese'', so only one of them is output. For example, it outputs something like, "Oh, that's nice. Are you going to eat Chinese food?"

(9) Use a word ending conversion means, for example, a word ending conversion API to create a sense of unity. At this time, endings may be converted for all replies, but endings may be converted only for either the last sentence to be output or the first sentence to be output.
This is good because the expressions are more approachable than the sentences of ordinary dialogue engines.
For example, when a certain engine receives response data such as "Sleepy", the ending of the word is converted using the ending conversion API and output as "Sleepy nya".
(10) In case of recognition failure, only some of the responses obtained from multiple engines are used for voice output, and the remaining responses are retained and used when the next voice recognition fails or from the dialogue system. If there is no response, return the saved response.
By using it when voice recognition fails or when a response from an external server is slow to come, it allows the dialogue to continue without interruption, contributing to natural dialogue.

<Embodiment 7>
As shown in FIG. 14, the seventh embodiment is a device (system) in which a smart speaker 51 is placed near the robot 1, and the robot 1 and the smart speaker 51 are combined. The distance between the robot 1 and the smart speaker 51 is such that each other's microphones can pick up sound, and it is preferable that the robot 1 and the smart speaker 51 be placed adjacently within, for example, 1 to 2 meters.
The smart speaker 51 is a network terminal with a built-in wireless LAN device, a wireless communication function using the Internet, a telephone line connection function, etc., and is equipped with a network module, and is also a type of computer equipped with a microphone and a speaker device. . The smart speaker 51 uses a terminal device such as a smartphone to perform various initial registrations (e.g. user's name, address, telephone number, email address registration, multiple voice registrations, Bluetooth network compatible AI) via a server. device settings, etc.), connects to the Internet based on voice-registered utterances (commands) from the user, uses the server's search engine to execute predetermined processing, and transmits the results as voice information from the speaker device. Output.

The robot 1 can complement each other's functions by cooperating with the smart speaker 51. Specifically, the robot 1 and the smart speaker 51 perform the following functions using audio as an interface.
(1) Function of instructing the smart speaker 51 from the robot 1 a. For example, when a user speaks a phrase that activates a smart speaker skill, the robot memorizes the character string that is the voice recognition result of that phrase, and then uses speech synthesis to speak that character string at a predetermined timing. Make it.
In the seventh embodiment, the controller MC of the robot 1 uses a voice recognition program that analyzes the frequency components of the user's utterances to identify individual voices, and acquires the user's utterances using the voice phone 12 that distinguishes the user's utterances for each individual. It is provided with a program for recording and reproducing phrases that are stored as string data, synthesized into speech based on the string data, and played back from the speaker device 13.
The robot 1 has a function of storing, for example, an utterance that is a trigger for memorizing an utterance, such as the words following the utterance "I'm going to speak now, so please record it." Then, the user uses this function to cause the smart speaker 51 to memorize words that cause it to start up or perform some kind of processing. For example, it is good to say something like, "OK, ×××. Turn on the lights." At this time, the user's individual voice registered in the robot 1 is the user's individual voice registered in the smart speaker 51.
Then, the robot 1 is made to speak at a predetermined timing. Settings for speaking at a predetermined timing can be registered by operating a terminal such as a smartphone, for example.
B. The robot 1's "utterance at a predetermined timing" includes, for example, the robot 1 detecting some kind of change, such as a touch operation on the touch panel section 7 or the detection of an object (person) by the Doppler sensor 22.
For example, when the controller MC of the robot 1 detects a person using the Doppler sensor 22, it causes the speaker device 13 to output a voice such as "OK, XXX. Turn on the lights." In response to this, the smart speaker 51 controls the indoor lighting, which is a network-compatible AI device, to turn on. Note that the lighting to be controlled is registered in advance to be controlled by the smart speaker 51. In addition to lighting, for example, air conditioners, televisions, curtain opening/closing devices, etc. may be used as AI devices.

(2) Function to distinguish between the utterances coming from the smart speaker 51 and the user's utterances By identifying the user's individual voice and registering the settings in the robot 1, the robot 1 can distinguish whether the utterances are coming from the smart speaker 51 or the user's utterances. It is possible to provide a function to distinguish between direct utterances and direct utterances. By doing this, it is possible to control the robot 1 so that it does not respond to the sound from the smart speaker 51 which is an unregistered voice, and conversely, as shown in (1), the robot 1 can be controlled so that it does not respond to the sound from the smart speaker 51 that is not registered. By registering the voice, not only the user but also the robot 1 can give instructions to the smart speaker 51. By being able to distinguish individual utterances in this way, it also has the function of not interfering with voice operations on the smart speaker 51.
(3) Display function of utterances from smart speaker 51 The robot 1 may acquire the utterance content instructed by the user to the smart speaker 51, convert it into text, and display it on the touch panel section 7.
The controller MC of the robot 1 preferably has a program that acquires the content of speech and converts it into text by itself or by connecting to a server.
For example, the user says, "OK, XXX. Tell me about today's weather." In response, the smart speaker 51 says, "Today's weather in Okazaki City, Aichi Prefecture is sunny, the maximum temperature is 15 degrees, and the probability of rain is is 20%.'', all of these conversations are displayed on the touch panel unit 7 of the robot 1, for example, as follows.
“User: OK, ×××. Tell me about today's weather.
Smart Speaker: Today's weather in Okazaki City, Aichi Prefecture is sunny, with a maximum temperature of 15 degrees and a 20% chance of rain.
In addition, it is preferable that the controller MC of the robot 1 has a program for translating or summarizing the content converted into text into short text. The robot 1 may output the translated or summarized content by voice or display it on the touch panel unit 7.
(4) A function for robot 1 to learn by asking various questions to smart speaker 51 when no one is around. For example, robot 1 can ask questions such as "What's the weather like tomorrow?" or "Are there any incidents?" as a built-in scenario. When the user is away, the controller MC of the robot 1 outputs the question word along with a phrase to activate the smart speaker 51 at a predetermined timing (the voice of the robot 1 is the voice of the smart speaker 51). ). At this time, the controller MC uses the microphone 12 of the robot 1 to acquire and store the content of the utterance from the smart speaker 51, and causes the robot 1 to output the content as voice at a predetermined timing. The predetermined timing is, for example, when a person is detected by the Doppler sensor 22 for a predetermined period of time, or when the user speaks to the robot 1 as a built-in scenario such as "Is there any news?" .

<Modification 1 of Embodiment 7>
You may also set a function that does not interfere with the voice operations of other voice recognition devices, such as smart speakers. For example, when the robot 1 recognizes the voice of another smart speaker's startup phrase (voice recognition start word), it may stop outputting its own voice.
For example, if the voice of a startup phrase such as "OK, , if it determines that it is a registered startup phrase, it will temporarily stop its own voice output.
This allows the voice recognition device to perform its functions without interfering with voice operations.

<Modification 2 of Embodiment 7>
The robot 1 recognizes the voice recognition activation keyword of other voice recognition devices such as the smart speaker 51, recognizes the user's subsequent pronunciation to the smart speaker 51, sends a request to the cloud server, and searches it on a search engine, etc. and get the corresponding answer.
The robot 1 recognizes when the smart speaker 51 fails in voice recognition (for example, "This is an error") or when the smart speaker 51 outputs voice indicating that it cannot output an appropriate answer to the voice recognition result (for example, "Excuse me"). In this case, the robot 1 outputs the answer that it has obtained in advance. Alternatively, the robot 1 may make a request to the cloud server to obtain an answer after receiving a voice output indicating that voice recognition has failed or that an appropriate response to the voice recognition result cannot be output.

<Embodiment 8>
The robot 1 may, for example, read news articles on a website aloud in response to a user's request. The controller MC of the robot 1 requests a news article using a search engine on the server, and the cloud server responds to the request by using, for example, news data from a registered site as text data.
The robot 1 synthesizes the news data and reads it out (outputs it), and at the same time synthesizes the name of the information source of the article and reads it out (outputs it). Additionally, the URL of the article information source is displayed on the touch panel section 7 serving as a display screen, and when that URL is touched on the touch panel section 7, the contents of the page of that URL is displayed on the touch panel section 7. Do it like this.
Furthermore, when reading out an article or an information source of an article, it is preferable to output it in a way that makes it clear that it is a quotation.
For example, a case will be described in which the content of an article with the URL "https://\\\\\.jp/archives/92###" is read aloud.
``I'll read out the article on the ``XX News'' site. "...will be on sale from January 15th of this year."That's right. ”
For example, ``I'm going to read out the article on the site of.'' or ``That's right.'' You can use regular expressions to quote articles other than the source of the article, such as ``I'm going to read out the article on the site of.'' or ``That's right.'' Then, the screen display will say, ``Information for https://\\\\\.jp/archives/92###. ” The URL will be displayed. By touching this URL portion, the content of the read article is displayed again on the touch panel section 7.
``Speak in a way that the listener can understand'' means, for example, changing the tone or voice between the article part and the non-article part.
In this way, the user can understand the news content just by listening to the content read out by the robot 1 without reading the web article, and in some cases, the user can visually confirm the news content just in case.

<Embodiment 9>
In Embodiment 9, a case will be described in which invisible movements of a computer are made visible by movements of actuators of a robot.
(1) About Robotics Process Automation Robotics Process Automation (hereinafter referred to as RPA) is software that automates simple computer tasks. The software can be set up on a server or on a user's computer. An example of a case where software is set on a server based on FIG. 15 and the robot 1 of each of the above embodiments is incorporated into an RPA network will be described.
As shown in FIG. 15, a cloud server 55 and user computers 56 and 57 are connected via a network using the Internet. Further, the cloud server 55 and the robot 1 are also connected via a network. The user computer 56 is controlled by the cloud server 55 using an RPA program. In addition, the robot 1 is notified of processing information such as that the process will be executed soon, is being executed, or has been executed in a predetermined process in the RPA program executed by the cloud server 55. It has become.
The cloud server 55 causes the user-side computer 56 to sequentially process processes 1 to 4. In this embodiment, processing 1 and processing 2 are executed by the computer 56, and processing 3 and processing 4 are executed by the computer 57. More processes may be set, and the number of user-side computers 56 involved in the processing may be one or more.
Process 1 includes, for example, user access and login to the computer 56, Process 2 includes, for example, creation and sorting of a list based on data in the computer 56, and Process 3 includes, for example, customer access and login performed following Process 2. Processing 4, which involves modifying the billing details for each request, is, for example, issuing a bill, which is executed subsequent to Processing 3. In the ninth embodiment, for example, in process 3, the user is prompted to input for correction, and after the input is received, the process moves to the next process 4. In other words, after process 2, the computer becomes a standby mode until the process 3 is completed.
The cloud server 55 outputs different notification information to the robot 1 immediately before, during, and after executing these processes, and the robot 1 notifies the surroundings of the processing status based on the notification information. good. Alternatively, notification may be made once for each process.

for example,
a. The robot 1 notifies which processing is in which state by voice, different sounds, music, etc.
b. Notify on the display screen. a. You can go at the same time.
c. Since process 3 requires user input, only process 3 may be notified, or only process 3 may be a notification that is different (identifiable) from other notifications.
d. The processing status is transferred from the robot 1 to other terminal devices and notified.
e. The robot 1 moves in such a way that the processing status can be understood. For example, if the computer 56 is processing, it is controlled to face that direction. Therefore, it is preferable to recognize the direction of each computer 56, 57 with respect to the robot 1 in advance using some direction specifying means, for example, the shape recognition program described above. The robot 1 may be provided with a direction indicating member such as an arrow or an arm member, and the robot 1 may operate so that the computers 56 and 57 as the notification target are located in the direction indicated by the direction indicating member.

(2) About blockchain Blockchain is a system in which many computers record data in a distributed manner. In particular, with public blockchains, the data to be recorded and the recorded data are made public.
Therefore, Robot 1 is placed in the blockchain network, and Robot 1 notifies the user before block data is sent. It is preferable to stop the transmission by having the robot 1 output a command "wait" (that is, requesting to wait without transmitting data).

<Embodiment 10>
Although the natural dialogue mode has been described in each embodiment, a foreign language learning mode may be provided in place of or in addition to the natural dialogue mode. When providing a foreign language learning mode in addition to the natural dialogue mode, it is preferable to switch from the natural dialogue mode to the foreign language learning mode, for example, when a voice saying "switch to foreign language learning mode" is recognized in the natural dialogue mode. It is also preferable to switch from the foreign language learning mode to the natural dialogue mode when the user recognizes a voice saying "switch to natural dialogue mode" in the "foreign language learning mode".
It is often said that the best way to learn a foreign language is to make foreign friends, but not many people have such an opportunity. Therefore, it is a good idea to have a foreign language learning function, which is a dialogue system that can be your partner in foreign language learning.
It can be a combination of any first language and second language. In the following, a configuration will be explained in which a Japanese dialogue system and an English dialogue system are linked.
Basically, it would be a good idea to have a system that converses in English, and output requests such as "Say that again" in Japanese during the conversation. Furthermore, by linking with English sentence analysis web services, it has a function that provides Japanese explanations of the English sentences being spoken in response to requests such as ``explain.'' If you don't know how to say what you want to say in English during a conversation, you can request it to translate and it will output what it means in English. You can continue the conversation by reading the output English text. Since you don't have to do any research on your own, you can expect to learn English conversation smoothly without interruptions.
In foreign language learning mode, both the speech recognition engine for your native language (for example, Japanese if you are Japanese) and the speech recognition engine for the foreign language (for example, English) use speech recognition engines running on the cloud. However, in the case of a native language (e.g. Japanese), the request content is a fixed phrase and the format of the response to the request is fixed, so it is recommended to install a local voice recognition engine (e.g. inside the robot 1) and use this engine. Good to use. The same goes for the dialogue engine. Although the speech synthesis engine may be provided at any location, it is particularly preferable to provide it locally.
When voice data based on a signal from the microphone 12 is sent to voice recognition engines for both languages, both voice recognition engines return some kind of result. For example, if you throw the Japanese phrase "Tell me one more time" to both engines, the Japanese engine will return the text data "Tell me one more time" and the English engine will return "Tell me one more time." Returns random text data interpreted as English. In such a case, since the Japanese request is a fixed phrase, the text data returned by the Japanese engine is compared with the fixed phrase of the request, and if they match, it is determined that a Japanese request has been made, and It is a good idea to perform processing to separate English and Japanese by determining that English was spoken if the language is not spoken.
Although requests may be accepted in English, the inventor has found that it is desirable to be able to accept requests in Japanese, especially assuming a case where the requester does not understand English.
The function to explain English sentences in Japanese in response to requests such as "explain" can be done simply by outputting the translation of the English sentence into Japanese, but it is especially useful for explaining the words and grammar used in the English sentences. It is a good idea to output an explanation. It is especially good to output an explanation of the syntax used in the English text. To explain the syntax, for example, draw each phrase as a vertex (node) to surround each phrase, and draw edges such as line segments that indicate the relationships between related phrases. For example, it is preferable to display it on the touch panel section 7 in the form of a graph structure (particularly preferably a tree structure).
Further, it is preferable to output the displayed content in the form of audio from the speaker device 13. For example, the AP of a parsing service as explained at https://gigazine.net/news/20160602-foxtype-review/
A good configuration would be to call I, receive the result, and output the analysis result in Japanese.
For example, perform the following processing and output.
`` Process Retrieve phrase from English dialogue engine Robot I'm a fantastic robot.
People: “Say it again.”
Robot I'm a fantastic robot.
People: “Please explain.”
Processing (recognizes "explain") → Syntax analysis API call → Generates Japanese explanation from syntax analysis results Robot I is the subject, am is the verb, and robot is the object.
Fantastic is an adjective that modifies robot.
In English, it means "I'm a wonderful robot."
People: "I don't think so."
Robot Don't say it!
If you don't understand what you want to say in English, you can request someone to teach you how to say it in English in Japanese, which is a great way to keep the conversation going. If you can't make a request in Japanese, and you don't know how to say it in English, you'll have to look it up in a dictionary or translate it online, which can make you feel like you're just studying, which can be stressful. If you can make requests in Japanese, you can learn without stress as if you were just talking to a bilingual person. Continuing language learning is very important, so it is extremely important to keep learning as stress-free as possible. According to this configuration, it is possible to realize a robot 1 that can continuously perform language learning.

<Embodiment 11>
In addition to the functions described in each embodiment, it is preferable to provide a function of speaking when it is detected that a person has entered the room where the robot 1 is installed. Further, it is preferable to provide a function that makes a speech when it is detected that a person leaves the room where the robot 1 is installed.
For example, if a sensor is installed in a passageway between the room and a place other than the room, and detects when a person enters the room where the robot 1 is installed or when a person leaves the room where the robot 1 is installed, good. In particular, when there is an automatic door for entering and exiting the room in which the robot 1 is installed, it is preferable to use a sensor that detects when a person approaches the automatic door to open or close the automatic door. In particular, the first person detection sensor located outdoors across the automatic door, the second person detection sensor located indoors across the automatic door, and the person who is at the automatic door when the automatic door is open. It is preferable to connect the controller MC of the robot 1 to at least two of the third human detection sensors (sensors for preventing people from getting caught in the door) to detect entering and exiting the room. In this way, it is possible to detect the entry and exit of the robot 1 into the room where it is installed, without installing a new sensor. For example, detection may be performed by capturing the edge of a signal that rises when a person is detected by each sensor.
For example, if a person is detected by the first person detection sensor and then a person is detected by the third person detection sensor, the speaker device 13 plays a phrase such as "Welcome" to welcome the incoming person. It is best to output from For example, if a person is detected by the second person detection sensor and then a person is detected by the third person detection sensor, the speaker device 13 sends a voice saying "Thank you very much" to the person leaving the room. It is best to output from
At these times, it is preferable to move the first to third motors 23 to 25 to cause the robot 1 to face a preset automatic door from its installation position. Note that when the first human detection sensor and the second human detection sensor detect a person at the same time, it is preferable to give priority to the first human detection sensor. In this way, it becomes easier for people entering the room to notice the presence of the robot 1, and it also reduces the sense of discomfort that the person entering the room feels when the robot 1 suddenly says, "Thank you."

<Other embodiments>
(1) In each of the embodiments, the wireless LAN device 21 is provided, but instead of or in addition to this, a wired LAN device may be provided and connected to a wired LAN network. The wired LAN device may be built into the robot 1 or may be attached externally. A configuration may also be adopted in which a wired LAN device is connected to the USB OTG (On-The-Go) terminal 18. The wired LAN network is preferably configured to be connected to the Internet via a router or the like. Depending on the environment, wireless LAN communication may be unstable or connection may not be possible. For example, when the robot 1 is installed in a place where a large number of wireless LAN devices such as smartphones are present, it is preferable to configure the robot 1 to access the Internet via a wired LAN device.
(2) In each of the embodiments, an example of interaction between a person and the robot 1 is shown in a half-duplex mode, but the interaction between a person and the robot 1 may be conducted in a full-duplex mode. . For example, in the dialogue shown in Table 1, after the person says ``Open...'', the recognition of the person's voice continues even while saying ``Are you sure?'' and the voice ``Cancel'' is heard. If the robot is recognized during the utterance of ``Are you sure?'', the utterance may be interrupted if the utterance is in progress, and the robot may immediately utter ``Cancelled.''
(3) A configuration in which half-duplex interaction is performed is particularly advantageous because it can simplify configuration and processing and reduce costs. However, it is difficult to tell when the robot 1 turns on the microphone 12, and even when the user speaks, the beginning of the user's voice that the robot recognizes, that is, the beginning of the word, is often missing. The inventors have discovered. In order to solve this problem, it is preferable to display a characteristic screen at the timing when the controller MC turns on the microphone. This supports smooth conversation. As a mode of displaying a characteristic screen at the timing when the controller MC turns on the microphone, it is preferable to perform at least one of the following: 1) lighting up the four corners of the screen, and 2) displaying a microphone icon. In particular, 1) , 2) are both effective.
(4) The motors constituting the first to third motors 23 to 25 can be various types of motors such as DC motors, but stepping motors are particularly preferred, and the posture of the robot 1 is controlled by the stepping motors. It is good to have a configuration. The torque of a stepping motor changes in proportion to the current flowing through the motor. If a large amount of current flows, a large amount of torque can be obtained, but this poses problems such as heat generation and battery life. Therefore, it is particularly preferable that when the robot 1 is stationary, the minimum amount of current required to maintain its posture is passed, and that a large current is passed only when the robot 1 changes its posture. It should be noted that although it may be possible to supply all of the servo motors 23 to 25 with the minimum amount of current necessary to maintain their posture when they are stationary, only the body parts that can be supported by detent torque (torque when not energized) While the first motor 23 is not energized, it is preferable to energize the second motor 24 and the third motor 25 even when the motor is stationary, since the head portion loses detent torque.
Also, if you rotate the motor with the same torque when it is at rest, the lack of torque will cause it to step out and it will not rotate properly, so when rotating it, it is better to increase the torque by passing more current than when it is at rest. On the other hand, when the motor is stationary, it does not require the same large torque as when it is rotating, so it is better to lower the current compared to when it is rotating.
Furthermore, when changing the direction in a certain direction, it is preferable to perform control such that the speed is increased to a constant speed while accelerating, and when the target angle approaches, the speed is decelerated and stopped.
Furthermore, when a robot is driven by a motor, mechanical and electrical noise is generated. Since this noise reduces the recognition rate of speech recognition, it is preferable to control the motor to stop during speech recognition.
The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the invention disclosed herein. Of the present invention, the structure for which a patent is sought has been specified in the attached claims, but currently, even if the structure is not specified in the claims, it is not disclosed in this specification. The applicant intends to make such a configuration the scope of a patent claim in the future.
The present invention is not limited to the configuration described in the embodiments described above. The components of each of the embodiments and modifications described above may be arbitrarily selected and combined. Also, any component of each embodiment or modification, any component described in the means for solving the invention, or a component that embodies any component described in the means for solving the invention. It may be configured in any combination. The applicant intends to obtain rights to these matters through amendments to the application or divisional applications. Furthermore, even if there is a description of ``in the case of'' or ``in the case of'', the description is not intended to be limited to those cases or times. We have also disclosed these cases and other configurations, and we intend to acquire the rights. Furthermore, the sections described in order are not limited to this order. It also discloses a configuration in which some parts have been deleted or the order has been changed, and we have the intention to acquire the rights.
In addition, the applicant intends to acquire rights to the entire design or partial design by filing a conversion application to a design application. Although the drawing depicts the entire device using solid lines, the drawing includes not only the overall design but also a partial design that claims some parts of the device. For example, it is a drawing that not only includes some parts of the device as a partial design, but also includes some parts of the device as a partial design regardless of the components. The part of the device may be a part of the device or a part of the device. We intend to obtain rights not only for the entire design, but also for partial designs in which any part of the solid line part of the drawing is a broken line part.

1...Robot as a device, 41...Smartphone as another device, 51...Smart speaker as another device.

Claims (6)

A robot equipped with an interaction function that displays an image of the robot's face on a screen and interacts with a user,
Equipped with a screen that displays a clock as a standby screen for dialogue,
A robot having a function of displaying a faint image of the robot's face on the background of a clock displayed as a standby screen for the dialogue. The robot according to claim 1, further comprising a function of displaying an image of the robot's face without displaying a clock on the screen when interacting with the user. While displaying an image of the robot's open eyes as an image of the robot's face without displaying a clock on the screen when interacting with the user,
A function is provided to display an image of closed eyes as a face image of the robot, which is displayed faintly in the background of the clock displayed as a standby screen for the dialogue.
The robot according to claim 2, characterized in that: Provided with a function to switch the screen during dialogue with the user to a screen that displays the content of the dialogue as a character string as a chat screen, without displaying an image of the robot's face.
The robot according to any one of claims 1 to 3, characterized by: As the screen when interacting with the user, the content of the interaction is displayed as a chat screen without displaying the image of the robot's face. be equipped with a function to transition to a screen that displays
The robot according to claim 4, characterized by: A program for causing a computer to realize the functions of the robot according to any one of claims 1 to 5.

Images