JP2024154417A - Behavior Control System - Google Patents

Abstract

To make a robot take an appropriate action in response to an action of a user.SOLUTION: A robot inputs a text representing a user and a text representing a selected scene into an image generation model to generate an image representing the scene. The robot also analyzes a text representing an action of the user generated as a prompt to be input to the image generation model (S200-S202), extracts user information including hobbies of the user to update history data (S206), determines an action of the user based on the updated user information, acquires information necessary for a proposed action, and provides the information to the user.SELECTED DRAWING: Figure 9

Description

The present invention relates to a behavior control system.

Patent Document 1 discloses a technology for determining an appropriate robot behavior in response to a user's state. The conventional technology in Patent Document 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine an action to be taken in response to the recognized user reaction, it updates the robot's behavior by receiving information about an action appropriate to the recognized user's state from a server.

Patent No. 6053847

However, conventional technology leaves room for improvement in terms of enabling robots to perform appropriate actions in response to user actions.

According to a first aspect of the present invention, the system includes a user state recognition unit that recognizes a user state including a user's behavior, an emotion determination unit that determines the user's emotion or the robot's emotion, and a behavior determination unit that determines the robot's behavior corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having a dialogue function that allows the user and the robot to converse. The behavior determination unit selects a scene that satisfies a predetermined selection criterion based on history data including the user's behavior, inputs text representing the selected scene into an image generation model to generate an image representing the scene, and extracts user information including the user's hobbies based on the text, and determines content to be suggested to the user based on the extracted user information.

In a second aspect, the action decision unit collects related information related to the extracted user information and decides on an action to be suggested to the user, including the collected related information.

In a third aspect, the robot further includes a behavior control unit that suggests actions to the user by controlling speech from a speaker provided on the robot based on the action determined by the behavior determination unit.

In a fourth aspect, the behavior decision unit stores the extracted user information in the history data, and the behavior control unit suggests an action to the user at a timing related to the user information stored in the history data.

In a fifth aspect, the behavior decision unit stores the extracted user information in the history data, and matches the user with other users using the user information stored in the history data.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions.

1 illustrates a schematic diagram of an example of a system 5 according to the present embodiment. 2 shows a schematic functional configuration of the robot 100. 10 shows an example of an operation flow of the robot 100. 1 illustrates an example of a hardware configuration of a computer 1200. 4 shows an emotion map 400 onto which multiple emotions are mapped. 9 shows an emotion map 900 onto which multiple emotions are mapped. FIG. 13 is an external view of a stuffed animal according to another embodiment. 13 is an example of a picture diary according to another embodiment. 11 is a flowchart showing an example of a specific processing flow when the robot 100 analyzes a prompt generated when creating a picture diary P and updates information including the hobbies of the user 10 in the history data 222. 11 is a flowchart showing an example of a specific process flow when the robot 100 suggests an action to the user 10. 11 is a flowchart showing an example of a specific processing flow when the robot 100 analyzes a prompt created for a picture diary P and matches the user 10 with other users using information about the user 10, including hobbies.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

1 is a schematic diagram of an example of a system 5 according to the present embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102. In the description of the present embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. The user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. The user 12a and the user 12b may be collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described by mainly focusing on the functions of the robot 100.

The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via the communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured image data of the user 10 in the server 300, and requests the image data, etc. from the server 300 as necessary to provide it to the user 10.

The robot 100 also has an emotion value that represents the type of its own emotion. For example, the robot 100 has emotion values that represent the strength of each of the emotions of "happiness," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sorrow," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." When the robot 100 converses with the user 10 when its excitement emotion value is high, for example, it speaks at a fast speed. In this way, the robot 100 can express its own emotions through its actions.

The robot 100 also uses AI (Artificial Intelligence)

The robot 100 may be configured to determine an action of the robot 100 corresponding to the emotion of the user 10 by matching a sentence generation model using the above with an emotion engine. Specifically, the robot 100 may be configured to recognize the action of the user 10, determine the emotion of the user 10 regarding the action of the user, and determine an action of the robot 100 corresponding to the determined emotion.

More specifically, when the robot 100 recognizes the behavior of the user 10, the robot 100 automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The sentence generation model is publicly known as disclosed in, for example, JP 2018-081444 A and chatGPT (Internet search <URL: https://openai.com/blog/chatgpt>), and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured by a large language model (LLM: Large Language Model).

As described above, this embodiment combines a large-scale language model with an emotion engine, thereby making it possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100. In other words, according to this embodiment, a synergistic effect can be obtained by combining a sentence generation model with an emotion engine.

The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing the facial image of the user 10 acquired by the camera function and the voice of the user 10 acquired by the microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

The robot 100 stores rules that define the actions that the robot 100 will take based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10, and performs various actions according to the rules.

Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing". The reaction rules also define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry". The reaction rules also define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question". The reaction rules also define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad".

When the robot 100 recognizes that the behavior of the user 10 is "angry" based on the reaction rules, the robot 100 selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing", the robot 100 performs the motion of "apologizing" and outputs a voice expressing the words "apologize".

In addition, when the emotion of the robot 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, "happy" = 0) and the condition that the state of the user 10 is "alone and looks lonely" is satisfied, it is defined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to execute the action of "calling out."

When the robot 100 recognizes based on the reaction rules that the current emotion of the robot 100 is "normal" and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. In addition, the robot 100 selects the action of "calling out" defined in the reaction rules as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", it converts the words "What's wrong?", which express concern, into a worried voice and outputs it.

The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user 10's reaction was positive, and the attributes of the user 10.

The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102. The server 300 then analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.

The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

Figure 2 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.

The controlled object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors. In addition, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.

Note that, among the components of the robot 100 shown in FIG. 2, the components other than the control target 252 and the sensor unit 200 are examples of components of the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

The storage unit 220 includes reaction rules 221 and history data 222. The history data 222 includes the user 10's past emotional values and behavioral history. The emotional values and behavioral history are recorded for each user 10, for example, by being associated with the user 10's identification information. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, the attribute information of the user 10, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2, excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program that operates on the OS.

The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the user state recognition unit 230.

The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 to recognize the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts features such as frequency components of the voice, and recognizes the emotion of the user 10 based on the extracted features. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201, and outputs text information representing the content of the user 10's utterance.

The facial expression recognition unit 213 recognizes the facial expression and emotions of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotions of the user 10 based on the shape, positional relationship, etc. of the eyes and mouth.

The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.

The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely."

The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

Here, the emotion value indicating the emotion of user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the more cheerful the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("normal"), the value is 0.

In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

The emotion value of the robot 100 includes emotion values for each of a number of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of the emotions "joy," "anger," "sorrow," and "happiness."

Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 according to rules for updating the emotion value of the robot 100 that are determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

For example, if the user state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. Also, if the user state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "happy" of the robot 100.

The emotion determination unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is completely dark, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.

The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a number of predetermined behavioral categories (e.g., "laughing," "anger," "asking a question," "sad") is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.

As described above, in this embodiment, the robot 100 identifies the user 10 and acquires the contents of the user's utterance. When acquiring and using the contents of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

The behavior determination unit 236 determines an action corresponding to the behavior of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, the history data 222 of past emotion values determined by the emotion determination unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use the most recent multiple emotion values as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. In addition, the behavior determination unit 236 may determine an action corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values of the robot 100. The behavior determined by the behavior determination unit 236 includes gestures performed by the robot 100 or the contents of speech by the robot 100.

The behavior decision unit 236 according to this embodiment decides the behavior of the robot 100 as the behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotion values of the user 10, the emotion value of the robot 100, the behavior of the user 10, and the reaction rules 221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior decision unit 236 decides the behavior for changing the emotion value of the user 10 to a positive value as the behavior corresponding to the behavior of the user 10.

The reaction rules 221 define the behavior of the robot 100 according to a combination of the past and current emotion values of the user 10, the emotion value of the robot 100, and the behavior of the user 10. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.

For example, the reaction rule 221 defines the behavior of the robot 100 for all combinations of patterns of the emotion values of the robot 100 (1296 patterns, which are the fourth power of six values of "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); combination patterns of the past emotion values and the current emotion values of the user 10; and behavior patterns of the user 10. That is, for each pattern of the emotion values of the robot 100, behavior of the robot 100 is defined according to the behavior patterns of the user 10 for each of a plurality of combinations of the past emotion values and the current emotion values of the user 10, such as negative values and negative values, negative values and positive values, positive values and negative values, positive values and positive values, negative values and normal values, and normal values and normal values. Note that the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222, for example, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

The reaction rules 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotion value of the robot 100. Alternatively, the reaction rules 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotion value of the robot 100.

For each gesture included in the behavior of the robot 100 defined in the reaction rules 221, the strength of the gesture is predefined. For each utterance content included in the behavior of the robot 100 defined in the reaction rules 221, the strength of the utterance content is predefined.

The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

Specifically, if the total intensity value, which is the sum of the emotion values for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all surrounding information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).

The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

The behavior control unit 250 may recognize a change in the user 10's emotions in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotions may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotions may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. If an impact is detected by the touch sensor included in the sensor unit 200, the user 10's emotions may be recognized as having worsened, and if the detection result of the touch sensor included in the sensor unit 200 indicates that the user 10 is smiling or happy, the user 10's emotions may be recognized as having improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

In addition, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution form determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is not bad. In addition, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is bad.

Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the robot 100, it controls the control object 252 to make the robot 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the robot 100, it controls the control object 252 to make the robot 100 assume a droopy posture.

The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives updated reaction rules from the server 300, it updates the reaction rules 221.

The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

Figure 3 shows an example of an outline of an operation flow relating to the operation of determining an action in the robot 100. The operation flow shown in Figure 3 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates the step that is executed.

First, in step S100, the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210.

In step S102, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 to the history data 222.

In step S104, the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

In step S106, the behavior decision unit 236 decides the behavior of the robot 100 based on a combination of the current emotion value of the user 10 determined in step S102 and the past emotion values included in the history data 222, the emotion value of the robot 100, the behavior of the user 10 recognized by the behavior recognition unit 234, and the reaction rules 221.

In step S108, the behavior control unit 250 controls the control object 252 based on the behavior determined by the behavior determination unit 236.

In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S114.

In step S114, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 230 are stored in the history data 222.

As described above, according to the robot 100, an emotion value indicating the emotion of the robot 100 is determined based on the user state, and whether or not to store data including the behavior of the user 10 in the history data 222 is determined based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the user state 10 years from now is the same as that 10 years ago, the robot 100 can present to the user 10 all kinds of peripheral information, such as the state of the user 10 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), and data on the sound, image, smell, etc. of the location.

According to the robot 100, the robot 100 can be made to perform an appropriate action in response to the action of the user 10. Conventionally, the user's actions were classified and the action including the robot's facial expression and appearance was determined. In contrast, the robot 100 determines the current emotional value of the user 10 and performs an action on the user 10 based on the past emotional value and the current emotional value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the robot 100 can make an utterance such as "You were cheerful yesterday, but what's wrong with you today?" The robot 100 can also make an utterance with gestures. For example, if the user 10 who was depressed yesterday is cheerful today, the robot 100 can make an utterance such as "You were depressed yesterday, but you look cheerful today, don't you?" For example, if the user 10 who was cheerful yesterday is more cheerful today than yesterday, the robot 100 can make an utterance such as "You're more cheerful today than yesterday. Has something better happened than yesterday?" Furthermore, for example, the robot 100 can say to a user 10 whose emotion value is equal to or greater than 0 and whose emotion value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."

For example, the robot 100 can ask the user 10, "Did you finish the homework I told you about yesterday?", and if the user 10 responds, "I did it," it can utter a positive utterance such as "Great!" and make a positive gesture such as clapping or a thumbs up. For example, when the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can utter a positive utterance such as "You did a great job!" and make the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of closeness to the robot 100 by performing actions based on the state history of the user 10.

In the above embodiment, the robot 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect. For example, the robot 100 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.

The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. Furthermore, the functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 300 may be implemented in the cloud.

Figure 4 shows an example of a hardware configuration of a computer 1200 functioning as the robot 100 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or to execute operations or one or more "parts" associated with an apparatus according to the present embodiment, and/or to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input/output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a ROM 1230 and a legacy input/output unit such as a keyboard, which are connected to the input/output controller 1220 via an input/output chip 1240.

The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 1230 stores therein a boot program or the like executed by computer 1200 upon activation, and/or a program that depends on the hardware of computer 1200. I/O chip 1240 may also connect various I/O units to I/O controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the operation or processing of information according to the use of the computer 1200.

For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, DVD drive 1226 (DVD-ROM 1227), IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and/or analog hardware circuits and may include integrated circuits (ICs) and/or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as, for example, field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, so that the processor of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or the programmable circuit, executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

The present invention has been described above using an embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiment. It is clear from the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in that order.

(Other embodiments)

In other embodiments, the robot 100 may be applied to a robot mounted on an electrical appliance, a computer, an automobile, or a motorcycle. The robot 100 may also be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a controlled device (speaker, camera, microphone) mounted on the stuffed toy. Specifically, other embodiments are configured as follows. For example, the robot 100 may be applied to a cohabitant (specifically, stuffed toy 100N shown in FIG. 7) that creates a picture diary P based on events that include conversations with the user 10 while spending daily life with the user 10. In addition, in this embodiment, the stuffed toy is an animal-shaped stuffed toy, but the stuffed toy is not limited to an animal-shaped stuffed toy and may be another character.

As shown in FIG. 7, in this embodiment (and other embodiments), the stuffed animal 100N has the shape of a bear covered in soft cloth fabric. Although not shown, in the space formed inside, input/output devices include a microphone 201 (see FIG. 2) of the sensor unit 200 arranged in the part corresponding to the ear 54, a 2D camera 203 (see FIG. 2) of the sensor unit 200 arranged in the part corresponding to the eye 56, and a speaker constituting part of the control target 252 (see FIG. 2) arranged in the part corresponding to the mouth 58. The microphone 201 and the speaker 60 do not necessarily need to be separate bodies, and may be an integrated unit. In the case of a unit, it is recommended to place them in a position where speech can be heard naturally, such as at the nose of the stuffed animal 100N.

The robot 100 also stores events including dialogue with the user 10, who is, for example, a child. Specifically, the storage unit 220 stores events of the day captured by the microphone 201 and the 2D camera 203. Note that, as the events, the history data 222 in which the user 10's past emotion values and behavior history are stored may be used as the event data. Then, at a timing such as the end of the day, a predetermined scene is selected from the stored events, and the selected scene is converted into text. Here, the scene to be converted into text is selected from a scene in which the emotion of the user or the emotion of the robot determined by the emotion determination unit 232 satisfies a predetermined selection criterion. The selection criterion includes the strongest emotion of the user 10 and the strongest emotion of the robot 100, or the strongest emotion of a specific emotion among multiple emotions, such as "joy," or the most frequent smiles of the user, for the emotion of the user 10 or the emotion of the robot 100. For example, based on the history data 222, the scene where the sum of the emotion value of the user 10 and the emotion value of the robot 100 is the strongest is selected, the behavior of the user 10 stored in the history data 222 in the selected scene is acquired, and the text representing the acquired behavior of the user 10 is input to the image generation model. Then, for example, if the scene is a scene of going to the zoo, the text is converted to "The teddy bear and the user went to the zoo". In addition, the behavior determination unit 236 accepts an image representing the user 10 in advance, inputs the text representing the user of the image and the text representing the selected scene to the image generation model, and generates an image representing the screen. The text representing the user is a text that is a representation of the user's appearance, particularly the clothes and appearance of the user, from the image data of the event, and is made to resemble the user. For example, if the user is wearing a blue T-shirt, the text is converted to "The user is wearing a blue T-shirt".

The robot 100 also uses a preset image generation AI to create a picture diary P for the textualized scenes. The image generation AI is publicly known, for example as disclosed in DALL-E 2 (Internet search <URL: https://openai.com/product/dall-e-2>), so a detailed description is omitted. Here, the image generation AI is an example of an image generation model. The image generation AI may be stored in the storage unit 220 or in another device connected to the robot 100 via a network.

Specifically, the robot 100 inputs text representing the user and text representing the selected scene into the image generation model to generate an image representing the scene. For example, "Draw a picture diary of the user's crayon drawing of a teddy bear at the zoo" is input to the image generation AI to create a picture P1 of the picture diary P as an image representing the scene. Also, "Write a diary about the events with the teddy bear at the zoo" is input to the text generation model to create a title P2, text P3, etc. of the picture diary P. FIG. 8 shows an example of a picture diary P created by the image generation AI and the text generation model. As shown in FIG. 8, the image generation AI creates a picture P1, and the text generation model creates a title P2 and text P3. Then, the robot 100 creates a picture diary P that combines the picture P1, the title P2, and the text P3. Also, the picture P1 created by the image generation AI and the title P2 and text P3 created by the text generation model may be created in multiple patterns and made selectable by the user. Furthermore, the title P2 and the sentence P3 are not limited to those created by the sentence generation model, but may be input by the user 10.

The robot 100 then outputs the created picture diary P and transmits it, for example, by email to a predetermined person, including the user 10. Predetermined people would preferably be the user 10's parents or grandparents who live far away.

In addition, in this embodiment (other embodiments), a teddy bear 100N is exemplified, but it may be another animal or a doll. The skin may also be removable. Furthermore, the material of the skin is not limited to cloth, but may be other materials such as soft vinyl, although a soft material is preferable.

A display may be attached to the skin of the stuffed animal 100N to add a control object 252 that provides visual information to the user 10. For example, a display may be provided on the abdomen to display the picture diary P. The eyes 56 may also function as a projector to project the picture diary P onto a wall. The picture diary P may also be associated with a calendar function so that when a date is selected with the calendar function, the picture diary P for that date is displayed.

The robot 100 (including the robot 100 having the appearance of a stuffed toy 100N) executes a process of creating a picture diary P through the following steps 1 to 4.
(Step 1) The robot 100 acquires the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, the history data 222, and event data. Specifically, the robot 100 performs the same processes as in steps S100 to S103 described above to acquire the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, the history data 222, and event data.

(Step 2) The robot 100 selects a scene to be included in the picture diary P.
Specifically, at the end of each day, the behavior determining section 236 looks back on the events of the day based on the history data 222 and selects scenes to be included in the picture diary P.

(Step 3) The robot 100 generates the picture diary P.
Specifically, the behavior decision unit 236 converts the scene selected in step 2 into text. For example, the text "A teddy bear and the user are at the zoo" is obtained. A fixed sentence, "A picture diary drawn with crayons," is then added to this text and input to the image generation AI to create a picture P1 of the picture diary P. The behavior decision unit 236 also accepts designation of the title and text of the picture diary from the user 10, and obtains a title P2 and text P3 of the picture diary P. The behavior decision unit 236 generates the picture diary P by combining the picture P1 of the picture diary P with the title P2 and text P3 of the picture diary P.

(Step 4) The robot 100 outputs the generated picture diary P to a predetermined person.

In this way, the robot 100 can execute the process of creating the picture diary P based on the history data 222.

The robot 100 may analyze the prompts used when creating the picture diary P and input to the image generation model, extract information about the user 10 including the hobbies of the user 10, and update the history data 222. For example, when creating the picture diary P, text representing the actions of the user 10 stored in the history data 222 is generated as a prompt to be input to the image generation model, and the robot 100 may analyze the text, extract information about the user 10, and update the history data 222. Then, based on the updated information about the user 10, the robot 100 may decide and propose the content to the user 10. Furthermore, the robot 100 may obtain information necessary for the proposal and provide it to the user 10.

For example, if the robot 100 writes in the picture diary P that the user ate strawberries with a smile, the robot 100 stores the user's information that "I like strawberries" in the history data. Then, when strawberry picking season arrives, the robot 100 suggests to the user 10, "Would you like to go strawberry picking?" Furthermore, the robot 100 obtains information from the web and conveys to the user relevant information related to "strawberries," such as the date and price of strawberry picking.

Here, a specific process will be described when the robot 100 analyzes a prompt generated when creating the picture diary P and updates the information including the hobbies of the user 10 in the history data 222. FIG. 9 is a flowchart showing an example of a specific process flow when the robot 100 analyzes a prompt generated when creating the picture diary P and updates the information including the hobbies of the user 10 in the history data 222. Note that the process in FIG. 9 starts when the robot 100 generates a prompt when creating the picture diary P.

In step S200, the action decision unit 236 obtains text representing the action of the user 10 used in creating the picture diary P.

In step S202, the behavior decision unit 236 analyzes the acquired text. For example, the scenes selected for the picture diary P are those in which the emotional values of the user 10 and the robot 100 are strong, and so the user's hobbies and preferences are extracted from the words contained in the text used when creating the picture diary P. For example, words related to hobbies are set in advance, and words corresponding to hobbies are extracted from the text as information containing hobbies.

In step S204, the action decision unit 236 determines whether there is information including the hobbies of the user 10. For example, this determination is made by determining whether words related to hobbies have been extracted as a result of analyzing the text. If the determination is negative, the process ends, and if the determination is positive, the process proceeds to step S206.

In step S206, the behavior decision unit 236 updates the information of the user 10. For example, the history data 222 is updated with the extracted information on the hobbies of the user 10. This updates the information of the user 10, such as hobbies, interests, actions, and lifestyle, making it possible for the behavior control unit 250 to suggest actions that match the user's hobbies, preferences, and trends.

Next, a specific process performed when the robot 100 suggests an action to the user 10 will be described. FIG. 10 is a flowchart showing an example of a specific process flow performed when the robot 100 suggests an action to the user 10. The process in FIG. 10 starts at a predetermined timing, such as the end of a day.

In step S300, the behavior decision unit 236 acquires the history data 222. For example, updated information about the user 10 (information about the hobbies of the user 10) is acquired.

In step S302, the behavior decision unit 236 analyzes the history data 222. For example, words corresponding to hobbies are extracted from the updated information of the user 10, and information related to hobbies is collected. Specifically, from the information of the user 10 in the history data 222 that the user 10 "likes strawberries," relevant information related to "strawberries" (for example, the season, etc.) is collected from a website.

In step S304, the action decision unit 236 judges whether it is time to make a suggestion. This judgement is made by judging whether it is time to make a suggestion regarding information of the user 10 based on the result of collection of related information. For example, if information of the user 10 that the user 10 "likes strawberries" exists in the history data 222, it is judged whether it is "strawberry picking" season based on the related information collected by analyzing the history data. If the judgement is negative, the process ends as it is, and if the judgement is positive, the process proceeds to step S304.

In step S306, the behavior decision unit 236 makes a suggestion to the user 10 based on the collected related information. For example, the behavior decision unit 236 inputs "Make a suggestion about strawberries" into the sentence generation model to create a document making a suggestion about "strawberries." As a result, the behavior control unit 250 controls the speech from the speaker, causing the robot 100 to make a suggestion such as "Would you like to go strawberry picking?".

In step S308, the behavior decision unit 236 collects relevant information necessary for the proposed behavior. For example, it obtains information from the web and collects relevant information such as the strawberry picking date, price, nearby locations, etc.

In step S310, the behavior decision unit 236 presents the collected related information to the user 10. For example, the behavior control unit 250 controls the robot 100 based on the behavior determined by the behavior decision unit 236, causing the robot 100 to say something like, "Starting in April, you can pick strawberries at a nearby strawberry farm for a fee of XX yen." Alternatively, the behavior decision unit 236 may transmit the collected information to a mobile phone of a pre-registered user 10 via the communication processing unit 280.

In this way, by providing web information, highly real-time information can be provided to the user 10.

The user's hobbies are also understood from the prompts created for the picture diary P, but because the prompts are created by the robot 100, the robot can learn about hobbies and special skills that it does not know about.

In addition, matching with other users may be performed using information about user 10, including hobbies, updated by analyzing the prompts created for the picture diary P. For example, by linking with a social networking service (SNS), users with similar hobbies may be matched. In addition to hobbies, users with similar values, such as interests, behavior, and lifestyles, may also be matched based on the history data 222. For example, aggressive people may be matched with each other, or people who like strawberries may be matched with each other by analyzing the prompts.

The scenes that are converted into text as prompts are scenes that evoke positive emotions in the user 10, and therefore the values of the user 10 are extracted. Therefore, by analyzing the prompts created for the picture diary P, the user's values can be extracted.

For example, by analyzing the prompts created for the picture diary P, it can be used for matchmaking or recruitment. It is also possible to display advertisements that match the values of the user 10.

Here, a specific process will be described when the robot 100 matches with other users using information about the user 10, including hobbies, updated by analyzing the prompts created for the picture diary P. FIG. 11 is a flowchart showing an example of a specific process flow when the robot 100 matches with other users using information about the user 10, including hobbies, by analyzing the prompts created for the picture diary P. Note that the process in FIG. 11 may be started, for example, when or after the picture diary P is created. Alternatively, it may be started at a timing preset by the user 10, or at another timing.

In step S400, the behavior decision unit 236 acquires the history data 222. That is, the behavior decision unit 236 acquires information about the user 10 stored in the updated history data by analyzing the prompts created for the picture diary P.

In step S402, the behavior decision unit 236 extracts the values of the user 10 from the information of the user 10. For example, at least one of the values of the user 10, including hobbies, interests, behaviors, and lifestyles, is extracted from the information of the user 10 stored in the history data 222.

In step S404, the behavior decision unit 236 determines whether multiple types of values have been extracted. If the determination is positive, the process proceeds to step S406, and if the determination is negative, the process proceeds to step S408.

In step S406, the behavior decision unit 236 focuses on one value. That is, it focuses on one value from among the values of the multiple types of users 10 that have been extracted.

In step S408, the behavior decision unit 236 searches for SNS users whose values correspond to those of the user 10. For example, the behavior decision unit 236 accesses a website via the communication processing unit 280 and searches for users whose values correspond to those of the user 10 from among pre-registered SNS users.

In step S410, the behavior decision unit 236 determines whether or not the search has been completed for all values of the extracted user 10. If the determination is negative, the process proceeds to step S412, where the process returns to step S408 and repeats the above-mentioned process while focusing on other values. On the other hand, if the determination is positive, the process proceeds to step S414.

In step S414, the behavior decision unit 236 outputs the search results and ends the series of processes. For example, the behavior decision unit 236 may output the search results to a mobile phone or the like of the user 10 that has been registered in advance via the communication processing unit 280. Alternatively, the behavior decision unit 236 may control the robot 100 via the behavior control unit 250 and output the search results by speech.

In this way, the values of user 10 can be extracted by analyzing the prompts for picture diary P. Then, a search is made for SNS users with values that correspond to the extracted values of user 10, making it possible to match aggressive people with each other, or people who like strawberries with each other, etc.

In addition, the emotion determination unit 232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 232 may determine the user's emotion according to an emotion map (see FIG. 5), which is a specific mapping.

5 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. Emotions that represent states and actions arising from a state of mind are arranged on the outer side of the concentric circles. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

(1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions at approximately 100 msec, the frequency of the determination of the reaction action of the robot 100 (e.g., a backchannel) may be set to at least the same timing as the detection frequency of the emotion engine (100 msec), or may be set to an earlier timing. The detection frequency of the emotion engine may be interpreted as the sampling rate.

By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized. The robot 100 performs a reaction (such as a backchannel) according to the directionality and the degree (strength) of the mandala in the emotion map 400. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.

(2) The directionality of emotions and the strength of their intensity may be preset in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

These emotions are distributed in the three o'clock direction of emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness takes precedence over internal sensations, resulting in a sense of calm.

(3) If the robot 100 feels good after being praised, the filler "ah" may be inserted before the line, and if the robot 100 feels hurt after receiving harsh words, the filler "ugh!" may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 400.

(4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression that the person is reacting unconsciously.

When the robot 100 feels a favorable feeling in its situational awareness while also feeling an internal sensation (reaction) of satisfaction, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, acceptance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

Conversely, even when the robot 100 is aware of a situation while experiencing an internal sensation (reaction) of discomfort, the robot 100 may shake its head when it feels disgust, or turn the eye LEDs red and glare at the other person when it feels hatred. These types of emotions are distributed around the 6 o'clock position on the emotion map 400.

(5) The inside of emotion map 400 represents what is going on inside one's mind, while the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).

(6) When listening to someone with a sense of relief, which is distributed around the 3 o'clock area of the emotion map 400, the robot 100 may lightly nod its head and say "hmm," but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.

The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210 and the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 400. In addition, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6. Figure 6 shows an example in which multiple emotions, such as "peace of mind," "calm," and "reassuring," have similar emotion values.

Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on learning data that indicates that when the robot 100 is recognized as being stroked by the user 10 from the output of a touch sensor (not shown), the emotional value becomes "happy" at "3," and that when the robot 100 is recognized as being hit by the user 10 from the output of an acceleration sensor (not shown), the emotional value becomes "anger" at "3." Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6.

The behavior decision unit 236 generates the robot's behavior by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the results into a sentence generation model with a dialogue function.

For example, the behavior determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232, using an emotion table such as that shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to index number "2", the text "very happy state" is obtained. Note that when the emotion of the robot 100 corresponds to multiple index numbers, multiple pieces of text representing the state of the robot 100 are obtained.

In addition, an emotion table like that shown in Table 2 is prepared for the emotions of user 10.

Here, if the user's action is speaking "Create a picture diary," the emotion of the robot 100 is index number "2," and the emotion of the user 10 is index number "3," then the following is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user spoke to the robot, saying, 'Create a picture diary.' How would you respond as the robot?", and the robot's action content is obtained. The action decision unit 236 decides on the robot's action from this action content.

In this way, the robot 100 can change its behavior according to the index number that corresponds to the robot's emotions, so that the user gets the impression that the robot has a heart, and is encouraged to take actions such as talking to the robot.

Furthermore, the behavior decision unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior content corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has a personality, and is encouraged to take actions such as talking to the robot. The history data may also further include the robot's emotions and actions.

5 System, 10, 11, 12 User, 20 Communication network, 100, 101, 102 Robot, 200 Sensor unit, 201 Microphone, 202 Depth sensor, 203 Camera, 204 Distance sensor, 210 Sensor module unit, 211 Voice emotion recognition unit, 212 Speech understanding unit, 213 Facial expression recognition unit, 214 Face recognition unit, 220 Storage unit, 221 Response rules, 222 History data, 230 User state recognition unit, 232 Emotion determination unit, 234 Behavior recognition unit, 236 Behavior determination unit, 238 Memory control unit, 250 Behavior control unit, 252 Control target, 280 Communication processing unit, 300 Server, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphic controller, 1218 Display device, 1220 Input/output controller, 1222
Communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 input/output chip

Claims (5)

A user state recognition unit that recognizes a user state including a user's behavior;
an emotion determining unit for determining an emotion of a user or an emotion of a robot;
a behavior determination unit that determines a behavior of the robot corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having an interaction function that allows a user and a robot to interact with each other;
The behavior determination unit selects a scene that satisfies predetermined selection criteria based on historical data including the user's behavior, inputs text representing the selected scene into an image generation model to generate an image representing the scene, and extracts user information including the user's hobbies based on the text, and determines content to be suggested to the user based on the extracted user information. The behavior control system according to claim 1, wherein the behavior decision unit collects related information related to the extracted user information and decides the behavior to be proposed to the user, including the collected related information. The behavior control system according to claim 1, further comprising a behavior control unit that suggests actions to the user by controlling speech from a speaker provided on the robot based on the behavior determined by the behavior determination unit. The behavior control system according to claim 3, wherein the behavior decision unit stores the extracted user information in the history data, and the behavior control unit suggests a behavior to the user at a timing related to the user information stored in the history data. 2. The behavior control system according to claim 1, wherein the behavior determination unit stores the extracted user information in the history data, and performs matching with other users using the user information stored in the history data.