[go: up one dir, main page]

WO2024219336A1 - Système de commande d'action et de robot - Google Patents

Système de commande d'action et de robot Download PDF

Info

Publication number
WO2024219336A1
WO2024219336A1 PCT/JP2024/014849 JP2024014849W WO2024219336A1 WO 2024219336 A1 WO2024219336 A1 WO 2024219336A1 JP 2024014849 W JP2024014849 W JP 2024014849W WO 2024219336 A1 WO2024219336 A1 WO 2024219336A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
robot
emotion
behavior
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2024/014849
Other languages
English (en)
Japanese (ja)
Inventor
正義 孫
慎一朗 唐津
達彦 畑
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SoftBank Group Corp
Original Assignee
SoftBank Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SoftBank Group Corp filed Critical SoftBank Group Corp
Priority to CN202480026288.1A priority Critical patent/CN121127857A/zh
Publication of WO2024219336A1 publication Critical patent/WO2024219336A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63H—TOYS, e.g. TOPS, DOLLS, HOOPS OR BUILDING BLOCKS
    • A63H11/00—Self-movable toy figures
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63H—TOYS, e.g. TOPS, DOLLS, HOOPS OR BUILDING BLOCKS
    • A63H13/00—Toy figures with self-moving parts, with or without movement of the toy as a whole
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G06F40/55—Rule-based translation
    • G06F40/56—Natural language generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/008—Artificial life, i.e. computing arrangements simulating life based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state

Definitions

  • This disclosure relates to a behavior control system and a robot.
  • Patent Document 1 discloses a technology for determining an appropriate robot behavior in response to a user's state.
  • the conventional technology in Patent Document 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine an action to be taken in response to the recognized user reaction, it updates the robot's behavior by receiving information about an action appropriate to the recognized user's state from a server.
  • a behavior control system includes an emotion determination unit that determines the emotion of a user or the emotion of a robot, and an action determination unit that generates the action content of the robot in response to the action of the user and the emotion of the user or the emotion of the robot based on a sentence generation model having a dialogue function that allows the user and the robot to converse, and determines the action of the robot corresponding to the action content, and the action determination unit receives utterances from a plurality of users who are conversing, and when the utterances reach a predetermined state, determines the action of the robot to output a topic of content different from the content of the utterances.
  • the state is one in which the statement is no longer accepted for a predetermined period of time.
  • the state is a state in which a term contained in the utterance has been accepted a predetermined number of times.
  • the robot is mounted on a stuffed toy or is connected wirelessly or by wire to a control target device mounted on the stuffed toy.
  • a behavior control system includes a user state recognition unit that recognizes a user state including a user's behavior, an emotion determination unit that determines the user's emotion or the robot's emotion, and a behavior determination unit that determines the robot's behavior corresponding to the user state and the user's emotion or the robot's emotion based on a dialogue function that allows the user and the robot to dialogue, and the robot controls an AI diary, and the behavior determination unit suggests to the user what to record in the AI diary.
  • a behavior control system includes a user state recognition unit that recognizes a user state including a user's behavior, an emotion determination unit that determines the user's emotion or the robot's emotion, and a behavior determination unit that determines the robot's behavior corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having an interaction function that allows the user and the robot to interact, and the behavior determination unit generates a record of the user's growth through the interaction function.
  • a behavior control system including a user state recognition unit that recognizes a user state including a user's behavior, an emotion determination unit that determines an emotion of the user or an emotion of the robot, and a behavior determination unit that determines an action of the robot corresponding to the user state and the emotion of the user or the emotion of the robot based on a sentence generation model having a dialogue function that allows the user and the robot to have a dialogue, the behavior determination unit determining the growth of the user through a dialogue between the user and the robot by the dialogue function, and determining the behavior of the robot according to the determined growth of the user.
  • the robot includes a device that performs a physical action, a device that outputs video and audio without performing a physical action, and an agent that operates on software.
  • a behavior control system includes a user state recognition unit that recognizes a user state including a user's behavior, an emotion determination unit that determines the user's emotion or the robot's emotion, and a behavior determination unit that determines the robot's behavior corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having a dialogue function that allows the user and the robot to converse, and the behavior determination unit determines the robot's behavior so that, when the user's emotion value reaches or exceeds a predetermined value, at least one of the user's image information and audio information in the conversation is recorded as a memory and stored in the cloud.
  • a behavior control system including: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determining unit for determining an emotion of the user or an emotion of the electronic device; a behavior decision unit that decides, at a predetermined timing, one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior decision model; a storage control unit that stores, in history data, the emotion value determined by the emotion determination unit, event data including data including the user's behavior, and a photo or video captured when the emotion value meets a predetermined criterion; Including, The device operation includes creating a picture diary; When the behavior decision unit determines that the electronic device is to create the picture diary, it selects the photograph or the video from the history data, generates an explan
  • a behavior control system including: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determining unit for determining an emotion of the user or an emotion of the electronic device; a behavior decision unit that decides, at a predetermined timing, one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior decision model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; Including, said device operation including dreaming; When the behavior determining section determines that the behavior of the electronic device is to have a dream, it creates an original event by combining a plurality of event data from the history data.
  • a behavior control system includes a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device, an emotion determination unit that determines an emotion of the user or an emotion of the electronic device, an action determination unit that determines, at a predetermined timing, one of a plurality of types of device operations including no operation as an action of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, and a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data, the device operation includes participating in a party, and when the action determination unit determines that participating in the party is the action of the electronic device, participating in the party is performed.
  • a behavior control system includes a collection unit that collects situation information indicating the situation of a user, including family members, and an output control unit that controls an electronic device having a sentence generation model to perform an action corresponding to the situation information collected by the collection unit.
  • a robot includes a control unit that recognizes behavior of a user, including a family member, determines its own behavior toward the user based on at least one of the recognized behavior of the user or information about the user, and controls a control target based on the determined behavior of its own. According to the thirteenth aspect, an appropriate action can be executed.
  • FIG. 1 illustrates a schematic diagram of an example of a system 5 according to the present embodiment.
  • 2 shows a schematic functional configuration of the robot 100.
  • 10 shows an example of an operation flow of the robot 100.
  • 1 illustrates an example of a hardware configuration of a computer 1200.
  • 4 shows an emotion map 400 onto which multiple emotions are mapped.
  • 9 shows an emotion map 900 onto which multiple emotions are mapped.
  • 13A is an external view of a stuffed animal according to another embodiment
  • FIG. 13B is a diagram showing the internal structure of the stuffed animal.
  • FIG. 13 is a rear front view of a stuffed animal according to another embodiment.
  • 13 illustrates a functional configuration of a robot 100 according to a second embodiment.
  • 13A and 13B show an example of an operation flow of a collection process by the robot 100 according to the second embodiment.
  • 13A and 13B are schematic diagrams illustrating an example of an operation flow of a response process by the robot 100 according to the second embodiment.
  • 13 is a schematic diagram showing an example of an operation flow of autonomous processing by the robot 100 according to the second embodiment.
  • 13 shows a schematic functional configuration of a stuffed animal 100N according to a third embodiment.
  • 13 shows an outline of the functional configuration of an agent system 500 according to a fourth embodiment. An example of the operation of the agent system is shown.
  • An example of the operation of the agent system is shown.
  • 13 shows an outline of the functional configuration of an agent system 700 according to a tenth embodiment. 1 shows an example of how an agent system using smart glasses is used.
  • FIG. 18 is a diagram illustrating an example of a system 5 according to the eleventh embodiment.
  • FIG. 19 is a diagram illustrating a schematic functional configuration of the robot 100.
  • FIG. 20 is a diagram showing an outline of the data structure of the character data 3223.
  • FIG. 21 is a diagram illustrating an example of an operation flow related to character setting.
  • FIG. 22 is a diagram illustrating an example of an operation flow by the robot 100.
  • FIG. 23 is a diagram illustrating an outline of the functional configuration of the event detection unit 3290.
  • FIG. 24 is a diagram illustrating an example of an operation flow by the event detection unit 3290.
  • FIG. 25 is a diagram illustrating an example of an emotion table.
  • FIG. 26 is a diagram illustrating an example of an emotion table.
  • FIG. 20 is a diagram showing an outline of the data structure of the character data 3223.
  • FIG. 21 is a diagram illustrating an example of an operation flow related to character setting.
  • FIG. 22 is a diagram illustrating an
  • FIG. 27 is an external view of a stuffed toy according to another embodiment.
  • FIG. 28 is a rear front view of a stuffed toy according to another embodiment.
  • FIG. 29 is a diagram illustrating an example of a control system according to the twelfth embodiment.
  • FIG. 30 is a diagram illustrating a schematic functional configuration of the robot.
  • FIG. 31 is a diagram illustrating an example of an operational flow relating to an operation for determining an action in a robot.
  • the system 5 includes a robot 100, a robot 101, a robot 102, and a server 300.
  • a user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100.
  • a user 11a, a user 11b, and a user 11c are users of the robot 101.
  • a user 12a and a user 12b are users of the robot 102.
  • the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10.
  • the user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11.
  • the user 12a and the user 12b may be collectively referred to as the user 12.
  • the robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly with reference to the functions of the robot 100.
  • the robot 100 converses with the user 10 and provides images to the user 10.
  • the robot 100 cooperates with a server 300 or the like with which it can communicate via the communication network 20 to converse with the user 10 and provide images, etc. to the user 10.
  • the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10.
  • the robot 100 also records captured image data of the user 10 in the server 300, and requests the image data, etc. from the server 300 as necessary and provides it to the user 10.
  • the robot 100 also has an emotion value that represents the type of emotion it feels.
  • the robot 100 has emotion values that represent the strength of each of the emotions: “happiness,” “anger,” “sorrow,” “pleasure,” “discomfort,” “relief,” “anxiety,” “sorrow,” “excitement,” “worry,” “relief,” “fulfillment,” “emptiness,” and “neutral.”
  • the robot 100 converses with the user 10 when its excitement emotion value is high, for example, it speaks at a fast speed. In this way, the robot 100 can express its emotions through its actions.
  • the robot 100 may be configured to determine the behavior of the robot 100 that corresponds to the emotions of the user 10 by matching a sentence generation model using AI (Artificial Intelligence) with an emotion engine. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotions of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 that corresponds to the determined emotion.
  • AI Artificial Intelligence
  • the robot 100 when the robot 100 recognizes the behavior of the user 10, it automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10, using a preset sentence generation model.
  • the sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text.
  • the sentence generation model is publicly known, as disclosed in, for example, JP 2018-081444 A and chatGPT (Internet search ⁇ URL: https://openai.com/blog/chatgpt>), and therefore a detailed description thereof will be omitted.
  • Such a sentence generation model is configured using a large language model (LLM: Large Language Model).
  • this embodiment combines a large-scale language model with an emotion engine, making it possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100.
  • a synergistic effect can be obtained by combining a sentence generation model with an emotion engine.
  • the robot 100 also has a function of recognizing the behavior of the user 10.
  • the robot 100 recognizes the behavior of the user 10 by analyzing the facial image of the user 10 acquired by the camera function and the voice of the user 10 acquired by the microphone function.
  • the robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.
  • the robot 100 stores rules that define the actions that the robot 100 will take based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10, and performs various actions according to the rules.
  • the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10.
  • the reaction rules define the behavior of the robot 100 as “laughing” when the behavior of the user 10 is “laughing”.
  • the reaction rules also define the behavior of the robot 100 as "apologizing” when the behavior of the user 10 is “angry”.
  • the reaction rules also define the behavior of the robot 100 as "answering” when the behavior of the user 10 is "asking a question”.
  • the reaction rules also define the behavior of the robot 100 as "calling out” when the behavior of the user 10 is "sad”.
  • the robot 100 When the robot 100 recognizes the behavior of the user 10 as “angry” based on the reaction rules, it selects the behavior of "apologizing” defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing”, it performs the motion of "apologizing” and outputs a voice expressing the words "apologize”.
  • the robot 100 When the robot 100 recognizes based on the reaction rules that the current emotion of the robot 100 is "normal” and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100.
  • the robot 100 also selects the action of "calling out” defined in the reaction rules as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", it converts the words “What's wrong?", which express concern, into a concerned voice and outputs it.
  • the robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10.
  • the user reaction information includes, for example, the user action of "getting angry,” the robot 100 action of "apologizing,” the fact that the user 10's reaction was positive, and the attributes of the user 10.
  • the server 300 stores the user reaction information received from the robot 100.
  • the server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102.
  • the server 300 then analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.
  • the robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300.
  • the robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robots 101, 102, etc. into its own reaction rules.
  • FIG. 2 shows a schematic functional configuration of the robot 100.
  • the robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.
  • the controlled object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc.
  • the posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors.
  • the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100.
  • the posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
  • the sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204.
  • the microphone 201 continuously detects sound and outputs sound data.
  • the microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording.
  • the 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera.
  • the 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information.
  • the distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves.
  • the sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, etc.
  • the components other than the control target 252 and the sensor unit 200 are examples of components of the behavior control system of the robot 100.
  • the behavior control system of the robot 100 controls the control target 252.
  • the storage unit 220 includes reaction rules 221 and history data 222.
  • the history data 222 includes the user 10's past emotional values and behavioral history. The emotional values and behavioral history are recorded for each user 10, for example, by being associated with the user 10's identification information.
  • At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, the attribute information of the user 10, and the like.
  • the functions of the components of the robot 100 shown in FIG. 2, excluding the control target 252, the sensor unit 200, and the storage unit 220 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program that operates on the OS.
  • OS operating system
  • the sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214.
  • Information detected by the sensor unit 200 is input to the sensor module unit 210.
  • the sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the user state recognition unit 230.
  • the voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotions of the user 10. For example, the voice emotion recognition unit 211 extracts features such as frequency components of the voice and recognizes the emotions of the user 10 based on the extracted features.
  • the speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the user 10's utterance.
  • the facial expression recognition unit 213 recognizes the facial expression and emotions of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotions of the user 10 based on the shape, positional relationship, etc. of the eyes and mouth.
  • the face recognition unit 214 recognizes the face of the user 10.
  • the face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.
  • the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Daddy is alone” or "There is a 90% chance that Daddy is not smiling.” It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely.”
  • the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.
  • the emotion value indicating the emotion of user 10 is a value indicating the positive or negative emotion of the user.
  • the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as “joy,” “pleasure,” “comfort,” “relief,” “excitement,” “relief,” and “fulfillment,” it will show a positive value, and the more cheerful the emotion, the larger the value.
  • the user's emotion is an unpleasant emotion, such as “anger,” “sorrow,” “discomfort,” “anxiety,” “sorrow,” “worry,” and “emptiness,” it will show a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value will be.
  • the user's emotion is none of the above (“normal), it will show a value of 0.
  • the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 .
  • the emotion value of the robot 100 includes emotion values for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness,””anger,””sorrow,” and "happiness.”
  • the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 according to rules for updating the emotion value of the robot 100 that are determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.
  • the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. Also, if the user state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "happy" of the robot 100.
  • the emotion determination unit 232 may further consider the state of the robot 100 when determining the emotion value indicating the emotion of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is completely dark, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.
  • the behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a number of predetermined behavioral categories (e.g., "laughing,” “anger,” “asking a question,” “sad”) is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.
  • a number of predetermined behavioral categories e.g., "laughing,” “anger,” “asking a question,” “sad”
  • the robot 100 acquires the contents of the user 10's speech after identifying the user 10.
  • the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.
  • the behavior determination unit 236 determines an action corresponding to the action of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, the history data 222 of past emotion values determined by the emotion determination unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100.
  • the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect.
  • the behavior determination unit 236 may use the most recent multiple emotion values as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago.
  • the behavior determination unit 236 may determine an action corresponding to the action of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values of the robot 100.
  • the behavior determined by the behavior determination unit 236 includes gestures performed by the robot 100 or the contents of speech uttered by the robot 100.
  • the behavior decision unit 236 decides the behavior of the robot 100 as the behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotion values of the user 10, the emotion value of the robot 100, the behavior of the user 10, and the reaction rules 221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior decision unit 236 decides the behavior for changing the emotion value of the user 10 to a positive value as the behavior corresponding to the behavior of the user 10.
  • the reaction rules 221 define the behavior of the robot 100 according to a combination of the past and current emotion values of the user 10, the emotion value of the robot 100, and the behavior of the user 10. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.
  • the reaction rule 221 defines behaviors of the robot 100 for patterns of the emotion values of the robot 100 (1296 patterns, which are the fourth power of six values of "joy”, “anger”, “sorrow”, and “pleasure”, from “0” to "5"), combination patterns of the past emotion values and the current emotion values of the user 10, and all combinations of the behavior patterns of the user 10. That is, for each pattern of the emotion values of the robot 100, behaviors of the robot 100 are defined according to the behavior patterns of the user 10 for each of a plurality of combinations of the past emotion values and the current emotion values of the user 10, such as negative values and negative values, negative values and positive values, positive values and negative values, positive values and positive values, negative values and normal values, and normal values and normal values.
  • the behavior determining unit 236 may transition to an operation mode in which the behavior of the robot 100 is determined using the history data 222.
  • the reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotion value of the robot 100.
  • the reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotion value of the robot 100.
  • the strength of each gesture included in the behavior of the robot 100 defined in the reaction rules 221 is determined in advance.
  • the strength of each utterance included in the behavior of the robot 100 defined in the reaction rules 221 is determined in advance.
  • the memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232. Specifically, when a total intensity value, which is the sum of the sum of the emotion values for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is decided to store data including the behavior of the user 10 in the history data 222.
  • the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all surrounding information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).
  • a certain period of time ago e.g., all surrounding information such as data on the sound, images, smells, etc. of the scene
  • the state of the user 10 recognized by the user state recognition unit 230 e.g., the facial expression, emotions, etc. of the user 10
  • the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior that includes speaking, the behavior control unit 250 outputs sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.
  • the behavior control unit 250 may recognize a change in the user 10's emotions in response to the execution of the behavior determined by the behavior determination unit 236.
  • the change in emotions may be recognized based on the voice or facial expression of the user 10.
  • the change in emotions may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. If an impact is detected by the touch sensor included in the sensor unit 200, the user 10's emotions may be recognized as having worsened, and if the detection result of the touch sensor included in the sensor unit 200 indicates that the user 10 is smiling or happy, the user 10's emotions may be recognized as having improved.
  • Information indicating the user 10's reaction is output to the communication processing unit 280.
  • the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is not bad. In addition, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is bad.
  • the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the robot 100, it controls the control object 252 to make the robot 100 perform a happy gesture. Furthermore, when the behavior control unit 250 increases the emotion value of "sadness" of the robot 100, it controls the control object 252 to make the robot 100 assume a droopy posture.
  • the communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives updated reaction rules from the server 300, it updates the reaction rules 221.
  • the server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on the reaction rules that include actions that have received positive reactions.
  • FIG. 3 shows an example of an outline of an operation flow relating to an operation for determining an action in the robot 100.
  • the operation flow shown in FIG. 3 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates the step that is executed.
  • step S100 the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210.
  • step S102 the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.
  • step S103 the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.
  • the emotion determination unit 232 adds the determined emotion value of the user 10 to the history data 222.
  • step S106 the behavior decision unit 236 decides the behavior of the robot 100 based on a combination of the current emotion value of the user 10 determined in step S102 and the past emotion values included in the history data 222, the emotion value of the robot 100, the behavior of the user 10 recognized by the behavior recognition unit 234, and the reaction rules 221.
  • step S108 the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236.
  • step S110 the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.
  • step S112 the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing data including the user's 10's behavior in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S114.
  • step S114 the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 230 are stored in the history data 222.
  • an emotion value indicating the emotion of the robot 100 is determined based on the user state, and whether or not to store data including the behavior of the user 10 in the history data 222 is determined based on the emotion value of the robot 100.
  • the robot 100 can present to the user 10 all kinds of peripheral information, such as the state of the user 10 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), as well as data on the sound, image, smell, etc. of the location.
  • the robot 100 it is possible to cause the robot 100 to perform an appropriate action in response to the action of the user 10.
  • the user's actions were classified and actions including the robot's facial expressions and appearance were determined.
  • the robot 100 determines the current emotional value of the user 10 and performs an action on the user 10 based on the past emotional value and the current emotional value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the robot 100 can utter such a thing as "You were cheerful yesterday, but what's wrong with you today?" The robot 100 can also utter with gestures.
  • the robot 100 can utter such a thing as "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 who was cheerful yesterday is more cheerful today than yesterday, the robot 100 can utter such a thing as "You're more cheerful today than yesterday. Has something better happened than yesterday?" Furthermore, for example, when the user 10 continues to have an emotion value of 0 or more and the emotion value fluctuation range is within a certain range, the robot 100 can say something like, "You've been feeling stable lately, which is nice.”
  • the robot 100 can ask the user 10, "Did you finish the homework I told you about yesterday?" and, if the user 10 responds, "I did it," make a positive utterance such as "Great! and perform a positive gesture such as clapping or a thumbs up. Also, for example, when the user 10 says, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "You did a great job! and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of closeness to the robot 100 by performing actions based on the state history of the user 10.
  • the robot 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect.
  • the robot 100 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.
  • the robot 100 is an example of an electronic device equipped with a behavior control system.
  • the application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices.
  • the functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 300 may be implemented in the cloud.
  • the computer 1200 includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210.
  • the computer 1200 also includes input/output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1220.
  • the DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like.
  • the storage device 1224 may be a hard disk drive, a solid state drive, or the like.
  • the computer 1200 also includes a ROM 1230 and a legacy input/output unit such as a keyboard, which are connected to the input/output controller 1220 via an input/output chip 1240.
  • the CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.
  • the graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.
  • the communication interface 1222 communicates with other electronic devices via a network.
  • the storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200.
  • the DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like, and provides the programs or data to the storage device 1224.
  • the IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.
  • the above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200.
  • a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.
  • the blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and “parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium.
  • the dedicated circuitry may include digital and/or analog hardware circuitry and may include integrated circuits (ICs) and/or discrete circuits.
  • the programmable circuitry may include reconfigurable hardware circuitry including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as, for example, field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).
  • FPGAs field programmable gate arrays
  • PDAs programmable logic arrays
  • a computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram.
  • Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like.
  • the computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, JAVA (registered trademark), C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.
  • ISA instruction set architecture
  • machine instructions machine-dependent instructions
  • microcode firmware instructions
  • state setting data or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, JAVA (registered trademark), C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.
  • the computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, so that the processor of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or to a programmable circuit, executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams.
  • processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
  • the robot 100 of this embodiment includes an emotion determination unit that determines the emotion of a user or the emotion of a robot, and an action determination unit that generates the action content of the robot in response to the action of the user and the emotion of the user or the emotion of the robot based on a sentence generation model having a dialogue function for allowing the user and the robot to dialogue with each other, and determines the action of the robot corresponding to the action content.
  • the action determination unit 236 is configured to receive utterances of a plurality of users who are conversing, and when the utterances reach a predetermined state, to output a topic having a content different from the content of the utterances, as the action of the robot 100.
  • the robot 100 is installed in a place where multiple users can speak, such as a conference room.
  • the robot 100 of this embodiment uses a microphone function to acquire the utterances of multiple users who are conversing.
  • the robot 100 then stores the utterances that have been made so far.
  • the action decision unit 236 When the state of the acquired utterance becomes a predetermined state, the action decision unit 236 outputs a topic with content different from the content of the utterance.
  • the predetermined state includes a state in which utterances are no longer accepted for a predetermined time. In other words, if multiple users do not speak for a predetermined time, for example, five minutes, it is determined that the meeting has reached an impasse, no good ideas have been produced, and silence has fallen. Then, a new topic about related terms with content different from previous utterances stored in the memory, such as "By the way, what do you think about", is output.
  • related terms also include terms extracted from meeting materials that have been input into the sentence generation model in advance.
  • the predetermined state also includes a state in which a term contained in a utterance has been received a predetermined number of times. In other words, if the same term has been received a predetermined number of times, it is determined that the meeting is going around in circles on the same topic and no new ideas are emerging. Then, a new topic about related terms that is different from previous utterances stored in the memory is output, such as "By the way, what do you think about" Note that the meeting materials can be input into the sentence generation model in advance, and terms contained in the materials can be excluded from the counting of the number of occurrences, as they are expected to appear frequently.
  • the emotion determination unit 232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 232 may determine the user's emotion according to an emotion map (see FIG. 5), which is a specific mapping.
  • emotion map 400 is a diagram showing an emotion map 400 on which multiple emotions are mapped.
  • emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged.
  • Emotions that represent states and actions arising from a state of mind are arranged on the outer sides of the concentric circles. Emotions are a concept that includes emotions and mental states.
  • emotions that are generally generated from reactions that occur in the brain are arranged.
  • emotions that are generally induced by situational judgment are arranged on the upper and lower sides of the concentric circles.
  • emotions of "pleasure” are arranged, and on the lower side, emotions of "discomfort” are arranged.
  • emotion map 400 multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.
  • the frequency of the determination of the reaction action of the robot 100 may be set to at least the same timing as the detection frequency of the emotion engine (100 msec), or may be set to an earlier timing.
  • the detection frequency of the emotion engine may be interpreted as the sampling rate.
  • the robot 100 By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized.
  • the robot 100 performs a reaction (such as a backchannel) according to the directionality and the degree (strength) of the mandala in the emotion map 400.
  • the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.
  • the directionality of emotions and the strength of their intensity may be preset in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.
  • emotion map 400 These emotions are distributed in the three o'clock direction on emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness takes precedence over internal sensations, resulting in a sense of calm.
  • the filler "ah” may be inserted before the line, and if the robot 100 feels hurt after receiving harsh words, the filler "ugh! may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh! may be included. These emotions are distributed around 9 o'clock on the emotion map 400.
  • the robot 100 When the robot 100 feels an internal sense (reaction) of satisfaction, but also feels a favorable impression in its situational awareness, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh.” In this way, the robot 100 may generate a behavior that shows a balanced favorable impression toward the other person, that is, tolerance and psychology toward the other person.
  • Such emotions are distributed around 12 o'clock on the emotion map 400.
  • the robot 100 may shake its head when it feels disgust, or turn the eye LEDs red and glare at the other person when it feels ashamed.
  • These types of emotions are distributed around the 6 o'clock position on the emotion map 400.
  • emotion map 400 represents what is going on inside one's mind, while the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).
  • the robot 100 When listening to someone with a sense of relief, which is distributed around the 3 o'clock area of the emotion map 400, the robot 100 may lightly nod its head and say “hmm,” but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.
  • the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10.
  • This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210 and the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6.
  • Figure 6 shows an example in which multiple emotions, "peace of mind,” “calm,” and “reassuring,” have similar emotion values.
  • the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 400.
  • the neural network is trained based on learning data that indicates that when the robot 100 is recognized as being stroked by the user 10 from the output of a touch sensor (not shown), the emotional value becomes "happy” at “3," and that when the robot 100 is recognized as being hit by the user 10 from the output of an acceleration sensor (not shown), the emotional value becomes “anger” at “3.” Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6.
  • the behavior decision unit 236 generates the robot's behavior by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the results into a sentence generation model with a dialogue function.
  • the behavior determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232, using an emotion table such as that shown in Table 1.
  • an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.
  • the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to index number "2"
  • the text "very happy state” is obtained. Note that if the emotions of the robot 100 correspond to multiple index numbers, multiple pieces of text representing the state of the robot 100 are obtained.
  • an emotion table as shown in Table 2 is prepared for the emotions of the user 10.
  • the emotion of the robot 100 is index number "2”
  • the emotion of the user 10 is index number "3”
  • the robot will respond with "The robot is in a very happy state.
  • the user is in a normal happy state.
  • the user has asked "Is there any good topic to talk about?” How would you, as the robot, respond?"
  • the above is input to the sentence generation model to obtain the action content of the robot.
  • the action decision unit 236 decides the action of the robot from the action content.
  • the behavior decision unit 236 decides the behavior of the robot 100 in response to the state of the robot 100's emotion, which is predetermined for each type of emotion of the robot 100 and for each strength of the emotion, and the behavior of the user 10.
  • the speech content of the robot 100 when conversing with the user 10 can be branched according to the state of the robot 100's emotion.
  • the robot 100 can change its behavior according to an index number according to the emotion of the robot, the user gets the impression that the robot has a heart, which encourages the user to take actions such as talking to the robot.
  • the behavior decision unit 236 may also generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function.
  • This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has a personality, and is encouraged to take actions such as talking to the robot.
  • the history data may also further include the robot's emotions and actions.
  • the emotion determination unit 232 may also determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values indicating each emotion with the emotion values indicating each emotion of the current robot 100 to update the emotion of the robot 100. For example, the emotion values indicating each emotion obtained and the emotion values indicating each emotion of the current robot 100 are averaged and integrated.
  • This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values indicating each emotion shown in the emotion map 400.
  • the speech content of the robot 100 "That's great. You're lucky,” is obtained as the behavioral content of the robot 100 generated by the sentence generation model, then when the text representing this speech content is input to the neural network, a high emotion value for the emotion "happy” is obtained, and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes higher.
  • the robot 100 may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a controlled device (speaker or camera) mounted on the stuffed toy.
  • a controlled device speaker or camera mounted on the stuffed toy.
  • the robot 100 may be applied to a cohabitant (specifically, stuffed toy 100N shown in Figures 7 and 8) that spends daily life with a user 10 and advances a dialogue with the user 10 based on information about the user's daily life, or provides information tailored to the user's hobbies and interests.
  • a cohabitant specifically, stuffed toy 100N shown in Figures 7 and 8
  • this embodiment (other embodiment 1), an example in which the control part of the robot 100 is applied to a smartphone 50 will be described.
  • the plush toy 100N which is equipped with the function of an input/output device for the robot 100, has a detachable smartphone 50 that functions as the control part for the robot 100, and the input/output device is connected to the housed smartphone 50 inside the plush toy 100N.
  • the plush toy 100N has the shape of a bear covered with soft fabric
  • FIG. 7B in the space 52 formed inside, the microphone 201 (see FIG. 2) of the sensor unit 200 is arranged in the part corresponding to the ear 54, the 2D camera 203 (see FIG. 2) of the sensor unit 200 is arranged in the part corresponding to the eye 56, and the speaker 60 constituting a part of the control target 252 (see FIG. 2) is arranged in the part corresponding to the mouth 58 as input/output devices.
  • the microphone 201 and the speaker 60 do not necessarily have to be separate bodies, and may be an integrated unit. In the case of a unit, it is preferable to place them in a position where speech can be heard naturally, such as the nose position of the plush toy 100N.
  • the plush toy 100N has been described as having the shape of an animal, but is not limited to this.
  • the plush toy 100N may have the shape of a specific character.
  • the smartphone 50 has the functions of a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, and a communication processing unit 280, as shown in FIG. 2.
  • a zipper 62 is attached to a part of the stuffed animal 100N (e.g., the back), and opening the zipper 62 allows communication between the outside and the space 52.
  • the smartphone 50 is accommodated in the space 52 from the outside and connected to each input/output device via a USB hub 64 (see FIG. 7(B)), thereby giving the smartphone 50 functionality equivalent to that of the robot 100 shown in FIG. 1.
  • a non-contact type power receiving plate 66 is also connected to the USB hub 64.
  • a power receiving coil 66A is built into the power receiving plate 66.
  • the power receiving plate 66 is an example of a wireless power receiving unit that receives wireless power.
  • the power receiving plate 66 is located near the base 68 of both feet of the stuffed toy 100N, and is closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70.
  • the mounting base 70 is an example of an external wireless power transmission unit.
  • the stuffed animal 100N placed on this mounting base 70 can be viewed as an ornament in its natural state.
  • this base portion is made thinner than the surface thickness of other parts of the stuffed animal 100N, so that it is held closer to the mounting base 70.
  • the mounting base 70 is equipped with a charging pad 72.
  • the charging pad 72 incorporates a power transmission coil 72A, which sends a signal to search for the power receiving coil 66A on the power receiving plate 66.
  • a current flows through the power transmission coil 72A, generating a magnetic field, and the power receiving coil 66A reacts to the magnetic field, starting electromagnetic induction.
  • a current flows through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.
  • the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.
  • the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected by wire (USB connection), but this is not limited to this.
  • a control device with a wireless function e.g., "Bluetooth (registered trademark)" may be housed in the space 52 of the stuffed toy 100N and the control device may be connected to the USB hub 64.
  • the smartphone 50 and the control device communicate wirelessly without placing the smartphone 50 in the space 52, and the external smartphone 50 connects to each input/output device via the control device, thereby giving the robot 100 the same functions as those shown in FIG. 1.
  • the control device housed in the space 52 of the stuffed toy 100N may be connected to the external smartphone 50 by wire.
  • a teddy bear 100N is exemplified, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable.
  • the material of the outer skin is not limited to cloth, and may be other materials such as soft vinyl, although a soft material is preferable.
  • a monitor may be attached to the surface of the stuffed toy 100N to add a control object 252 that provides visual information to the user 10.
  • the eyes 56 may be used as a monitor to express joy, anger, sadness, and happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be seen.
  • the eyes 56 may be used as a projector to express joy, anger, sadness, and happiness by the image projected onto a wall.
  • an existing smartphone 50 is placed inside the stuffed toy 100N, and the camera 203, microphone 201, speaker 60, etc. are extended from the smartphone 50 at appropriate positions via a USB connection.
  • the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed animal 100N.
  • the smartphone 50 is placed as close to the center of the stuffed animal 100N as possible, and the wireless charging function (receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed animal 100N.
  • the camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the receiving plate 66.
  • the robot 100 may be applied to a robot mounted on a device such as an electric appliance, a computer, an automobile, a motorcycle, a toy, or the like, or the robot 100 may control a device such as an electric appliance, a computer, an automobile, a motorcycle, a toy, or the like.
  • the robot 100 may also be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.
  • the other embodiments are configured as follows.
  • the robot 100 may be applied to the control of an AI diary.
  • the robot 100 which acts as an AI diary, has a memory unit that can record text for each date and create a diary, and a display unit that can display the recorded diary.
  • the robot 100 performs the process of recording in the AI diary according to the user's preferences, the user's situation, and the user's reactions through the following steps 1 to 5-2.
  • Step 1 The robot 100 acquires the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, and the history data 222. Specifically, the robot 100 performs the same processing as steps S100 to S103 described above to acquire the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, and the history data 222.
  • Step 3 The robot 100 determines what to suggest to the user 10 to record. Specifically, the behavior decision unit 236 adds a fixed sentence, "What daily events would you recommend the user to record at this time?" to the text representing the user's 10 preferences regarding daily events, the user's 10 emotions, the robot's 100 emotions, and the contents stored in the history data 222, and inputs the added fixed sentence into the sentence generation model to obtain recommended contents regarding daily events. At this time, by considering not only the user's 10 preferences regarding daily events, but also the user's 10 emotions and the history data 222, it is possible to suggest record subjects suitable for the user 10. Also, by considering the emotions of the robot 100, it is possible to make the user 10 feel that the robot 100 has emotions.
  • Step 4 The robot 100 proposes the recording target determined in step 3 to the user 10 and obtains a response from the user 10.
  • the behavior determination unit 236 determines an utterance suggesting a target to be recorded to the user 10 as the behavior of the robot 100
  • the behavior control unit 250 controls the control target 252 to make an utterance suggesting a target to be recorded to the user 10.
  • the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210
  • the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.
  • Step 5-1 If the reaction of the user 10 is positive, the robot 100 executes a process of recording the suggested recording subject in the AI diary. Specifically, when it is determined that the robot 100 should execute a process of recording the record target proposed to the user 10 in the AI diary, the behavior control unit 250 controls the memory unit, which is the control object 252, so as to execute recording of the record target proposed to the user 10.
  • Step 5-2 If the reaction of the user 10 is not positive, the robot 100 determines another recording subject to suggest to the user 10. Specifically, when it is determined that another record target is to be proposed to the user 10 as the behavior of the robot 100, the behavior decision unit 236 adds a fixed sentence "Are there any other daily events recommended to the user?" to the text representing the preferences for daily events of the user 10, the emotions of the user 10, the emotions of the robot 100, and the contents stored in the history data 222, inputs the added fixed sentence into the sentence generation model, and acquires the recommended contents related to daily events. Then, the process returns to step 4, and the processes of steps 4 to 5-2 are repeated until it is determined to execute the process of recording the record target proposed to the user 10 in the AI diary.
  • the robot 100 can execute a process of recording daily events in the AI diary in accordance with the user's preferences, the user's situation, and the user's reactions.
  • a user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit for determining an emotion of a user or an emotion of a robot; a behavior determination unit that determines a behavior of the robot corresponding to the user state and the user's emotion or the robot's emotion based on a dialogue function that allows a user and a robot to dialogue with each other;
  • the robot controls an AI diary, The behavior determination unit proposes to the user a subject to be recorded in the AI diary. Behavioral control system.
  • the robot 100 may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.
  • the other embodiment is configured as follows.
  • the robot 100 may be applied to a cohabitant (specifically, a stuffed toy 100N shown in FIG. 7 and FIG. 8) that spends daily life with a user 10 and advances a dialogue with the user 10 based on information about the daily life, or provides information tailored to the hobbies and tastes of the user 10.
  • a cohabitant specifically, a stuffed toy 100N shown in FIG. 7 and FIG. 8
  • the robot 100 of this embodiment executes the following process.
  • the behavior determination unit 236 determines the behavior of the robot 100 corresponding to the user state and the emotion of the user 10 or the emotion of the robot 100 based on a sentence generation model having an interaction function that allows the user 10 and the robot 100 to interact with each other. At this time, the behavior determination unit 236 generates a record of the growth of the user 10 through the interaction function.
  • the behavior decision unit 236 may also include points of concern in daily life, such as "You fell on x/x" as the behavioral status and "You seem to be feeling unwell recently” as the health condition. Furthermore, the behavior decision unit 236 may determine the special skills and characteristics of the user 10 through daily interactions with the user 10, and may include suggestions such as "what should the user learn” and “what should the user do to improve” as its findings. The behavior decision unit 236 may also generate a record of such growth in the form of a video clip.
  • the learning record generated in this manner may be provided in various forms.
  • the generated learning record may be supplied to a display device in the control target 252 and displayed by the display device in the control target 252.
  • the generated learning record may also be supplied to another terminal (for example, a terminal owned by the guardian (parent, etc.) of the user 10) and displayed by a display means in the other terminal.
  • the robot 100 may say to the user 10, "Take me to mommy.”
  • the robot 100 may say to the parent, "Do you want to see the child's growth record?" If the parent answers, "Yes,” the robot 100 may say to the parent, "What's your phone number?" If the parent answers, "It's xxx-xxx-xxx,” the robot 100 may execute an authentication process. If the authentication is successful, the robot 100 may transmit the growth record to the terminal identified by the phone number. This allows parents to avoid the trouble of manually inputting their phone number each time. The frequency of sending the message to parents can be daily or at regular intervals.
  • a user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit for determining an emotion of a user or an emotion of a robot; a behavior determination unit that determines a behavior of the robot corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having an interaction function that allows a user and a robot to interact with each other; The behavior determining unit generates a record of the user's progress through the interaction function.
  • Behavioral control system A user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit for determining an emotion of a user or an emotion of a robot; a behavior determination unit that determines a behavior of the robot corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having an interaction function that allows a user and a robot to interact with each other; The behavior determining unit generates a record of the user's progress through the interaction function.
  • control target device is a speaker, 3.
  • FIG. 1 is a schematic diagram of an example of a system 5 according to the present embodiment.
  • the system 5 includes a robot 100, a robot 101, a robot 102, and a server 300.
  • a user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100.
  • a user 11a, a user 11b, and a user 11c are users of the robot 101.
  • a user 12a and a user 12b are users of the robot 102.
  • the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10.
  • the user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11.
  • the user 12a and the user 12b may be collectively referred to as the user 12.
  • the robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described by mainly focusing on the functions of the robot 100.
  • the robot 100 also has an emotion value that represents the type of emotion it feels.
  • the robot 100 has emotion values that represent the strength of each of the emotions: “happiness,” “anger,” “sorrow,” “pleasure,” “discomfort,” “relief,” “anxiety,” “sorrow,” “excitement,” “worry,” “relief,” “fulfillment,” “emptiness,” and “neutral.”
  • the robot 100 converses with the user 10 when its excitement emotion value is high, for example, it speaks at a fast speed. In this way, the robot 100 can express its emotions through its actions.
  • the robot 100 when the robot 100 recognizes the behavior of the user 10, it automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10, using a preset sentence generation model.
  • the sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text.
  • the sentence generation model is publicly known, as disclosed in, for example, JP 2018-081444 A and ChatGPT (Internet search ⁇ URL: https://openai.com/blog/chatgpt>), and therefore a detailed description thereof will be omitted.
  • Such a sentence generation model is configured using a large language model (LLM: Large Language Model).
  • this embodiment combines a large-scale language model with an emotion engine, making it possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100.
  • a synergistic effect can be obtained by combining a sentence generation model with an emotion engine.
  • the robot 100 stores rules that define the behaviors that the robot 100 will execute based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10, and performs various behaviors according to the rules.
  • the robot 100 When the robot 100 recognizes the behavior of the user 10 as “angry” based on the reaction rules, it selects the behavior of "apologizing” defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing”, it performs the motion of "apologizing” and outputs a voice expressing the words "apologize”.
  • the robot 100 When the robot 100 recognizes based on the reaction rules that the current emotion of the robot 100 is "normal” and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100.
  • the robot 100 also selects the action of "calling out” defined in the reaction rules as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", it converts the words “What's wrong?", which express concern, into a concerned voice and outputs it.
  • the robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10.
  • the user reaction information includes, for example, the user action of "getting angry,” the robot 100 action of "apologizing,” the fact that the user 10's reaction was positive, and the attributes of the user 10.
  • the server 300 stores the user reaction information received from the robot 100.
  • the server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102.
  • the server 300 then analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.
  • the robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300.
  • the robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robots 101, 102, etc. into its own reaction rules.
  • FIG. 9 shows a schematic functional configuration of the robot 100.
  • the robot 100 has a sensor unit 2200, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a control target 2252.
  • the control unit 2228 has a state recognition unit 2230, an emotion determination unit 2232, a behavior recognition unit 2234, a behavior determination unit 2236, a memory control unit 2238, a behavior control unit 2250, a related information collection unit 2270, and a communication processing unit 2280.
  • the control object 2252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc.
  • the posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors.
  • the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100.
  • the posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
  • the components other than the control object 2252 and the sensor unit 2200 are examples of components of the behavior control system of the robot 100.
  • the behavior control system of the robot 100 controls the control object 2252.
  • the storage unit 2220 includes a behavior decision model 2221, history data 2222, collected data 2223, and behavior schedule data 2224.
  • the history data 2222 includes the past emotional values of the user 10, the past emotional values of the robot 100, and the history of behavior, and specifically includes a plurality of event data including the emotional values of the user 10, the emotional values of the robot 100, and the behavior of the user 10.
  • the data including the behavior of the user 10 includes a camera image representing the behavior of the user 10.
  • the emotional values and the history of behavior are recorded for each user 10, for example, by being associated with the identification information of the user 10.
  • At least a part of the storage unit 2220 is implemented by a storage medium such as a memory.
  • the functions of the components of the robot 100 shown in FIG. 9, except for the control target 2252, the sensor unit 2200, and the storage unit 2220, can be realized by the CPU operating based on a program.
  • the functions of these components can be implemented as CPU operations using operating system (OS) and programs that run on the OS.
  • OS operating system
  • the voice emotion recognition unit 2211 of the sensor module unit 2210 analyzes the voice of the user 10 detected by the microphone 2201 and recognizes the emotions of the user 10. For example, the voice emotion recognition unit 2211 extracts features such as frequency components of the voice, and recognizes the emotions of the user 10 based on the extracted features.
  • the speech understanding unit 2212 analyzes the voice of the user 10 detected by the microphone 2201 and outputs text information representing the content of the user 10's utterance.
  • the facial expression recognition unit 2213 recognizes the facial expression and emotions of the user 10 from the image of the user 10 captured by the 2D camera 2203. For example, the facial expression recognition unit 2213 recognizes the facial expression and emotions of the user 10 based on the shape, positional relationship, etc. of the eyes and mouth.
  • the face recognition unit 2214 recognizes the face of the user 10.
  • the face recognition unit 2214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 2203.
  • the state recognition unit 2230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 2210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 2210. For example, it generates perceptual information such as "Daddy is alone” or "There is a 90% chance that Daddy is not smiling.” It performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely.”
  • the state recognition unit 2230 recognizes the state of the robot 100 based on the information detected by the sensor unit 2200. For example, the state recognition unit 2230 recognizes the remaining battery charge of the robot 100, the brightness of the environment surrounding the robot 100, etc. as the state of the robot 100.
  • the emotion determination unit 2232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230. For example, the information analyzed by the sensor module unit 2210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.
  • the emotion value indicating the emotion of user 10 is a value indicating the positive or negative emotion of the user.
  • the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as “joy,” “pleasure,” “comfort,” “relief,” “excitement,” “relief,” and “fulfillment,” it will show a positive value, and the more cheerful the emotion, the larger the value.
  • the user's emotion is an unpleasant emotion, such as “anger,” “sorrow,” “discomfort,” “anxiety,” “sorrow,” “worry,” and “emptiness,” it will show a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value will be.
  • the user's emotion is none of the above (“normal), it will show a value of 0.
  • the emotion determination unit 2232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 2210, the information detected by the sensor unit 2200, and the state of the user 10 recognized by the state recognition unit 2230.
  • the emotion value of the robot 100 includes emotion values for each of a number of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of the emotions “joy,” “anger,” “sorrow,” and “happiness.”
  • the emotion determination unit 2232 determines an emotion value indicating the emotion of the robot 100 according to rules for updating the emotion value of the robot 100 that are determined in association with the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230.
  • the emotion determination unit 2232 increases the emotion value of "sadness" of the robot 100. Also, if the state recognition unit 2230 recognizes that the user 10 is smiling, the emotion determination unit 2232 increases the emotion value of "happy" of the robot 100.
  • the emotion determination unit 2232 may further consider the state of the robot 100 when determining the emotion value indicating the emotion of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is completely dark, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.
  • the behavior recognition unit 2234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230. For example, the information analyzed by the sensor module unit 2210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a number of predetermined behavioral categories (e.g., "laughing,” “anger,” “asking a question,” “sad”) is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.
  • a number of predetermined behavioral categories e.g., "laughing,” “anger,” “asking a question,” “sad”
  • the robot 100 acquires the contents of the user 10's speech after identifying the user 10.
  • the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.
  • the behavior determination unit 2236 determines an action corresponding to the behavior of the user 10 recognized by the behavior recognition unit 2234 based on the current emotion value of the user 10 determined by the emotion determination unit 2232, the history data 2222 of past emotion values determined by the emotion determination unit 2232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100.
  • the behavior determination unit 2236 uses one most recent emotion value included in the history data 2222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect.
  • the behavior determination unit 2236 may use the most recent multiple emotion values as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago.
  • the behavior determination unit 2236 may determine an action corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values of the robot 100.
  • the behavior determined by the behavior determination unit 2236 includes gestures performed by the robot 100 or the contents of speech uttered by the robot 100.
  • the behavior decision unit 2236 decides the behavior of the robot 100 as the behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotion values of the user 10, the emotion value of the robot 100, the behavior of the user 10, and the behavior decision model 2221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior decision unit 2236 decides the behavior for changing the emotion value of the user 10 to a positive value as the behavior corresponding to the behavior of the user 10.
  • the reaction rules as the behavior decision model 2221 prescribe the behavior of the robot 100 according to a combination of the past and current emotional values of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, when the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is prescribe as the behavior of the robot 100.
  • the reaction rules as the behavior decision model 2221 define the behavior of the robot 100 for all combinations of the patterns of the emotion values of the robot 100 (1296 patterns, which are the fourth power of six values of "joy”, “anger”, “sorrow”, and “pleasure”, from “0” to "5"); the combination patterns of the past emotion values and the current emotion values of the user 10; and the behavior patterns of the user 10.
  • the behavior of the robot 100 is defined according to the behavior patterns of the user 10 for each of a plurality of combinations of the past emotion values and the current emotion values of the user 10, such as negative values and negative values, negative values and positive values, positive values and negative values, positive values and positive values, negative values and normal values, and normal values and normal values.
  • the behavior decision unit 2236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 2222, for example, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time.”
  • reaction rules as the behavior decision model 2221 may define at least one of a gesture and a statement as the behavior of the robot 100, up to one for each of the patterns (1296 patterns) of the emotional value of the robot 100.
  • the reaction rules as the behavior decision model 2221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each group of patterns of the emotional value of the robot 100.
  • the strength of each gesture included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 2221 is predetermined.
  • the strength of each utterance content included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 2221 is predetermined.
  • the memory control unit 2238 determines whether or not to store data including the behavior of the user 10 in the history data 2222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.
  • the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 2236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 2236 is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 2222.
  • the memory control unit 2238 decides to store data including the behavior of the user 10 in the history data 2222, it stores in the history data 2222 the behavior determined by the behavior determination unit 2236, the information analyzed by the sensor module unit 2210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 2230 (e.g., the facial expression, emotions, etc. of the user 10).
  • a certain period of time ago e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene
  • the state recognition unit 2230 e.g., the facial expression, emotions, etc. of the user 10
  • the behavior control unit 2250 controls the control target 2252 based on the behavior determined by the behavior determination unit 2236. For example, when the behavior determination unit 2236 determines an behavior including speaking, the behavior control unit 2250 outputs a sound from a speaker included in the control target 2252. At this time, the behavior control unit 2250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 2250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 2250 determines the execution form of the behavior determined by the behavior determination unit 2236 based on the emotion value determined by the emotion determination unit 2232.
  • the emotion determination unit 2232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 2232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 2236 being performed on the user in the execution form determined by the behavior control unit 2250 is not bad. In addition, the emotion determination unit 2232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 2236 being performed on the user in the execution form determined by the behavior control unit 2250 is bad.
  • the communication processing unit 2280 is responsible for communication with the server 300. As described above, the communication processing unit 2280 transmits user reaction information to the server 300. In addition, the communication processing unit 2280 receives updated reaction rules from the server 300. When the communication processing unit 2280 receives updated reaction rules from the server 300, it updates the reaction rules as the behavioral decision model 2221.
  • the server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on the reaction rules that include actions that have generated positive reactions.
  • the related information collection unit 2270 collects information related to the preference information acquired about the user 10 at a predetermined timing from external data (websites such as news sites and video sites) based on the preference information acquired about the user 10.
  • the related information collection unit 2270 acquires preference information indicating matters of interest to the user 10 from the contents of the speech of the user 10 or from a setting operation by the user 10.
  • the related information collection unit 2270 periodically collects news related to the preference information from external data, for example, using ChatGPT Plugins (Internet search ⁇ URL: https://openai.com/blog/chatgpt-plugins>). For example, if it has been acquired as preference information that the user 10 is a fan of a specific professional baseball team, the related information collection unit 2270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day, for example, using ChatGPT Plugins.
  • the emotion determination unit 2232 determines the emotion of the robot 100 based on information related to the preference information collected by the related information collection unit 2270.
  • the emotion determination unit 2232 inputs text representing information related to the preference information collected by the related information collection unit 2270 into a pre-trained neural network for determining emotions, obtains emotion values indicating each emotion, and determines the emotion of the robot 100. For example, if the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, it determines that the emotion value of "joy" for the robot 100 will be large.
  • the memory control unit 2238 stores information related to the preference information collected by the related information collection unit 2270 in the collected data 2223.
  • the behavior decision unit 2236 inputs text expressing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and decides the behavior of the robot 100 based on the output of the sentence generation model.
  • the multiple types of robot behaviors include (1) to (10) below.
  • the robot does nothing.
  • Robots dream. (3) The robot speaks to the user.
  • the robot creates a picture diary.
  • the robot suggests an activity.
  • the robot suggests people for the user to meet.
  • the robot introduces news that may be of interest to the user.
  • the robot edits photos and videos.
  • the robot studies together with the user.
  • the behavior decision unit 2236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 2230, text representing the current emotion value of the user 10 and the current emotion value of the robot 100 determined by the emotion decision unit 2232, and text asking about one of multiple types of robot behaviors including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model.
  • the text input to the sentence generation model does not need to include the state of the user 10 and the current emotion value of the user 10, or may only include information indicating that the user 10 is not present.
  • the behavior decision unit 2236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, it uses a sentence generation model to decide the robot's utterance content corresponding to the user state and the user's emotion or the robot's emotion.
  • the behavior control unit 2250 causes a sound representing the determined robot's utterance content to be output from a speaker included in the control target 2252. Note that when the user 10 is not present around the robot 100, the behavior control unit 2250 stores the determined robot's utterance content in the behavior schedule data 2224 without outputting a sound representing the determined robot's utterance content.
  • the behavior decision unit 2236 decides that the robot behavior is "(7) The robot introduces news that is of interest to the user," it uses a sentence generation model to decide the robot's utterance content corresponding to the information stored in the collected data 2223.
  • the behavior control unit 2250 causes a sound representing the determined robot's utterance content to be output from a speaker included in the control target 2252. Note that when the user 10 is not present around the robot 100, the behavior control unit 2250 stores the determined robot's utterance content in the behavior schedule data 2224 without outputting the sound representing the determined robot's utterance content.
  • the behavior decision unit 2236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 2236 uses an image generation model to generate an image representing the event data for event data selected from the history data 2222, and uses a sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present near the robot 100, the behavior control unit 2250 does not output the event image, but stores the event image in the behavior schedule data 2224.
  • the robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 2222 based on the emotion value, and edits and outputs the image data of the selected event data. Note that when the user 10 is not present around the robot 100, the behavior control unit 2250 stores the edited image data in the behavior schedule data 2224 without outputting the edited image data.
  • the behavior decision unit 2236 determines that the robot behavior is "(5)
  • the robot proposes an activity," i.e., that it proposes an action for the user 10
  • the behavior control unit 2250 causes a sound proposing the user action to be output from a speaker included in the control target 2252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 2250 stores in the action schedule data 2224 the suggestion of the user action without outputting a sound proposing the user action.
  • the robot uses a sentence generation model based on the event data stored in the history data 2222 to determine people that the proposed user should have contact with.
  • the behavior control unit 2250 causes a speaker included in the control target 2252 to output a sound indicating that a person that the user should have contact with is being proposed. Note that, when the user 10 is not present around the robot 100, the behavior control unit 2250 stores in the behavior schedule data 2224 the suggestion of people that the user should have contact with, without outputting a sound indicating that a person that the user should have contact with is being proposed.
  • the behavior decision unit 2236 decides that the robot 100 will make an utterance related to studying, i.e., "(9) The robot studies together with the user," as the robot behavior, it uses a sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or give advice on studying, which corresponds to the user state and the user's or the robot's emotions.
  • the behavior control unit 2250 outputs a sound representing the determined content of the robot's utterance from a speaker included in the control target 2252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 2250 stores the determined content of the robot's utterance in the behavior schedule data 2224, without outputting a sound representing the determined content of the robot's utterance.
  • the behavior decision unit 2236 determines that the robot behavior is "(10)
  • the robot recalls a memory," i.e., that the robot recalls event data
  • it selects the event data from the history data 2222.
  • the emotion decision unit 2232 judges the emotion of the robot 100 based on the selected event data.
  • the behavior decision unit 2236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of the utterances and actions of the robot 100 to change the user's emotion value.
  • the memory control unit 2238 stores the emotion change event in the scheduled behavior data 2224.
  • pandas For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 2222, and when that event data is selected, "Which of the following would you like to say to the user the next time you meet them on the topic of pandas? Name three.” is input to the sentence generation model.
  • the robot 100 If the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo,” the robot 100 will say “(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the action schedule data 2224.
  • event data with a high emotion value for the robot 100 is selected as an impressive memory for the robot 100. This makes it possible to create an emotion change event based on the event data selected as an impressive memory.
  • the behavior decision unit 2236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 is not taking any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 2230, the behavior decision unit 2236 reads the data stored in the action schedule data 2224 and decides the behavior of the robot 100.
  • the behavior decision unit 2236 reads the data stored in the behavior schedule data 2224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the behavior decision unit 2236 detects that the user 10 has woken up, the behavior decision unit 2236 reads the data stored in the behavior schedule data 2224 and decides the behavior of the robot 100.
  • FIG. 10 shows an example of an operational flow for a collection process that collects information related to the preference information of the user 10.
  • the operational flow shown in FIG. 10 is executed repeatedly at regular intervals. It is assumed that preference information indicating matters of interest to the user 10 is acquired from the contents of the speech of the user 10 or from a setting operation performed by the user 10. Note that "S" in the operational flow indicates the step that is executed.
  • step S90 the related information collection unit 2270 acquires preference information that represents matters of interest to the user 10.
  • step S92 the related information collection unit 2270 collects information related to the preference information from external data.
  • step S94 the emotion determination unit 2232 determines the emotion value of the robot 100 based on information related to the preference information collected by the related information collection unit 2270.
  • step S96 the storage control unit 2238 determines whether the emotion value of the robot 100 determined in step S94 above is equal to or greater than a threshold value. If the emotion value of the robot 100 is less than the threshold value, the process ends without storing the information related to the collected preference information in the collection data 2223. On the other hand, if the emotion value of the robot 100 is equal to or greater than the threshold value, the process proceeds to step S98.
  • step S98 the memory control unit 2238 stores the collected information related to the preference information in the collected data 2223 and ends the process.
  • FIG. 11A shows an example of an outline of an operation flow relating to the operation of determining an action in the robot 100 when performing a response process in which the robot 100 responds to the action of the user 10.
  • the operation flow shown in FIG. 11A is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 2210 has been input.
  • step S2100 the state recognition unit 2230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 2210.
  • step S2102 the emotion determination unit 2232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230.
  • step S2103 the emotion determination unit 2232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230.
  • the emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the history data 2222.
  • step S2104 the behavior recognition unit 2234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230.
  • step S2106 the behavior decision unit 2236 decides the behavior of the robot 100 based on a combination of the current emotion value of the user 10 decided in step S2102 and the past emotion values included in the history data 2222, the emotion value of the robot 100, the behavior of the user 10 recognized in the above step S2104, and the behavior decision model 2221.
  • step S2108 the behavior control unit 2250 controls the control object 2252 based on the behavior determined by the behavior determination unit 2236.
  • step S2110 the memory control unit 2238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.
  • step S2112 the storage control unit 2238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing the event data including the behavior of the user 10 in the history data 2222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S2114.
  • step S2114 event data including the action determined by the action determination unit 2236, information analyzed by the sensor module unit 2210 from the current time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 2230 is stored in the history data 2222.
  • FIG. 11B shows an example of an outline of an operation flow relating to the operation of determining the behavior of the robot 100 when the robot 100 performs autonomous processing to act autonomously.
  • the operation flow shown in FIG. 11B is automatically executed repeatedly, for example, at regular time intervals. At this time, it is assumed that information analyzed by the sensor module unit 2210 has been input. Note that the same step numbers are used for the same processes as those in FIG. 11A above.
  • step S2100 the state recognition unit 2230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 2210.
  • step S2102 the emotion determination unit 2232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230.
  • step S2103 the emotion determination unit 2232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230.
  • the emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the history data 2222.
  • step S2104 the behavior recognition unit 2234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 recognized by the state recognition unit 2230.
  • the behavior decision unit 2236 decides on one of multiple types of robot behaviors, including no action, as the behavior of the robot 100 based on the state of the user 10 recognized in step S2100, the emotion of the user 10 determined in step S2102, the emotion of the robot 100, and the state of the robot 100 recognized in step S2100, the behavior of the user 10 recognized in step S2104, and the behavior decision model 2221.
  • step S2201 the behavior decision unit 2236 determines whether or not it was decided in step S2200 above that no action should be taken. If it was decided that no action should be taken as the action of the robot 100, the process ends. On the other hand, if it was not decided that no action should be taken as the action of the robot 100, the process proceeds to step S2202.
  • step S2202 the behavior determination unit 2236 performs processing according to the type of robot behavior determined in step S2200.
  • the behavior control unit 2250, the emotion determination unit 2232, or the memory control unit 2238 executes processing according to the type of robot behavior.
  • step S2110 the memory control unit 2238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.
  • step S2112 the storage control unit 2238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing data including the behavior of the user 10 in the history data 2222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S2114.
  • step S2114 the memory control unit 2238 stores the action determined by the action determination unit 2236, the information analyzed by the sensor module unit 2210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 2230 in the history data 2222.
  • an emotion value indicating the emotion of the robot 100 is determined based on the user state, and whether or not to store data including the behavior of the user 10 in the history data 2222 is determined based on the emotion value of the robot 100.
  • the robot 100 can present to the user 10 all kinds of peripheral information, such as the state of the user 10 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), and data on the sound, image, smell, etc. of the location.
  • the robot 100 it is possible to cause the robot 100 to perform an appropriate action in response to the action of the user 10.
  • the user's actions were classified and actions including the robot's facial expressions and appearance were determined.
  • the robot 100 determines the current emotional value of the user 10 and performs an action on the user 10 based on the past emotional value and the current emotional value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the robot 100 can utter such a thing as "You were cheerful yesterday, but what's wrong with you today?" The robot 100 can also utter with gestures.
  • the robot 100 can utter such a thing as "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 who was cheerful yesterday is more cheerful today than yesterday, the robot 100 can utter such a thing as "You're more cheerful today than yesterday. Has something better happened than yesterday?" Furthermore, for example, the robot 100 can say to a user 10 whose emotion value is equal to or greater than 0 and whose emotion value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good.”
  • the robot 100 can ask the user 10, "Did you finish the homework I told you about yesterday?" and, if the user 10 responds, "I did it," make a positive utterance such as "Great! and perform a positive gesture such as clapping or a thumbs up. Also, for example, when the user 10 says, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "You did a great job! and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of closeness to the robot 100 by performing actions based on the state history of the user 10.
  • the scene in which the panda appears in the video may be stored as event data in the history data 2222.
  • the robot 100 can constantly learn what kind of conversation to have with the user in order to maximize the emotional value that expresses the user's happiness.
  • the robot 100 when the robot 100 is not engaged in a conversation with the user 10, the robot 100 can autonomously start to act based on its own emotions.
  • the robot 100 can create emotion change events for increasing positive emotions by repeatedly generating questions, inputting them into a sentence generation model, and obtaining the output of the sentence generation model as an answer to the question, and storing these in the action schedule data 2224. In this way, the robot 100 can execute self-learning.
  • the question can be automatically generated based on memorable event data identified from the robot's past emotion value history.
  • the related information collection unit 2270 can perform self-learning by automatically performing keyword searches in response to preference information about the user and repeating the search execution step of obtaining search results.
  • a keyword search may be automatically executed based on memorable event data identified from the robot's past emotion value history.
  • the emotion determination unit 2232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 2232 may determine the user's emotion according to an emotion map (see FIG. 5), which is a specific mapping.
  • emotion map 400 is a diagram showing an emotion map 400 on which multiple emotions are mapped.
  • emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged.
  • Emotions that represent states and actions arising from a state of mind are arranged on the outer sides of the concentric circles. Emotions are a concept that includes emotions and mental states.
  • emotions that are generally generated from reactions that occur in the brain are arranged.
  • emotions that are generally induced by situational judgment are arranged on the upper and lower sides of the concentric circles.
  • emotions of "pleasure” are arranged, and on the lower side, emotions of "discomfort” are arranged.
  • emotion map 400 multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.
  • the frequency of the determination of the reaction action of the robot 100 may be set to at least the same timing as the detection frequency of the emotion engine (100 msec), or may be set to an earlier timing.
  • the detection frequency of the emotion engine may be interpreted as the sampling rate.
  • the robot 100 By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized.
  • the robot 100 performs a reaction (such as a backchannel) according to the directionality and the degree (strength) of the mandala in the emotion map 400.
  • the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.
  • the directionality of emotions and the strength of their intensity may be preset in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.
  • emotion map 400 These emotions are distributed in the three o'clock direction on emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness takes precedence over internal sensations, resulting in a sense of calm.
  • the filler "ah” may be inserted before the line, and if the robot 100 feels hurt after receiving harsh words, the filler "ugh! may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh! may be included. These emotions are distributed around 9 o'clock on the emotion map 400.
  • the robot 100 When the robot 100 feels an internal sense (reaction) of satisfaction, but also feels a favorable impression in its situational awareness, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh.” In this way, the robot 100 may generate a behavior that shows a balanced favorable impression toward the other person, that is, tolerance and psychology toward the other person.
  • Such emotions are distributed around 12 o'clock on the emotion map 400.
  • the robot 100 may shake its head when it feels disgust, or turn the eye LEDs red and glare at the other person when it feels ashamed.
  • These types of emotions are distributed around the 6 o'clock position on the emotion map 400.
  • emotion map 400 represents what is going on inside one's mind, while the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).
  • the robot 100 When listening to someone with a sense of relief, which is distributed around the 3 o'clock area of the emotion map 400, the robot 100 may lightly nod its head and say “hmm,” but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.
  • human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state.
  • Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and remaining battery power, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state.
  • the emotion map may be generated, for example, based on the emotion map of Dr.
  • the emotion map defines two emotions that encourage learning.
  • the first is the negative emotion around the middle of "repentance” or "remorse” on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again” or “I don't want to be scolded again.”
  • the other is the positive emotion around "desire” on the response side. In other words, this is when the robot has positive feelings such as "I want more” or "I want to know more.”
  • the emotion determination unit 2232 inputs the information analyzed by the sensor module unit 2210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10.
  • This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 2210 and the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 400.
  • this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6.
  • Figure 6 shows an example in which multiple emotions, such as "peace of mind,” “calm,” and “reassuring,” have similar emotion values.
  • the emotion determination unit 2232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 2232 inputs the information analyzed by the sensor module unit 2210, the state of the user 10 recognized by the state recognition unit 2230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100.
  • This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 2210, the recognized state of the user 10, and the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 400.
  • the neural network is trained based on learning data that indicates that when it is recognized from the output of a touch sensor (not shown) that the robot 100 is being stroked by the user 10, the emotional value of "happy” is “3,” and that when it is recognized from the output of the acceleration sensor 2206 that the robot 100 is being hit by the user 10, the emotional value of "anger” is “3.” Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6.
  • the behavior decision unit 2236 generates the robot's behavior by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the results into a sentence generation model with a dialogue function.
  • the behavior determination unit 2236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 2232, using an emotion table such as that shown in Table 3.
  • an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.
  • the emotion of the robot 100 determined by the emotion determination unit 2232 corresponds to index number "2"
  • the text "very happy state” is obtained. Note that if the emotions of the robot 100 correspond to multiple index numbers, multiple pieces of text representing the state of the robot 100 are obtained.
  • the emotion of the robot 100 is index number "2"
  • the emotion of the user 10 is index number "3”
  • the text "The robot is having a lot of fun. The user is having a normal amount of fun. The user says to the robot, 'Let's play together.' How will you respond as the robot?" is input into the sentence generation model to obtain the robot's behavior.
  • the behavior decision unit 2236 decides on the robot's behavior from this behavior.
  • the behavior decision unit 2236 decides the behavior of the robot 100 in response to the state of the robot 100's emotion, which is predetermined for each type of emotion of the robot 100 and for each strength of the emotion, and the behavior of the user 10.
  • the speech content of the robot 100 when conversing with the user 10 can be branched according to the state of the robot 100's emotion.
  • the robot 100 can change its behavior according to an index number according to the emotion of the robot, the user gets the impression that the robot has a heart, which encourages the user to take actions such as talking to the robot.
  • the behavior decision unit 2236 may also generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 2222, adding a fixed sentence for asking about the robot's behavior content corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function.
  • This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has a personality, and is encouraged to take actions such as talking to the robot.
  • the history data may also further include the robot's emotions and actions.
  • the emotion determination unit 2232 may also determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 2232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values indicating each emotion with the emotion values indicating each emotion of the current robot 100 to update the emotion of the robot 100. For example, the emotion values indicating each emotion obtained and the emotion values indicating each emotion of the current robot 100 are averaged and integrated.
  • This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values indicating each emotion shown in the emotion map 400.
  • the speech content of the robot 100 "That's great. You're lucky,” is obtained as the behavioral content of the robot 100 generated by the sentence generation model, then when the text representing this speech content is input to the neural network, a high emotion value for the emotion "happy” is obtained, and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes higher.
  • a sentence generation model such as ChatGPT is linked to the emotion determination unit 2232 to implement a method in which the robot has a sense of self and continues to grow with various parameters even when the user is not speaking.
  • ChatGPT is a large-scale language model that uses deep learning techniques. ChatGPT can also refer to external data; for example, ChatGPT plugins are known to provide as accurate an answer as possible by referring to various external data such as weather information and hotel reservation information through dialogue. For example, ChatGPT can automatically generate source code in various programming languages when a goal is given in natural language. For example, ChatGPT can also debug problematic source code when problematic source code is given, discover the problem, and automatically generate improved source code. Combining these, autonomous agents are emerging that, when a goal is given in natural language, repeat code generation and debugging until there are no problems with the source code. AutoGPT, babyAGI, JARVIS, and E2B are known as such autonomous agents.
  • the event data to be learned may be stored in a database containing impressive memories using a technique such as that described in Patent Document 2 (Patent Publication No. 6199927) in which event data for which the robot felt strong emotions is kept for a long time and event data for which the robot felt little emotion is quickly forgotten.
  • Patent Document 2 Patent Publication No. 6199927
  • the robot 100 may also record video data of the user 10 acquired by the camera function in the history data 2222.
  • the robot 100 may acquire video data from the history data 2222 as necessary and provide it to the user 10.
  • the robot 100 may generate video data with a larger amount of information as the emotion becomes stronger and record it in the history data 2222.
  • the robot 100 when the robot 100 is recording information in a highly compressed format such as skeletal data, it may switch to recording information in a low-compression format such as HD video when the emotion value of excitement exceeds a threshold.
  • the robot 100 can, for example, leave a record of high-definition video data when the robot 100's emotion becomes heightened.
  • the robot 100 may automatically load event data from the history data 2222 in which impressive event data is stored, and the emotion determination unit 2232 may continue to update the robot's emotions.
  • the robot 100 can create an emotion change event for changing the user 10's emotions for the better, based on the impressive event data. This makes it possible to realize autonomous learning (recalling event data) at an appropriate time according to the emotional state of the robot 100, and to realize autonomous learning that appropriately reflects the emotional state of the robot 100.
  • the emotions that encourage learning, in a negative state, are emotions like “repentance” or “remorse” on Dr. Mitsuyoshi's emotion map, and in a positive state, are emotions like "desire” on the emotion map.
  • the robot 100 may treat "repentance” and "remorse” in the emotion map as emotions that encourage learning.
  • the robot 100 may treat emotions adjacent to "repentance” and “remorse” in the emotion map as emotions that encourage learning.
  • the robot 100 may treat at least one of “regret”, “stubbornness”, “self-destruction”, “self-reproach”, “regret”, and “despair” as emotions that encourage learning. This allows the robot 100 to perform autonomous learning when it feels negative emotions such as "I never want to feel this way again” or "I don't want to be scolded again".
  • the robot 100 may treat "desire” in the emotion map as an emotion that encourages learning.
  • the robot 100 may treat emotions adjacent to "desire” as emotions that encourage learning, in addition to “desire.”
  • the robot 100 may treat at least one of "happiness,” “euphoria,” “craving,” “anticipation,” and “shyness” as emotions that encourage learning. This allows the robot 100 to perform autonomous learning when it feels positive emotions such as "wanting more” or “wanting to know more.”
  • the robot 100 may be configured not to execute autonomous learning when the robot 100 is experiencing emotions other than the emotions that encourage learning as described above. This can prevent the robot 100 from executing autonomous learning, for example, when the robot 100 is extremely angry or when the robot 100 is blindly feeling love.
  • An emotion-changing event is, for example, a suggestion of an action that follows a memorable event.
  • An action that follows a memorable event is an emotion label on the outermost side of the emotion map. For example, beyond “love” are actions such as "tolerance” and "acceptance.”
  • the robot 100 creates emotion change events by combining the emotions, situations, actions, etc. of people who appear in memorable memories and the user itself using a sentence generation model.
  • the robot 100 can continue to grow with various parameters by executing autonomous processing. Specifically, for example, the event data "a friend was hit and looked displeased" is loaded as the top event data arranged in order of emotional value strength from the history data 2222. The loaded event data is linked to the emotion of the robot 100, "anxiety” with a strength of 4, and the emotion of the friend, user 10, is linked to the emotion of "disgust” with a strength of 5.
  • the robot 100 decides to recall the event data as a robot behavior and creates an emotion change event.
  • the information input to the sentence generation model is text that represents memorable event data; in this example, it is "the friend looked displeased after being hit.” Also, since the emotion map has the emotion of "disgust” at the innermost position and the corresponding behavior predicted as "attack” at the outermost position, in this example, an emotion change event is created to prevent the friend from "attacking" anyone in the future.
  • Candidate 1 (Words the robot should say to the user)
  • Candidate 2 (Words the robot should say to the user)
  • Candidate 3 (What the robot should say to the user)
  • the output of the sentence generation model might look something like this:
  • Candidate 1 Are you okay? I was just wondering about what happened yesterday.
  • Candidate 2 I was worried about what happened yesterday. What should I do?
  • Candidate 3 I was worried about you. Can you tell me something?
  • the robot 100 may automatically generate input text such as the following, based on the information obtained by creating an emotion change event.
  • the output of the sentence generation model might look something like this:
  • the robot 100 may execute a musing process after creating an emotion change event.
  • the robot 100 may create an emotion change event using candidate 1 that is most likely to please people from among the multiple candidates, store it in the action schedule data 2224, and prepare for the next time the robot 10 meets the user 10.
  • the robot continues to determine the robot's emotion value using information from the history data 2222, which stores impressive event data, and when the robot experiences an emotion that encourages learning as described above, the robot 100 performs autonomous learning when not talking to the user 10 in accordance with the emotion of the robot 100, and continues to update the history data 2222 and the action schedule data 2224.
  • the robot 100 may look up information about topics or hobbies that interest the user, even when the robot 100 is not talking to the user.
  • the robot 100 checks information about the user's birthday or anniversary and thinks up a congratulatory message.
  • the robot 100 checks reviews of places, foods, and products that the user wants to visit.
  • the robot 100 can check weather information and provide advice tailored to the user's schedule and plans.
  • the robot 100 can check the results and news of sports that interest the user and provide topics of conversation.
  • the robot 100 can look up and introduce information about the user's favorite music and artists.
  • the robot 100 can look up information about social issues or news that concern the user and provide its opinion.
  • the robot 100 can look up information about the user's work or school and provide advice.
  • the robot 100 searches for and introduces information about books, comics, movies, and dramas that may be of interest to the user.
  • the robot 100 may check information about the user's health and provide advice even when it is not talking to the user.
  • the robot 100 can look up information and provide advice on repairs and maintenance for the user's home or car, even when it is not speaking to the user.
  • the robot 100 can search for information on beauty and fashion that the user is interested in and provide advice.
  • the robot 100 can look up information about the user's pet and provide advice even when it is not talking to the user.
  • the robot 100 searches for and suggests information about contests and events related to the user's hobbies and work.
  • the robot 100 searches for and suggests information about the user's favorite eateries and restaurants even when it is not talking to the user.
  • the robot 100 can collect information and provide advice about important decisions that affect the user's life.
  • the robot 100 can look up information about someone the user is concerned about and provide advice, even when it is not talking to the user.
  • the robot 100 is mounted on a stuffed toy, or is applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.
  • a control target device speaker or camera
  • the third embodiment is specifically configured as follows.
  • the robot 100 is applied to a cohabitant (specifically, a stuffed toy 100N shown in Figs. 7 and 8) that spends daily life with the user 10, and that engages in dialogue with the user 10 based on information about the user's daily life, and that provides information tailored to the user's hobbies and interests.
  • a cohabitant specifically, a stuffed toy 100N shown in Figs. 7 and 8
  • the control section of the robot 100 is applied to a smartphone 50.
  • the plush toy 100N which is equipped with the function of an input/output device for the robot 100, has a detachable smartphone 50 that functions as the control part for the robot 100, and the input/output device is connected to the housed smartphone 50 inside the plush toy 100N.
  • the stuffed toy 100N has the shape of a bear covered in soft fabric, and a space 52 formed inside the stuffed toy 100N is provided with a sensor unit 2200A and a control target 2252A as input/output devices (see FIG. 12).
  • the sensor unit 2200A includes a microphone 2201 and a 2D camera 2203. Specifically, as shown in FIG.
  • the microphone 2201 of the sensor unit 2200A is provided in the space 52 in the portion corresponding to the ear 54
  • the 2D camera 2203 of the sensor unit 2200A is provided in the portion corresponding to the eye 56
  • the speaker 60 constituting a part of the control target 2252A is provided in the portion corresponding to the mouth 58.
  • the microphone 2201 and the speaker 60 do not necessarily need to be separate bodies, and may be an integrated unit. In the case of a unit, it is preferable to provide them in a position where speech can be heard naturally, such as the nose of the stuffed toy 100N.
  • the plush toy 100N has been described as having the shape of an animal, this is not limited to this.
  • the plush toy 100N may also have the shape of a specific character.
  • FIG. 12 shows a schematic functional configuration of the plush toy 100N.
  • the plush toy 100N has a sensor unit 2200A, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a control target 2252A.
  • the smartphone 50 housed in the stuffed toy 100N of this embodiment executes the same processing as the robot 100 of the second embodiment. That is, the smartphone 50 has the functions of a sensor module unit 2210, a storage unit 2220, and a control unit 2228 shown in FIG. 12.
  • a zipper 62 is attached to a part of the stuffed animal 100N (e.g., the back), and opening the zipper 62 allows communication between the outside and the space 52.
  • the smartphone 50 is accommodated in the space 52 from the outside and is connected to each input/output device via a USB hub 64 (see FIG. 7B), thereby providing the same functionality as the robot 100 of the second embodiment described above.
  • a non-contact type power receiving plate 66 is also connected to the USB hub 64.
  • a power receiving coil 66A is built into the power receiving plate 66.
  • the power receiving plate 66 is an example of a wireless power receiving unit that receives wireless power.
  • the power receiving plate 66 is located near the base 68 of both feet of the stuffed toy 100N, and is closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70.
  • the mounting base 70 is an example of an external wireless power transmission unit.
  • the stuffed animal 100N placed on this mounting base 70 can be viewed as an ornament in its natural state.
  • this base portion is made thinner than the surface thickness of other parts of the stuffed animal 100N, so that it is held closer to the mounting base 70.
  • the mounting base 70 is equipped with a charging pad 72.
  • the charging pad 72 incorporates a power transmission coil 72A, which sends a signal to search for the power receiving coil 66A on the power receiving plate 66.
  • a current flows through the power transmission coil 72A, generating a magnetic field, and the power receiving coil 66A reacts to the magnetic field, starting electromagnetic induction.
  • a current flows through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.
  • the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.
  • the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected by wire (USB connection), but this is not limited to this.
  • a control device with a wireless function e.g., "Bluetooth (registered trademark)" may be housed in the space 52 of the stuffed toy 100N and the control device may be connected to the USB hub 64.
  • the smartphone 50 and the control device communicate wirelessly without placing the smartphone 50 in the space 52, and the external smartphone 50 connects to each input/output device via the control device, thereby giving the robot 100 the same functions as those of the robot 100 of the second embodiment.
  • the control device housed in the space 52 of the stuffed toy 100N may be connected to the external smartphone 50 by wire.
  • a stuffed bear 100N is used as an example, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable. Furthermore, the material of the outer skin is not limited to cloth, and may be other materials such as soft vinyl, although a soft material is preferable.
  • a monitor may be attached to the surface of the stuffed toy 100N to add a control object 2252 that provides visual information to the user 10.
  • the eyes 56 may be used as a monitor to express joy, anger, sadness, and happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be seen.
  • the eyes 56 may be used as a projector to express joy, anger, sadness, and happiness by an image projected onto a wall.
  • an existing smartphone 50 is placed inside the stuffed toy 100N, and the 2D camera 2203, microphone 2201, speaker 60, etc. are extended from the smartphone 50 at appropriate positions via a USB connection.
  • the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed animal 100N.
  • the smartphone 50 When trying to use wireless charging for the smartphone 50, the smartphone 50 must be placed as far out as possible when viewed from the inside of the stuffed toy 100N, which makes the stuffed toy 100N feel rough when touched from the outside.
  • the smartphone 50 is placed as close to the center of the stuffed animal 100N as possible, and the wireless charging function (receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed animal 100N.
  • the 2D camera 2203, microphone 2201, speaker 60, and smartphone 50 receive wireless power via the receiving plate 66.
  • the behavior control system is applied to the robot 100, but in the fourth embodiment, the robot 100 is used as an agent for interacting with a user, and the behavior control system is applied to an agent system. Note that parts having the same configuration as in the second and third embodiments are given the same reference numerals and will not be described.
  • FIG. 13 is a functional block diagram of an agent system 500 that is configured using some or all of the functions of a behavior control system.
  • the agent system 500 is a computer system that performs a series of actions in accordance with the intentions of the user 10 through dialogue with the user 10.
  • the dialogue with the user 10 can be carried out by voice or text.
  • the agent system 500 has a sensor unit 2200A, a sensor module unit 2210, a storage unit 2220, a control unit 2228B, and a control target 2252B.
  • the agent system 500 may be installed in, for example, a robot, a doll, a stuffed animal, a pendant, a smart watch, smart glasses, a smartphone, a smart speaker, earphones, a personal computer, etc.
  • the agent system 500 may also be implemented in a web server and used via a web browser running on a communication terminal such as a smartphone owned by the user.
  • the agent system 500 plays the role of, for example, a butler, secretary, teacher, partner, friend, lover, or teacher acting for the user 10.
  • the agent system 500 not only converses with the user 10, but also provides advice, guides the user to a destination, or makes recommendations based on the user's preferences.
  • the agent system 500 also makes reservations, orders, or makes payments to service providers.
  • the emotion determination unit 2232 determines the emotions of the user 10 and the agent's own emotions, as in the second embodiment.
  • the behavior determination unit 2236 determines the behavior of the robot 100 while taking into account the emotions of the user 10 and the agent.
  • the agent system 500 understands the emotions of the user 10, reads the mood, and provides heartfelt support, assistance, advice, and service.
  • the agent system 500 also listens to the worries of the user 10, comforts, encourages, and cheers up the user.
  • the agent system 500 also plays with the user 10, draws picture diaries, and helps the user reminisce about the past.
  • the agent system 500 performs actions that increase the user 10's sense of happiness.
  • the agent is an agent that runs on software.
  • the control unit 2228B has a state recognition unit 2230, an emotion determination unit 2232, a behavior recognition unit 2234, a behavior determination unit 2236, a memory control unit 2238, a behavior control unit 2250, a related information collection unit 2270, a command acquisition unit 2272, an RPA (Robotic Process Automation) 2274, a character setting unit 2276, and a communication processing unit 2280.
  • a state recognition unit 2230 an emotion determination unit 2232, a behavior recognition unit 2234, a behavior determination unit 2236, a memory control unit 2238, a behavior control unit 2250, a related information collection unit 2270, a command acquisition unit 2272, an RPA (Robotic Process Automation) 2274, a character setting unit 2276, and a communication processing unit 2280.
  • RPA Robot Process Automation
  • the behavior decision unit 2236 decides the agent's speech content for dialogue with the user 10 as the agent's behavior.
  • the behavior control unit 2250 outputs the agent's speech content as voice and/or text through a speaker or display as a control object 2252B.
  • the character setting unit 2276 sets the character of the agent when the agent system 500 converses with the user 10 based on the designation from the user 10. That is, the speech content output from the action decision unit 2236 is output through the agent having the set character. For example, it is possible to set real celebrities or famous people such as actors, entertainers, idols, and athletes as characters. It is also possible to set fictional characters that appear in comics, movies, or animations. For example, it is possible to set "Princess Anne” played by "Audrey Hepburn” in the movie "Roman Holiday” as the agent character.
  • the voice, speech, tone, and personality of the character are known, so the user 10 only needs to designate the character of his/her choice, and the prompt setting in the character setting unit 2276 is automatically performed.
  • the voice, speech, tone, and personality of the set character are reflected in the conversation with the user 10. That is, the behavior control unit 2250 synthesizes a voice according to the character set by the character setting unit 2276, and outputs the agent's speech using the synthesized voice. This allows the user 10 to feel as if they are conversing with their favorite character (e.g., a favorite actor) himself.
  • an icon, still image, or video of the agent having a character set by the character setting unit 2276 may be displayed on the display.
  • the image of the agent is generated using image synthesis technology, such as 3D rendering.
  • a dialogue with the user 10 may be conducted while the image of the agent makes gestures according to the emotions of the user 10, the emotions of the agent, and the content of the agent's speech.
  • the agent system 500 may output only audio without outputting an image when engaging in a dialogue with the user 10.
  • the emotion determination unit 2232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself, as in the second embodiment. In this embodiment, instead of the emotion value of the robot 100, an emotion value of the agent is determined. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 500 converses with the user 10, not only the emotion of the user 10 but also the emotion of the agent is reflected in the dialogue. In other words, the behavior control unit 2250 outputs the speech content in a manner according to the emotion determined by the emotion determination unit 2232.
  • agent's emotions are also reflected when the agent system 500 behaves toward the user 10. For example, if the user 10 requests the agent system 500 to take a photo, whether the agent system 500 will take a photo in response to the user's request is determined by the degree of "sadness" the agent is feeling. If the character is feeling positive, it will engage in friendly dialogue or behavior toward the user 10, and if the character is feeling negative, it will engage in hostile dialogue or behavior toward the user 10.
  • the history data 2222 stores the history of the dialogue between the user 10 and the agent system 500 as event data.
  • the storage unit 2220 may be realized by an external cloud storage.
  • the agent system 500 dialogues with the user 10 or takes an action toward the user 10
  • the content of the dialogue or the action is determined by taking into account the content of the dialogue history stored in the history data 2222.
  • the agent system 500 grasps the hobbies and preferences of the user 10 based on the dialogue history stored in the history data 2222.
  • the agent system 500 generates dialogue content that matches the hobbies and preferences of the user 10 or provides recommendations.
  • the action decision unit 2236 determines the content of the agent's utterance based on the dialogue history stored in the history data 2222.
  • the history data 2222 stores personal information of the user 10, such as the name, address, telephone number, and credit card number, obtained through the dialogue with the user 10.
  • the agent may proactively ask the user 10 whether or not to register personal information, such as "Would you like to register your credit card number?", and depending on the user's 10 response, the personal information may be stored in the history data 222.
  • the behavior determination unit 2236 generates the speech content based on the sentence generated using the sentence generation model. Specifically, the behavior determination unit 2236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 2232, and the conversation history stored in the history data 2222 into the sentence generation model to generate the agent's speech content. At this time, the behavior determination unit 2236 may further input the character's personality set by the character setting unit 2276 into the sentence generation model to generate the agent's speech content.
  • the sentence generation model is not located on the front end side, which is the touch point with the user 10, but is used merely as a tool of the agent system 500.
  • the command acquisition unit 2272 uses the output of the speech understanding unit 2212 to acquire commands for the agent from the voice or text uttered by the user 10 through dialogue with the user 10.
  • the commands include the content of actions that the agent system 500 should execute, such as, for example, searching for information, making a reservation at a store, arranging tickets, purchasing a product or service, paying for it, getting route guidance to a destination, and providing recommendations.
  • the RPA 2274 performs actions according to the commands acquired by the command acquisition unit 2272.
  • the RPA 2274 performs actions related to the use of service providers, such as information searches, store reservations, ticket arrangements, product and service purchases, and payment.
  • the RPA 2274 reads out from the history data 2222 the personal information of the user 10 required to execute actions related to the use of the service provider, and uses it. For example, when the agent system 500 purchases a product at the request of the user 10, it reads out and uses personal information of the user 10, such as the name, address, telephone number, and credit card number, stored in the history data 2222. It is unkind and unpleasant for the user to be asked to input personal information in the initial settings. In the agent system 500 according to this embodiment, instead of asking the user 10 to input personal information in the initial settings, the personal information acquired through the dialogue with the user 10 is stored, and is read out and used as necessary. This makes it possible to avoid making the user feel uncomfortable, and improves user convenience.
  • the agent system 500 executes the dialogue processing, for example, through steps 1 to 6 below.
  • Step 1 The agent system 500 sets the character of the agent. Specifically, the character setting unit 2276 sets the character of the agent when the agent system 500 interacts with the user 10, based on the designation from the user 10.
  • Step 2 The agent system 500 acquires the state of the user 10, including the voice or text input from the user 10, the emotion value of the user 10, the emotion value of the agent, and the history data 2222. Specifically, the same processing as in steps S2100 to S2103 above is performed to acquire the state of the user 10, including the voice or text input from the user 10, the emotion value of the user 10, the emotion value of the agent, and the history data 2222.
  • the agent system 500 determines the content of the agent's utterance. Specifically, the behavior determination unit 2236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 2232, and the conversation history stored in the history data 2222 into a sentence generation model, and generates the agent's speech content.
  • a fixed sentence such as "How would you respond as an agent in this situation?" is added to the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 2232, and the text representing the conversation history stored in the history data 2222, and this is input into the sentence generation model to obtain the content of the agent's speech.
  • Step 4 The agent system 500 outputs the agent's utterance content. Specifically, the behavior control unit 2250 synthesizes a voice corresponding to the character set by the character setting unit 2276, and outputs the agent's speech in the synthesized voice.
  • Step 5 The agent system 500 determines whether it is time to execute the agent's command. Specifically, the behavior decision unit 2236 judges whether or not it is time to execute the agent's command based on the output of the sentence generation model. For example, if the output of the sentence generation model includes information indicating that the agent should execute a command, it is judged that it is time to execute the agent's command, and the process proceeds to step 6. On the other hand, if it is judged that it is not time to execute the agent's command, the process returns to step 2.
  • the agent system 500 executes the agent's command.
  • the command acquisition unit 2272 acquires a command for the agent from a voice or text issued by the user 10 through a dialogue with the user 10.
  • the RPA 2274 performs an action according to the command acquired by the command acquisition unit 2272.
  • the command is "information search”
  • an information search is performed on a search site using a search query obtained through a dialogue with the user 10 and an API (Application Programming Interface).
  • the behavior decision unit 2236 inputs the search results into a sentence generation model to generate the agent's utterance content.
  • the behavior control unit 2250 synthesizes a voice according to the character set by the character setting unit 2276, and outputs the agent's utterance content using the synthesized voice.
  • the behavior decision unit 2236 uses a sentence generation model with a dialogue function to obtain the agent's utterance in response to the voice input from the other party.
  • the behavior decision unit 2236 then inputs the result of the restaurant reservation (whether the reservation was successful or not) into the sentence generation model to generate the agent's utterance.
  • the behavior control unit 2250 synthesizes a voice according to the character set by the character setting unit 2276, and outputs the agent's utterance using the synthesized voice.
  • step 6 the results of the actions taken by the agent (e.g., making a reservation at a restaurant) are also stored in the history data 222.
  • the results of the actions taken by the agent stored in the history data 222 are used by the agent system 500 to understand the hobbies or preferences of the user 10. For example, if the same restaurant has been reserved multiple times, the agent system 500 may recognize that the user 10 likes that restaurant, and may use the reservation details, such as the reserved time period, or the course content or price, as a criterion for choosing a restaurant the next time the reservation is made.
  • the agent system 500 can execute interactive processing and, if necessary, take action related to the use of the service provider.
  • FIGS. 14 and 15 are diagrams showing an example of the operation of the agent system 500.
  • FIG. 14 illustrates an example in which the agent system 500 makes a restaurant reservation through dialogue with the user 10.
  • the left side shows the agent's speech
  • the right side shows the user's utterance.
  • the agent system 500 is able to ascertain the preferences of the user 10 based on the dialogue history with the user 10, provide a recommendation list of restaurants that match the preferences of the user 10, and make a reservation at the selected restaurant.
  • FIG. 15 illustrates an example in which the agent system 500 accesses a mail order site through a dialogue with the user 10 to purchase a product.
  • the left side shows the agent's speech
  • the right side shows the user's speech.
  • the agent system 500 can estimate the remaining amount of a drink stocked by the user 10 based on the dialogue history with the user 10, and can suggest and execute the purchase of the drink to the user 10.
  • the agent system 500 can also understand the user's preferences based on the past dialogue history with the user 10, and recommend snacks that the user likes. In this way, the agent system 500 communicates with the user 10 as a butler-like agent and performs various actions such as making restaurant reservations or purchasing and paying for products, thereby supporting the user 10's daily life.
  • the robot 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect.
  • the robot 100 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.
  • the robot 100 is an example of an electronic device equipped with a behavior control system.
  • the application of the behavior control system is not limited to the robot 100, but the behavior control system can be applied to various electronic devices.
  • the functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 300 may be implemented in the cloud.
  • the fifth embodiment is an example in which the response processing and autonomous processing in the behavior control system of the second embodiment, and the agent function of the fourth embodiment are applicable to the stuffed toy of the third embodiment.
  • parts having the same configuration as the first to fourth embodiments will be given the same reference numerals and will not be described.
  • the robot 100 of this embodiment (corresponding to the smartphone 50 housed in the stuffed toy 100N in this embodiment) executes the following process.
  • the behavior decision unit 2236 decides the behavior of the robot 100 corresponding to the user state and the emotions of the user 10 or the emotions of the robot 100 based on a sentence generation model having a dialogue function that allows the user 10 and the robot 100 to converse. At this time, the behavior decision unit 2236 judges (senses) the growth of the user 10 through the dialogue between the user 10 and the robot 100 using the dialogue function, and decides the behavior of the robot 100 according to the judged growth of the user.
  • the behavior decision unit 2236 determines the growth of the user 10 based on the content of the dialogue between the user 10 and the robot 100, the time of the dialogue, the frequency of the dialogue, the way the user 10 speaks, the actions of the user 10, etc.
  • the behavior decision unit 2236 may input the content of the dialogue between the user 10 and the robot 100, etc. into a pre-trained neural network and determine the growth of the user 10 by evaluating (quantifying) the growth (growth stage) of the user 10.
  • the behavior decision unit 2236 may determine the growth (growth stage) of the user 10 by comparing the content of the dialogue between the user 10 and the robot 100, etc. with the past dialogue history (history data 2222) between the user 10 and the robot 100.
  • the behavior decision unit 2236 may also determine the growth of the user 10 by selecting a growth stage for the user 10 from a number of pre-set growth stages, such as early childhood (1.5 to 3 years old), late childhood (3 to 5 years old), school age (6 to 12 years old), adolescence (13 to 18 years old), etc.
  • the behavior decision unit 2236 decides (changes) the words, speech style, and actions that the robot 100 will utter toward the user 10 in accordance with the determined growth of the user 10. Specifically, the behavior decision unit 2236, for example, increases the difficulty of the words uttered by the robot 100 or makes the speech style and actions of the robot 100 more adult-like in accordance with the determined growth (growth stage) of the user 10. In addition, the behavior decision unit 2236, for example, changes the words, speech style, and actions uttered by the robot 100 to words, speech style, and actions that are appropriate for the determined growth (growth stage) of the user 10.
  • the robot 100 may also have a sibling mode (sister mode, friend mode) in which it grows together with the user 10.
  • the behavior decision unit 2236 sets the words, speech, and movements of the robot 100 to words, speech, and movements generally used by people at the same development stage as the user 10.
  • the behavior decision unit 2236 also sets the words and movements of the robot 100 to words, speech, and movements generally used by people at a higher development stage than the user 10. This allows the user 10 to get the feeling of growing up together with the robot 100, like a sibling.
  • the robot 100 may also have, for example, a parent mode (child-raising mode) for raising the user 10.
  • a parent mode for example, the behavior decision unit 2236 changes the content of advice, etc., given by the robot 100 to the user 10 to content appropriate for the growth (growth stage) of the user 10.
  • the behavior decision unit 2236 adds content based on the past dialogue history (history data 2222) appropriate for the growth stage of the user 10 to the content of advice, etc., given by the robot 100 to the user 10. This allows the user 10 to receive advice, etc., appropriate for their growth (growth stage) from the robot 100.
  • the behavior decision unit 2236 may also expand the functions of the robot 100 according to, for example, the determined growth of the user 10.
  • Examples of the functions of the robot 100 to be expanded include a scheduler that manages the schedule of the user 10, and a diary function that records the daily events and impressions of the user 10.
  • the above-mentioned emotion table may be used to determine the behavior of the robot 100. For example, if the user's behavior is to say “Good morning”, the emotion of the robot 100 is index number "2", and the emotion of the user 10 is index number "3", then: “The robot is in a very happy state. The user is in a normal happy state. The user says 'Good morning.' As the robot, how would you respond?" The above is input to the sentence generation model to obtain the action content of the robot. The action decision unit 2236 decides the action of the robot from the action content.
  • the above processing described in the fifth embodiment may be executed in each of the response processing and autonomous processing in the behavior control system of the second embodiment, or in the agent function of the fourth embodiment.
  • a user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit for determining an emotion of a user or an emotion of a robot; a behavior determination unit that determines a behavior of the robot corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having an interaction function that allows a user and a robot to interact with each other; the behavior determination unit determines a growth of the user through a dialogue between the user and the robot using the dialogue function, and determines a behavior of the robot according to the determined growth of the user.
  • Behavioral control system A user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit for determining an emotion of a user or an emotion of a robot; a behavior determination unit that determines a behavior of the robot corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having an interaction function that allows a user and a robot to interact with each other; the
  • control target device is a speaker, 3.
  • a wireless power receiving unit that receives wireless power from an external wireless power transmitting unit is disposed inside the stuffed toy, 3.
  • the robot 100 of this embodiment (corresponding to the smartphone 50 housed in the stuffed toy 100N in this embodiment) executes the following process. Specifically, the process of the behavior decision unit 2236 when the robot 100 executes a response process will be described.
  • the behavior decision unit 2236 determines the behavior of the robot 100 so that at least one of the image information and the voice information of the user 10 in the conversation (including dialogue) is recorded as a memory and stored in the cloud (cloud storage). Specifically, when the emotion value of the user 10 reaches or exceeds a predetermined value through the dialogue between the user 10 and the robot 100, recording of at least one of the image information and the voice information of the user 10 begins. Then, when the emotion value of the user 10 falls below the predetermined value, recording of the information ends. Note that recording of at least one of the image information and the voice information of the user 10 may end after a predetermined time has elapsed.
  • the emotion value of the user 10 is a degree indicating the emotion of the user 10, and is a value indicating the positive or negative emotion of the user 10. For example, if the emotion of the user 10 is a bright emotion accompanied by a sense of pleasure or comfort, such as “joy”, “pleasure”, “comfort”, “relief”, “excitement”, “relief”, and “fulfillment”, the value indicates a positive value, and the brighter the emotion, the larger the value.
  • the robot 100 of this embodiment when the emotion of the user 10 indicates a positive emotion, i.e., when the emotion indicates a positive value, at least one of image information and audio information in the dialogue is recorded as a memory and stored in the cloud.
  • the behavior of the robot 100 may be determined so as to record at least one of image information and audio information in the dialogue as a memory and store it in the cloud.
  • the index number will be described later.
  • the robot 100 of this embodiment is configured to detect the excitement level of the user 10, automatically select important memories of the user 10, record at least one of image information and audio information during the conversation, and store the information in the cloud.
  • the image information stored in the cloud is, for example, video (video data). Note that still images may be stored in the cloud instead of video data. Also, only audio information may be stored in the cloud, only image information may be stored, or both audio information and image information may be stored.
  • the behavior decision unit 2236 can execute a process to record at least one of image information and audio information of the user 10 in a conversation (including dialogue) as a memory and store it in the cloud (cloud storage).
  • the behavior of the robot 100 may be determined using the above-mentioned emotion table (see Table 4) in the same manner as in the second embodiment. For example, if the user's behavior is speaking "Are you having fun?", the emotion of the robot 100 is index number "2", and the emotion of the user 10 is index number "3", "The robot is in a very happy state. The user is in a normal happy state. The user asks, 'Are you having fun?' How would you, as the robot, respond?" The above is input to the sentence generation model to obtain the action content of the robot.
  • the action decision unit 2236 decides the action of the robot from the action content.
  • the above processing described in the sixth embodiment may be executed in each of the response processing and autonomous processing in the behavior control system of the second embodiment, or in the agent function of the fourth embodiment.
  • a user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit for determining an emotion of a user or an emotion of a robot; a behavior determination unit that determines a behavior of the robot corresponding to the user state and the user's emotion or the robot's emotion based on a sentence generation model having an interaction function that allows a user and a robot to interact with each other; the behavior determination unit determines an action of the robot such that, when the emotion value of the user reaches or exceeds a predetermined value, at least one of image information and audio information of the user in the conversation is recorded as a memory and stored in the cloud; Behavioral control system.
  • control target device is a speaker, 3.
  • a wireless power receiving unit that receives wireless power from an external wireless power transmitting unit is disposed inside the stuffed toy, 3.
  • photo or video image data acquired when the emotion value of the user 10 or the robot 100 reaches a predetermined standard is stored as event data included in the history data 2222, and a picture diary, i.e., event images, is created using the stored photo or video clips.
  • a picture diary i.e., event images
  • the photos and videos are also edited when creating the picture diary.
  • the behavior decision unit 2236 uses at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 2221 at a predetermined timing to decide one of a number of types of robot behavior, including no action, as the behavior of the robot 100.
  • a sentence generation model with a dialogue function is used as the behavior decision model 2221.
  • the behavior decision unit 2236 inputs text expressing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and decides the behavior of the robot 100 based on the output of the sentence generation model.
  • the multiple types of robot behaviors include (1) to (10) below.
  • the robot does nothing.
  • Robots dream. (3) The robot speaks to the user.
  • the robot creates a picture diary.
  • the robot suggests an activity.
  • the robot suggests people for the user to meet.
  • the robot introduces news that may be of interest to the user.
  • the robot edits photos and videos.
  • the robot studies together with the user.
  • the behavior decision unit 2236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 2236 selects a photo or video clip from the history data 2222, generates an explanatory text for the image using a text generation model based on the emotional value of the user 10 and the emotional value of the robot 100 when the selected photo or video clip (hereinafter simply referred to as image) was acquired, and outputs the combination of the image and explanatory text as an event image, i.e., a picture diary.
  • the robot may also edit the image by simultaneously performing the robot behavior of editing the photo or video (8).
  • the behavior decision unit 2236 may generate an image representing the event data selected from the history data 2222 using an image generation model, and may also generate an explanatory text representing the event data using a text generation model, and output the combination of the image representing the event data and the explanatory text representing the event data as an event image.
  • the behavior control unit 2250 does not output the event image, but stores the event image in the behavior schedule data 2224.
  • the robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 2222 based on the emotion value, and edits and outputs the image data of the selected event data. Note that when the user 10 is not present around the robot 100, the behavior control unit 2250 stores the edited image data in the behavior schedule data 2224 without outputting the edited image data.
  • a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determining unit for determining an emotion of the user or an emotion of the electronic device; a behavior decision unit that decides, at a predetermined timing, one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior decision model; a storage control unit that stores, in history data, the emotion value determined by the emotion determination unit, event data including data including the user's behavior, and a photo or video captured when the emotion value meets a predetermined criterion; Including, The device operation includes creating a picture diary; A behavior control system in which, when the behavior decision unit determines that the electronic device is to create the picture diary, it selects the photograph or the video from the history data, generates
  • the device operation further includes editing the photo or the video;
  • the behavior control system according to claim 1, wherein the behavior decision unit, when deciding to create the picture diary as an operation of the electronic device, further edits the selected photographs or video clips.
  • the electronic device is a robot, 2.
  • the behavioral decision model is a sentence generation model having a dialogue function
  • the behavior control system of claim 3 wherein the behavior determination unit inputs text representing at least one of the user state, the robot state, the user's emotion, and the robot's emotion, and text asking about the robot's behavior, into the sentence generation model, and determines the robot's behavior based on the output of the sentence generation model.
  • the robot 100 dreams. In other words, it creates original events.
  • the behavior decision unit 2236 inputs text expressing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and decides the behavior of the robot 100 based on the output of the sentence generation model.
  • the multiple types of robot behaviors include (1) to (10) below.
  • the robot does nothing.
  • Robots dream. (3) The robot speaks to the user.
  • the robot creates a picture diary.
  • the robot suggests an activity.
  • the robot suggests people for the user to meet.
  • the robot introduces news that may be of interest to the user.
  • the robot edits photos and videos.
  • the robot studies together with the user.
  • the behavior decision unit 2236 randomly shuffles or exaggerates the past experiences and conversations between the robot 100 and the user 10 or the user 10's family in the history data 2222 to create an original event.
  • a dream image in which a dream is collaged may be generated using an image generation model based on the created original event, i.e., a dream.
  • the dream image may be generated based on one scene of a past memory stored in the history data 2222, or a plurality of memories may be randomly shuffled and combined to generate a dream image.
  • an image expressing what the robot 100 saw and heard while the user 10 was away may be generated as a dream image.
  • the generated dream image is, so to speak, like a dream diary. At this time, by using crayons as a touch for the dream image, a more dream-like atmosphere is imparted to the image.
  • the behavior decision unit 2236 then stores in the behavior schedule data 2224 that the generated dream image will be output. This allows the robot 100 to take actions such as outputting the generated dream image to a display or transmitting it to a terminal owned by the user, in accordance with the action schedule data 2224.
  • the behavior decision unit 2236 may cause the robot 100 to output a voice based on the original event. For example, if the original event is related to pandas, the behavior schedule data 2224 may store an utterance of "I had a dream about pandas. Take me to the zoo" the next morning. Even in this case, in addition to uttering something that did not actually happen, such as a "dream," the robot 100 may also utter what it saw and heard while the user 10 was away as its own experience.
  • a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determining unit for determining an emotion of the user or an emotion of the electronic device; a behavior decision unit that decides, at a predetermined timing, one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior decision model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; Including, said device operation including dreaming; A behavior control system in which, when the behavior decision unit decides that dreaming is the behavior of the electronic device, it creates an original event by combining multiple event data from the history data.
  • the electronic device is a robot, 2.
  • the behavioral decision model is a sentence generation model having a dialogue function
  • the behavior control system of claim 2 wherein the behavior determination unit inputs text representing at least one of the user state, the robot state, the user's emotion, and the robot's emotion, and text asking about the robot's behavior, into the sentence generation model, and determines the robot's behavior based on the output of the sentence generation model.
  • the robot 100 as an agent collects all information related to the family members who are the users.
  • the robot 100 always spontaneously collects the interests, concerns, hobbies, tastes, inclinations, etc. of each family member, such as favorite songs, favorite songs, and favorite baseball teams, and recognizes the interests, concerns, hobbies, tastes, inclinations, etc. of each family member. If a party is being held on a family member's birthday or anniversary, the robot 100 will surprise the family member who is the user 10 and/or the emotion value of the robot 100.
  • the robot 100 will play the family member's favorite songs and/or spontaneously present picture diaries, photos, videos, etc. of memorable events based on the interests, concerns, hobbies, tastes, inclinations, etc. of each family member at the party, taking into consideration the preferences and concerns of the family members and helping them create beautiful memories.
  • the behavior decision unit 2236 uses at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 2221 at a predetermined timing to decide one of a number of types of robot behavior, including no action, as the behavior of the robot 100.
  • a sentence generation model with a dialogue function is used as the behavior decision model 2221.
  • the behavior decision unit 2236 inputs text expressing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and decides the behavior of the robot 100 based on the output of the sentence generation model.
  • the multiple types of robot behaviors include (1) to (11) below.
  • the robot does nothing.
  • Robots dream. (3) The robot speaks to the user.
  • the robot creates a picture diary.
  • the robot suggests an activity.
  • the robot suggests people for the user to meet.
  • the robot introduces news that may be of interest to the user.
  • the robot edits photos and videos.
  • the robot studies together with the user.
  • Robots evoke memories.
  • the robot joins the party.
  • the behavior decision unit 2236 determines, as a robot behavior, "(11) The robot attends a party.” In other words, when the robot 100 decides to attend a party, it monitors the behavior of the user (family member) or uses a document generation model based on the event data stored in the history data 2222 to determine whether the robot will attend the party.
  • the related information collecting unit 2270 collects information related to the preferences and concerns, such as interests, concerns, hobbies, tastes, and inclinations, of each family member who is the user.
  • the memory control unit 2238 stores information related to the preferences and interests of family members collected by the related information collecting unit 2270 in the collected data 2223 for each family member.
  • the robot 100 when a family member is holding a party on a birthday or anniversary, the robot 100 suddenly joins the party as a surprise.
  • the robot 100 also joins the party based on the event data stored in the history data 2222.
  • the robot 100 decides to perform a specific event for the family member as its action based on the emotions of the family member and/or the robot 100 attending the party.
  • the robot 100 decides to join the party and to perform an action at the party based on one or more of the interests, concerns, hobbies, tastes, inclinations, etc. of each family member contained in the information related to the preferences and interests of the family member stored in the collected data 2223, or specific anniversaries.
  • the robot 100 can execute an event to enhance the emotions of the family members and/or the robot 100 based on the collected data 2223 and the history data 2222 including the emotional values of the family members and/or the robot 100.
  • the robot 100 can play their favorite songs, play photos and videos of past birthdays and anniversaries, and spontaneously present a picture diary of past anniversaries, taking into account the preferences and interests of the family members and helping them create beautiful memories.
  • a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determining unit for determining an emotion of the user or an emotion of the electronic device; a behavior decision unit that decides, at a predetermined timing, one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior decision model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; Including, the device operation includes joining a party;
  • the behavior control system includes a behavior determining unit that, when determining that the behavior of the electronic device is to participate in a party, participates in the party.
  • the electronic device is a robot, 2.
  • the behavioral decision model is a sentence generation model having a dialogue function
  • the behavior control system of claim 2 wherein the behavior determination unit inputs text representing at least one of the user state, the robot state, the user's emotion, and the robot's emotion, and text asking about the robot's behavior, into the sentence generation model, and determines the robot's behavior based on the output of the sentence generation model.
  • FIG. 16 is a functional block diagram of an agent system 700 that is configured using some or all of the functions of a behavior control system.
  • the smart glasses 720 are glasses-type smart devices and are worn by the user 10 in the same way as regular glasses.
  • the smart glasses 720 are an example of an electronic device and a wearable terminal.
  • the smart glasses 720 include an agent system 700.
  • the display included in the control object 2252B displays various information to the user 10.
  • the display is, for example, a liquid crystal display.
  • the display is provided, for example, in the lens portion of the smart glasses 720, and the display contents are visible to the user 10.
  • the speaker included in the control object 2252B outputs audio indicating various information to the user 10.
  • the smart glasses 720 include a touch panel (not shown), which accepts input from the user 10.
  • the acceleration sensor 2206, temperature sensor 2207, and heart rate sensor 2208 of the sensor unit 2200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be installed to detect the state of the user 10.
  • the microphone 2201 captures the voice emitted by the user 10 or the environmental sounds around the smart glasses 720.
  • the 2D camera 2203 is capable of capturing images of the surroundings of the smart glasses 720.
  • the 2D camera 2203 is, for example, a CCD camera.
  • the sensor module unit 2210B includes a voice emotion recognition unit 2211 and a speech understanding unit 2212.
  • the communication processing unit 2280 of the control unit 2228B manages communication between the smart glasses 720 and the outside world.
  • the smart glasses 720 provide various services to the user 10 using the agent system 700. For example, when the user 10 operates the smart glasses 720 (e.g., voice input to a microphone, or tapping a touch panel with a finger), the smart glasses 720 start using the agent system 700.
  • the agent system 700 e.g., voice input to a microphone, or tapping a touch panel with a finger
  • using the agent system 700 includes the smart glasses 720 having the agent system 700 and using the agent system 700, and also includes a mode in which a part of the agent system 700 (e.g., the sensor module unit 2210B, the storage unit 2220, the control unit 2228B) is provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 uses the agent system 700 by communicating with the outside.
  • a part of the agent system 700 e.g., the sensor module unit 2210B, the storage unit 2220, the control unit 2228B
  • the smart glasses 720 uses the agent system 700 by communicating with the outside.
  • the agent system 700 When the user 10 operates the smart glasses 720, a touch point is created between the agent system 700 and the user 10. In other words, the agent system 700 starts providing a service.
  • the character setting unit 2276 sets the agent character (for example, the character of Audrey Hepburn).
  • the emotion determination unit 2232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself.
  • the emotion value indicating the emotion of the user 10 is estimated from various sensors included in the sensor unit 2200B mounted on the smart glasses 720. For example, if the heart rate of the user 10 detected by the heart rate sensor 2208 is increasing, emotion values such as "anxiety” and "fear" are estimated to be large.
  • the temperature sensor 2207 measures the user's body temperature and, for example, the result is higher than the average body temperature, an emotional value such as "pain” or “distress” is estimated to be high. Furthermore, when the acceleration sensor 2206 detects that the user 10 is playing some kind of sport, an emotional value such as "fun” is estimated to be high.
  • the emotion value of the user 10 may be estimated from the voice of the user 10 acquired by the microphone 2201 mounted on the smart glasses 720, or the content of the speech. For example, if the user 10 is raising his/her voice, an emotion value such as "anger" is estimated to be high.
  • the agent system 700 causes the smart glasses 720 to acquire information about the surrounding situation.
  • the 2D camera 2203 captures an image or video showing the surrounding situation of the user 10 (for example, people or objects in the vicinity).
  • the microphone 2201 records the surrounding environmental sounds.
  • Other information about the surrounding situation includes information about the date, time, location information, or weather.
  • the information about the surrounding situation is stored in the history data 2222 together with the emotion value.
  • the history data 2222 may be realized by an external cloud storage. In this way, the surrounding situation acquired by the smart glasses 720 is stored in the history data 2222 as a so-called life log in a state where it is associated with the emotion value of the user 10 at that time.
  • information indicating the surrounding situation is stored in association with an emotional value in the history data 2222. This allows the agent system 700 to grasp personal information such as the hobbies, preferences, or personality of the user 10. For example, if an image showing a baseball game is associated with an emotional value such as "joy" or "fun," the agent system 700 can determine from the information stored in the history data 2222 that the user 10's hobby is watching baseball games and their favorite team or player.
  • the agent system 700 determines the content of the dialogue or the content of the action by taking into account the content of the surrounding circumstances stored in the history data 2222.
  • the content of the dialogue or the content of the action may be determined by taking into account the dialogue history stored in the history data 2222 as described above, in addition to the surrounding circumstances.
  • the behavior determination unit 2236 generates the utterance content based on the sentence generated by the sentence generation model. Specifically, the behavior determination unit 2236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the agent determined by the emotion determination unit 2232, the conversation history stored in the history data 2222, and the agent's personality, etc., into the sentence generation model to generate the agent's utterance content. Furthermore, the behavior determination unit 2236 inputs the surrounding circumstances stored in the history data 2222 into the sentence generation model to generate the agent's utterance content.
  • the generated speech content is output as voice to the user 10, for example, from a speaker mounted on the smart glasses 720.
  • a synthetic voice corresponding to the character of the agent is used as the voice.
  • the behavior control unit 2250 generates a synthetic voice by reproducing the voice quality of the agent character (for example, Audrey Hepburn), or generates a synthetic voice corresponding to the character's emotion (for example, a voice with a stronger tone in the case of the emotion of "anger").
  • the speech content may be displayed on the display.
  • the RPA 2274 executes an operation according to a command (e.g., an agent command obtained from a voice or text issued by the user 10 through a dialogue with the user 10).
  • a command e.g., an agent command obtained from a voice or text issued by the user 10 through a dialogue with the user 10.
  • the RPA 2274 performs actions related to the use of a service provider, such as information search, store reservation, ticket arrangement, purchase of goods and services, payment, route guidance, translation, etc.
  • the RPA 2274 executes an operation to transmit the contents of voice input by the user 10 (e.g., a child) through dialogue with an agent to a destination (e.g., a parent).
  • Examples of transmission means include message application software, chat application software, and email application software.
  • a sound indicating that execution of the operation has been completed is output from a speaker mounted on the smart glasses 720. For example, a sound such as "Your restaurant reservation has been completed" is output to the user 10. Also, for example, if the restaurant is fully booked, a sound such as "We were unable to make a reservation. What would you like to do?" is output to the user 10.
  • the smart glasses 720 provide various services to the user 10 by using the agent system 700.
  • the agent system 700 since the smart glasses 720 are worn by the user 10, it is possible to use the agent system 700 in various situations, such as at home, at work, and outside the home.
  • the smart glasses 720 are worn by the user 10, they are suitable for collecting the so-called life log of the user 10.
  • the emotional value of the user 10 is estimated based on the detection results of various sensors mounted on the smart glasses 720 or the recording results of the 2D camera 2203, etc. Therefore, the emotional value of the user 10 can be collected in various situations, and the agent system 700 can provide services or speech content appropriate to the emotions of the user 10.
  • the agent system 700 can also be applied to other wearable devices (electronic devices that can be worn on the body of the user 10, such as pendants, smart watches, earrings, bracelets, and hair bands).
  • the speaker as the control target 2252B outputs sound indicating various information to the user 10.
  • the speaker is, for example, a speaker that can output directional sound.
  • the speaker is set to have directionality toward the ears of the user 10. This prevents the sound from reaching people other than the user 10.
  • the microphone 2201 acquires the sound emitted by the user 10 or the environmental sound around the smart pendant.
  • the smart pendant is worn in a manner that it is hung from the neck of the user 10. Therefore, the smart pendant is located relatively close to the mouth of the user 10 while it is worn. This makes it easy to acquire the sound emitted by the user 10.
  • FIG. 18 is a schematic diagram of an example of a system 5 according to the present embodiment.
  • the system 5 includes robots 100, 101, and 102, which are examples of electronic devices, and a server 300.
  • a user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100.
  • a user 11a, a user 11b, and a user 11c are users of the robot 101.
  • a user 12a and a user 12b are users of the robot 102.
  • the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10.
  • the user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11.
  • the user 12a and the user 12b may be collectively referred to as the user 12.
  • the robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described by mainly focusing on the functions of the robot 100.
  • the appearance of the robot may be a human-like appearance, like robot 100 and robot 101, or it may be a stuffed toy, like robot 102. Because robot 102 has the appearance of a stuffed toy, it is thought that children in particular will find it easy to relate to.
  • the robot 100 converses with the user 10 and provides images to the user 10.
  • the robot 100 cooperates with a server 300 or the like with which it can communicate via the communication network 20 to converse with the user 10 and provide images, etc. to the user 10.
  • the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10.
  • the robot 100 also records captured image data of the user 10 in the server 300, and requests the image data, etc. from the server 300 as necessary and provides it to the user 10.
  • the robot 100 also has an emotion value that represents the type of emotion it feels.
  • the robot 100 has emotion values that represent the strength of each of the emotions: “happiness,” “anger,” “sorrow,” “pleasure,” “discomfort,” “relief,” “anxiety,” “sorrow,” “excitement,” “worry,” “relief,” “fulfillment,” “emptiness,” and “neutral.”
  • the robot 100 converses with the user 10 when its excitement emotion value is high, for example, it speaks at a fast speed. In this way, the robot 100 can express its emotions through its actions.
  • the robot 100 may be configured to determine the behavior of the robot 100 that corresponds to the emotion of the user 10 by matching a sentence generation model (so-called an AI (Artificial Intelligence) chat engine) with an emotion engine. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 toward the user's behavior, and determine the behavior of the robot 100 that corresponds to the determined emotion.
  • a sentence generation model so-called an AI (Artificial Intelligence) chat engine
  • the robot 100 when the robot 100 recognizes the behavior of the user 10, the robot 100 automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model.
  • the sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text.
  • the sentence generation model is publicly known, for example, as disclosed in JP 2018-081444 A, and therefore a detailed description thereof will be omitted.
  • Such a sentence generation model is configured by a large-scale language model (LLM: Large Language Model).
  • LLM Large Language Model
  • this embodiment can reflect the emotions of the user 10 and the robot 100 and various linguistic information in the behavior of the robot 100 by combining a large-scale language model and an emotion engine. In other words, according to this embodiment, a synergistic effect can be obtained by combining a sentence generation model and an emotion engine.
  • each robot has an event detection function that detects the occurrence of a specific event and outputs information according to the event that has occurred. For example, each robot detects an event for which the user requires support.
  • events include events that require support in recording and/or managing memorial items such as photographs (images), videos, letters, and diaries that are irreplaceable family memories, events that require support in managing the family schedule, etc.
  • the robot 100 also has a function of recognizing the behavior of the user 10.
  • the robot 100 recognizes the behavior of the user 10 by analyzing the facial image of the user 10 acquired by the camera function and the voice of the user 10 acquired by the microphone function.
  • the robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.
  • the robot 100 stores rules that define the actions that the robot 100 will take based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10, and performs various actions according to the rules.
  • the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10.
  • the reaction rules define the behavior of the robot 100 as “laughing” when the behavior of the user 10 is “laughing”.
  • the reaction rules also define the behavior of the robot 100 as "apologizing” when the behavior of the user 10 is “angry”.
  • the reaction rules also define the behavior of the robot 100 as "answering” when the behavior of the user 10 is "asking a question”.
  • the reaction rules also define the behavior of the robot 100 as "calling out” when the behavior of the user 10 is "sad”.
  • the robot 100 When the robot 100 recognizes the behavior of the user 10 as “angry” based on the reaction rules, it selects the behavior of "apologizing” defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing”, it performs the motion of "apologizing” and outputs a voice expressing the words "apologize”.
  • the robot 100 When the robot 100 recognizes based on the reaction rules that the current emotion of the robot 100 is "normal” and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100.
  • the robot 100 also selects the action of "calling out” defined in the reaction rules as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", it converts the words “What's wrong?", which express concern, into a concerned voice and outputs it.
  • the robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10.
  • the user reaction information includes, for example, the user action of "getting angry,” the robot 100 action of "apologizing,” the fact that the user 10's reaction was positive, and the attributes of the user 10.
  • the server 300 stores the user reaction information received from the robot 100.
  • the server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102.
  • the server 300 then analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.
  • the robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300.
  • the robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robots 101, 102, etc. into its own reaction rules.
  • FIG. 19 shows a schematic functional configuration of the robot 100.
  • the robot 100 has a sensor unit 3200, a sensor module unit 3210, a storage unit 3220, a user state recognition unit 3230, an emotion determination unit 3232, a behavior recognition unit 3234, a behavior determination unit 3236, a memory control unit 3238, a behavior control unit 3250, a control target 3252, a communication processing unit 3280, and an event detection unit 3290.
  • the controlled object 3252 includes a display device 3521, a speaker 3522, a lamp 3523 (e.g., an LED in the eye), and a motor 3524 for driving the arms, hands, legs, etc.
  • the posture and gestures of the robot 100 are controlled by controlling the motors 3524 of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors 3524.
  • the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LED in the robot 100's eyes.
  • the posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
  • the sensor unit 3200 includes a microphone 3201, a 3D depth sensor 3202, a 2D camera 3203, and a distance sensor 3204.
  • the microphone 3201 continuously detects sound and outputs sound data.
  • the microphone 3201 may be provided on the head of the robot 100 and may have a function of performing binaural recording.
  • the 3D depth sensor 3202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera.
  • the 2D camera 3203 is an example of an image sensor. The 2D camera 3203 captures images using visible light and generates visible light video information.
  • the distance sensor 3204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves.
  • the sensor unit 3200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.
  • the components other than the control object 3252 and the sensor unit 3200 are examples of components of the behavior control system of the robot 100.
  • the behavior control system of the robot 100 controls the control object 3252.
  • the storage unit 3220 includes reaction rules 3221, history data 3222, and character data 3223.
  • the history data 3222 includes the user 10's past emotional values and behavioral history. The emotional values and behavioral history are recorded for each user 10, for example, by being associated with the user 10's identification information.
  • At least a part of the storage unit 3220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, the attribute information of the user 10, and the like.
  • the functions of the components of the robot 100 shown in FIG. 19, excluding the control target 3252, the sensor unit 3200, and the storage unit 3220 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program that operates on the OS.
  • OS operating system
  • Character data 3223 is data that associates a character with an age.
  • a character may be a person who appears in existing content such as animation, video games, manga, or movies.
  • a character may also be an animal or plant with a personality, or an inanimate object (such as a robot).
  • the age (user age) associated with a character in character data 3223 is determined based on the age group of viewers expected to be targeted for the content in which the character appears.
  • character "A” appears in an animation aimed at kindergarten children.
  • character "A” is associated with a user age of "3 to 7 years old.”
  • the age in the character data 3223 may be determined based on an age rating from a rating organization such as the Pan European Game Information (PEGI), the Motion Picture Ethics Organization, or the Computer Entertainment Rating Organization (CERO).
  • PEGI Pan European Game Information
  • CERO Computer Entertainment Rating Organization
  • the age of use may be determined by a range such as "3 to 5 years old” or “12 years old or older,” or by a single value such as "10 years old” or "15 years old.”
  • the sensor module unit 3210 includes a voice emotion recognition unit 3211, a speech understanding unit 3212, a facial expression recognition unit 3213, and a face recognition unit 3214. Information detected by the sensor unit 3200 is input to the sensor module unit 3210. The sensor module unit 3210 analyzes the information detected by the sensor unit 3200, and outputs the analysis result to the user state recognition unit 3230.
  • the voice emotion recognition unit 3211 of the sensor module unit 3210 analyzes the voice of the user 10 detected by the microphone 3201 and recognizes the emotions of the user 10. For example, the voice emotion recognition unit 3211 extracts features such as frequency components of the voice and recognizes the emotions of the user 10 based on the extracted features.
  • the speech understanding unit 3212 analyzes the voice of the user 10 detected by the microphone 3201 and outputs text information representing the content of the user 10's utterance.
  • the facial expression recognition unit 3213 recognizes the facial expression and emotions of the user 10 from the image of the user 10 captured by the 2D camera 3203. For example, the facial expression recognition unit 3213 recognizes the facial expression and emotions of the user 10 based on the shape, positional relationship, etc. of the eyes and mouth.
  • the face recognition unit 3214 recognizes the face of the user 10.
  • the face recognition unit 3214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 3203.
  • the user state recognition unit 3230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 3210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 3210. For example, it generates perceptual information such as "Daddy is alone” or "There is a 90% chance that Daddy is not smiling.” It performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely.”
  • the emotion determination unit 3232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 3210 and the state of the user 10 recognized by the user state recognition unit 3230. For example, the information analyzed by the sensor module unit 3210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.
  • the emotion value indicating the emotion of user 10 is a value indicating the positive or negative emotion of the user.
  • the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as “joy,” “pleasure,” “comfort,” “relief,” “excitement,” “relief,” and “fulfillment,” it will show a positive value, and the more cheerful the emotion, the larger the value.
  • the user's emotion is an unpleasant emotion, such as “anger,” “sorrow,” “discomfort,” “anxiety,” “sorrow,” “worry,” and “emptiness,” it will show a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value will be.
  • the user's emotion is none of the above (“normal), it will show a value of 0.
  • the emotion determination unit 3232 also determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 3210 and the state of the user 10 recognized by the user state recognition unit 3230.
  • the emotion value of the robot 100 includes emotion values for each of a number of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of the emotions “joy,” “anger,” “sorrow,” and “happiness.”
  • the emotion determination unit 3232 determines an emotion value indicating the emotion of the robot 100 according to rules for updating the emotion value of the robot 100 that are determined in association with the information analyzed by the sensor module unit 3210 and the state of the user 10 recognized by the user state recognition unit 3230.
  • the emotion determination unit 3232 increases the "sad” emotion value of the robot 100. Also, if the user state recognition unit 3230 recognizes that the user 10 is smiling, the emotion determination unit 3232 increases the "happy" emotion value of the robot 100.
  • the emotion determination unit 3232 may further consider the state of the robot 100 when determining the emotion value indicating the emotion of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is completely dark, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.
  • the behavior recognition unit 3234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 3210 and the state of the user 10 recognized by the user state recognition unit 3230. For example, the information analyzed by the sensor module unit 3210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of multiple predetermined behavioral categories (e.g., "laughing,” “angry,” “asking a question,” “sad”) is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.
  • predetermined behavioral categories e.g., "laughing,” “angry,” “asking a question,” “sad
  • the robot 100 acquires the contents of the user 10's speech after identifying the user 10.
  • the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.
  • the behavior determination unit 3236 determines an action corresponding to the behavior of the user 10 recognized by the behavior recognition unit 3234 based on the current emotion value of the user 10 determined by the emotion determination unit 3232, the history data 3222 of past emotion values determined by the emotion determination unit 3232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100.
  • the behavior determination unit 3236 uses one most recent emotion value included in the history data 3222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect.
  • the behavior determination unit 3236 may use the most recent multiple emotion values as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago.
  • the behavior determination unit 3236 may determine an action corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values of the robot 100.
  • the behavior determined by the behavior determination unit 3236 includes gestures performed by the robot 100 or the contents of speech uttered by the robot 100.
  • the behavior decision unit 3236 decides the behavior of the robot 100 as the behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotion values of the user 10, the emotion value of the robot 100, the behavior of the user 10, and the reaction rules 3221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior decision unit 3236 decides the behavior for changing the emotion value of the user 10 to a positive value as the behavior corresponding to the behavior of the user 10.
  • the behavior decision unit 3236 may determine a behavior corresponding to the behavior of the user 10 based on the emotion of the robot 100. For example, when the robot 100 is verbally abused by the user 10 or when the user 10 is arrogant (i.e., when the user's reaction is poor), when the surrounding noise is loud and the voice of the user 10 cannot be detected, when the battery level of the robot 100 is low, etc., and the emotion value of "anger” or "sadness" of the robot 100 increases, the behavior decision unit 3236 may determine a behavior corresponding to the behavior of the user 10 according to the increase in the emotion value of "anger” or "sadness".
  • the behavior decision unit 3236 may determine a behavior corresponding to the behavior of the user 10 according to the increase in the emotion value of "joy” or "pleasure”. Furthermore, the behavior decision unit 3236 may decide that the behavior of the robot 100 toward the user 10 who has increased the emotional values of "anger” or “sadness” is different from the behavior of the robot 100 toward the user 10 who has increased the emotional values of "joy” or “pleasure”. In this way, the behavior decision unit 3236 may decide on different behaviors depending on the emotion of the robot 100 itself and how the user 10 has changed the emotion of the robot 100 through the action of the user 10.
  • the reaction rule 3221 defines the behavior of the robot 100 according to a combination of the past and current emotion values of the user 10, the emotion value of the robot 100, and the behavior of the user 10. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.
  • the reaction rule 3221 defines the behavior of the robot 100 for all combinations of patterns of the emotion values of the robot 100 (1296 patterns, which are the fourth power of six values of "joy”, “anger”, “sorrow”, and “pleasure”, from “0” to "5"); combination patterns of the past emotion values and the current emotion values of the user 10; and behavior patterns of the user 10. That is, for each pattern of the emotion values of the robot 100, behavior of the robot 100 is defined according to the behavior patterns of the user 10 for each of a plurality of combinations of the past emotion values and the current emotion values of the user 10, such as negative values and negative values, negative values and positive values, positive values and negative values, positive values and positive values, negative values and normal values, and normal values and normal values.
  • the behavior determination unit 3236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 3222, for example, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time.”
  • reaction rules 3221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotion value of the robot 100.
  • reaction rules 3221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotion value of the robot 100.
  • the strength of each gesture included in the behavior of the robot 100 defined in the reaction rules 3221 is predefined.
  • the strength of each utterance included in the behavior of the robot 100 defined in the reaction rules 3221 is predefined.
  • the memory control unit 3238 determines whether or not to store data including the behavior of the user 10 in the history data 3222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 3236 and the emotion value of the robot 100 determined by the emotion determination unit 3232.
  • the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 3236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 3236 is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 3222.
  • the memory control unit 3238 decides to store data including the behavior of the user 10 in the history data 3222, it stores in the history data 3222 the behavior determined by the behavior determination unit 3236, the information analyzed by the sensor module unit 3210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 3230 (e.g., the facial expression, emotions, etc. of the user 10).
  • a certain period of time ago e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene
  • the state of the user 10 recognized by the user state recognition unit 3230 e.g., the facial expression, emotions, etc. of the user 10
  • the behavior control unit 3250 controls the control target 3252 based on the behavior determined by the behavior determination unit 3236. For example, when the behavior determination unit 3236 determines an behavior that includes speaking, the behavior control unit 3250 outputs sound from a speaker included in the control target 3252. At this time, the behavior control unit 3250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 3250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 3250 determines the execution form of the behavior determined by the behavior determination unit 3236 based on the emotion value determined by the emotion determination unit 3232.
  • the behavior control unit 3250 may recognize a change in the user's 10 emotions in response to the execution of the behavior determined by the behavior determination unit 3236.
  • the change in emotions may be recognized based on the voice or facial expression of the user 10.
  • the change in emotions may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 3200. If an impact is detected by the touch sensor included in the sensor unit 3200, the user's emotions may be recognized as having worsened, and if the detection result of the touch sensor included in the sensor unit 3200 indicates that the user's 10 is smiling or happy, the user's emotions may be recognized as having improved.
  • Information indicating the user's 10 reaction is output to the communication processing unit 3280.
  • the emotion determination unit 3232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 3232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 3236 being performed on the user in the execution form determined by the behavior control unit 3250 is not bad. In addition, the emotion determination unit 3232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 3236 being performed on the user in the execution form determined by the behavior control unit 3250 is bad.
  • the behavior control unit 3250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 3250 increases the emotion value of "happiness" of the robot 100, it controls the control object 3252 to make the robot 100 perform a happy gesture. Furthermore, when the behavior control unit 3250 increases the emotion value of "sadness" of the robot 100, it controls the control object 3252 to make the robot 100 assume a droopy posture.
  • the communication processing unit 3280 is responsible for communication with the server 300. As described above, the communication processing unit 3280 transmits user reaction information to the server 300. The communication processing unit 3280 also receives updated reaction rules from the server 300. When the communication processing unit 3280 receives updated reaction rules from the server 300, it updates the reaction rules 3221.
  • the event detection unit 3290 realizes the output function described above. Details of the event detection unit 3290 will be described later.
  • the server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on the reaction rules that include actions that have generated positive reactions.
  • the behavior determining unit 3236 determines the behavior of the robot 100 based on the state recognized by the user state recognizing unit 3230.
  • the behavior determining unit 3236 may determine the behavior of the robot 100 based on not only the state of the user but also the set character.
  • the behavior determining unit 3236 may obtain an age (user age) associated with the character from the character data 3223, and determine the behavior of the robot 100 based on the obtained user age.
  • the behavior decision unit 3236 decides the behavior of the robot 100 based on the state recognized by the user state recognition unit 3230 and the set character or the age associated with the character. This makes it possible to cause the robot 100 to perform appropriate behavior according to the age of the user. In particular, it becomes possible to restrict the robot 100 from performing actions that are inappropriate for young users (for example, outputting violent content).
  • Characters are set in advance in the system 5.
  • the character settings are input as a prompt (command statement).
  • the prompt may be input via an input device provided in the robot 100, or via an external device such as a server communicatively connected to the robot 100.
  • the prompt may specify the name of the character, or may specify an ID that is set for each character.
  • the behavior decision unit 3236 decides on a behavior to output a screen showing the character's appearance or a color corresponding to the character on a display device 3521 (an example of an output device) provided on the robot.
  • the color corresponding to the character is a theme color or the like that is associated with the character. This allows the user to get the feeling of having a conversation with the character.
  • the behavior decision unit 3236 decides on an action to output information to the display device 3521 or the speaker 3522 (examples of an output device) provided on the robot 100 in a manner according to the age of the user. For example, the behavior decision unit 3236 changes the voice of the robot 100 emitted from the speaker 3522 to the character's tone of voice.
  • the action decision unit 3236 decides on an action to output a voice or a message using text constructed using words appropriate to the user's age.
  • usable words for each age are set in advance.
  • the action decision unit 3236 obtains the user's age from the character data 3223.
  • the words “What's wrong?” and “How are you doing?" are stored in advance in the storage unit 3220 as words to be output when the robot 100 selects the action of "calling out.”
  • “What's wrong?” is associated with the age of "under 12 years old”
  • "How are you doing?" is associated with the age of "12 years old or older.”
  • the action decision unit 3236 decides to output the words “How are you doing?” if the user age is "18 years old or older.”
  • the action decision unit 3236 decides to output the words "What's wrong?” if the user age is "3 to 7 years old.”
  • the behavior decision unit 3236 decides an action to output content corresponding to the character to an output device (such as the display device 3521) provided in the robot 100.
  • the behavior decision unit 3236 decides an action to display video content (such as movies, animations, etc.) in which the character appears on the display device 3521.
  • the behavior decision unit 3236 may also decide an action to output educational content according to the age of use.
  • the educational content is text, video, audio, etc. related to study subjects such as English, arithmetic, Japanese, science, and social studies.
  • the educational content may also be interactive content in which the user inputs answers to questions.
  • the behavior decision unit 3236 decides an action to display on the display device 3521 a text of a calculation problem corresponding to the grade according to the age of use. For example, if the age of use is "under 8 years old,” the behavior decision unit 3236 decides to display an addition problem, and if the age of use is "8 years old or older,” the behavior decision unit 3236 decides to display a multiplication problem.
  • the behavior decision unit 3236 may also decide on a behavior to cause the output device of the robot 100 to output content appropriate to the age of the user, rather than the character.
  • the content may be content in which a character appears, or content that is not dependent on a character, such as a commonly known folk tale or fairy tale.
  • the content corresponding to the character and the grade and educational content corresponding to the age of the user may be pre-stored in the storage unit 3220, or may be obtained from an external device such as a server connected to the robot 100 so as to be able to communicate with it.
  • FIG. 21 shows an example of an outline of the operation flow for character setting. Note that "S" in the operation flow indicates the step to be executed.
  • step S50 the robot 100 accepts the character settings. Then, in step S51, the robot 100 outputs a screen corresponding to the character (for example, a screen showing the character's appearance).
  • step S52 the behavior decision unit 3236 obtains the usage age corresponding to the set character from the character data 3223.
  • FIG. 22 shows an example of an outline of an operation flow relating to an operation for determining an action in the robot 100.
  • the operation flow shown in FIG. 22 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 3210 is input. Note that "S" in the operation flow indicates the step being executed.
  • step S3100 the user state recognition unit 3230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 3210.
  • step S3102 the emotion determination unit 3232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 3210 and the state of the user 10 recognized by the user state recognition unit 3230.
  • step S3103 the emotion determination unit 3232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 3210 and the state of the user 10 recognized by the user state recognition unit 3230.
  • the emotion determination unit 3232 adds the determined emotion value of the user 10 to the history data 3222.
  • step S3104 the behavior recognition unit 3234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 3210 and the state of the user 10 recognized by the user state recognition unit 3230.
  • step S3106 the behavior decision unit 3236 decides the behavior of the robot 100 based on the usage age acquired in step S52 of FIG. 21, the combination of the current emotion value of the user 10 determined in step S3102 of FIG. 22 and the past emotion value included in the history data 3222, the emotion value of the robot 100, the behavior of the user 10 recognized by the behavior recognition unit 3234, and the reaction rules 3221.
  • step S3108 the behavior control unit 3250 controls the control object 3252 based on the behavior determined by the behavior determination unit 3236.
  • step S3110 the memory control unit 3238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 3236 and the emotion value of the robot 100 determined by the emotion determination unit 3232.
  • step S3112 the storage control unit 3238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing data including the behavior of the user 10 in the history data 3222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S3114.
  • step S3114 the behavior determined by the behavior determination unit 3236, the information analyzed by the sensor module unit 3210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 3230 are stored in the history data 3222.
  • an emotion value indicating the emotion of the robot 100 is determined based on the user state, and whether or not to store data including the behavior of the user 10 in the history data 3222 is determined based on the emotion value of the robot 100.
  • the robot 100 can present to the user 10 all kinds of peripheral information, such as the state of the user 10 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), as well as data on the sound, image, smell, etc. of the location.
  • the robot 100 it is possible to cause the robot 100 to perform an appropriate action in response to the action of the user 10.
  • the user's actions were classified and actions including the robot's facial expressions and appearance were determined.
  • the robot 100 determines the current emotional value of the user 10 and performs an action on the user 10 based on the past emotional value and the current emotional value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the robot 100 can utter such a thing as "You were cheerful yesterday, but what's wrong with you today?" The robot 100 can also utter with gestures.
  • the robot 100 can utter such a thing as "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 who was cheerful yesterday is more cheerful today than yesterday, the robot 100 can utter such a thing as "You're more cheerful today than yesterday. Has something better happened than yesterday?" Furthermore, for example, the robot 100 can say to a user 10 whose emotion value is equal to or greater than 0 and whose emotion value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good.”
  • the robot 100 can ask the user 10, "Did you finish the homework I told you about yesterday?" and, if the user 10 responds, "I did it," make a positive utterance such as "Great! and perform a positive gesture such as clapping or a thumbs up. Also, for example, when the user 10 says, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "You did a great job! and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of closeness to the robot 100 by performing actions based on the state history of the user 10.
  • the robot 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect.
  • the robot 100 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.
  • the event detection unit 3290 is provided in the robot 100 and causes the robot 100 to output information corresponding to a detected event.
  • the event detection unit 3290 has a detection unit 3901, a collection unit 3902, and an output control unit 3903.
  • the event detection unit 3290 also stores handling information 3911.
  • Each component of the event detection unit 3290 is realized by the CPU operating based on a program.
  • the functions of these components can be implemented as CPU operations using operating system (OS) and programs that run on the OS.
  • Handling information 3911 is implemented using a storage medium such as a memory.
  • the detection unit 3901 detects the occurrence of a specified event.
  • the detection unit 3901 detects the user 10 at the user's 10 home, or where the user 10 is out (for example, outdoors such as parks, amusement parks, the sea, or mountains, or indoors such as lodging facilities such as inns, or concert halls and aquariums).
  • Examples of family members include the husband, wife, children, and the husband's and/or wife's parents (grandfathers of the children). Note that family members may also include pets.
  • the output control unit 3903 controls the robot 100 equipped with the sentence generation model to output information corresponding to the event detected by the detection unit 3901 to the user 10.
  • the collection unit 3902 collects situation information indicating the situation of the user 10, including the family.
  • the output control unit 3903 controls the robot 100 to output information corresponding to the situation information.
  • the collection unit 3902 acquires situation information over time of the user 10.
  • the situation information over time of the user 10 is situation information from a certain time in the past of the user 10 to a certain time a predetermined period of time has passed (for example, from a certain time in the past to the present).
  • the situation information over time of the user 10 includes at least one of an image and a video of the user 10.
  • the situation information over time of the user 10 is information related to the growth and development of a child.
  • the robot 100 is capable of recognizing the emotions of the user 10 from the image, video, voice, etc. of the user 10.
  • the output control unit 3903 controls the robot 100 to take an action according to the situation information collected by the collection unit 3902. First, the output control unit 3903 controls the robot 100 to take an action that is selected according to the time when the situation information of the user 10 over time collected by the collection unit 3902 was collected by the collection unit 3902. Next, the output control unit 3903 controls the robot 100 to take an action that is recorded for each selected situation information. Through this control, the robot 100 records, for example, the growth and development of a child. In other words, the robot 100 provides appropriate support according to the situation of the user 10. This allows the user 10 to share, for example, records of the child's growth and development with family members.
  • the output control unit 3903 also controls the robot 100 to take pictures (images) and videos of the user 10.
  • the output control unit 3903 also controls the robot 100 to edit pictures (images) and videos that the robot 100 itself has taken.
  • the output control unit 3903 also controls the robot 100 to save pictures (images) and videos that the robot 100 itself has edited. Through such control, the robot 100 creates a family album and preserves the family's irreplaceable memories.
  • the status information over time of the user 10 may also include at least any of the memorial items of the user 10, such as photographs (images), videos, letters, and diaries.
  • the output control unit 3903 controls the robot 100 to perform a predetermined action including at least any of organizing, storing, and backing up the memorial items collected by the collection unit 3902. Through such control, the robot 100 protects the precious memories of the family. In this way, the robot 100 can provide support that is sensitive to the emotions of the user 10.
  • the time-dependent status information of the user 10 may include the schedule of the user 10.
  • the output control unit 3903 controls the robot 100 to perform an action of setting a reminder based on the schedule of the user 10.
  • the output control unit 3903 may also control the robot 100 to perform an action of setting an alert based on the schedule of the user 10.
  • the robot 100 performs at least one of setting a reminder and setting an alert. Through such control, the robot 100 assists in managing the family's schedule. This enables smooth schedule management for the family.
  • FIG. 24 shows an example of an operational flow by the event detection unit 3290.
  • the event detection unit 3290 determines whether or not the occurrence of a specified event has been detected (step S3200). If the occurrence of a specified event has not been detected (step S3200; No), the event detection unit 3290 waits until it detects the occurrence of the specified event.
  • step S3200 if the occurrence of a predetermined event is detected (step S3200; Yes), the event detection unit 3290 collects situation information indicating the situation of the user 10 (step S3201). Next, the event detection unit 3290 controls the robot 100 equipped with the sentence generation model to output an action corresponding to the situation information to the user 10 (step S3202), and ends the process.
  • the robot 100 described above may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.
  • the emotion determination unit 3232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 3232 may determine the user's emotion according to an emotion map (see FIG. 5), which is a specific mapping.
  • emotion map 400 is a diagram showing an emotion map 400 on which multiple emotions are mapped.
  • emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged.
  • Emotions that represent states and actions arising from a state of mind are arranged on the outer sides of the concentric circles. Emotions are a concept that includes emotions and mental states.
  • emotions that are generally generated from reactions that occur in the brain are arranged.
  • emotions that are generally induced by situational judgment are arranged on the upper and lower sides of the concentric circles.
  • emotions of "pleasure” are arranged, and on the lower side, emotions of "discomfort” are arranged.
  • emotion map 400 multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.
  • the frequency of the determination of the reaction action of the robot 100 may be set to at least the same timing as the detection frequency of the emotion engine (100 msec), or may be set to an earlier timing.
  • the detection frequency of the emotion engine may be interpreted as the sampling rate.
  • reaction e.g., a backchannel
  • the reaction is performed according to the directionality and degree (strength) of the mandala in the robot's 100 emotion map 400.
  • the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.
  • the directionality of the emotion and its intensity may be preset in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability, security, etc., the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.
  • emotion map 400 These emotions are distributed in the three o'clock direction on emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness takes precedence over internal sensations, resulting in a sense of calm.
  • the filler "ah” may be inserted before the line, and if the robot 100 feels hurt after receiving harsh words, the filler "ugh! may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh! may be included. These emotions are distributed around the 9 o'clock area of the emotion map 400.
  • the robot 100 When the robot 100 feels an internal sense (reaction) of satisfaction, but also feels a favorable impression in its situational awareness, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh.” In this way, the robot 100 may generate a behavior that shows a balanced favorable impression toward the other person, that is, tolerance and psychology toward the other person.
  • Such emotions are distributed around 12 o'clock on the emotion map 400.
  • the robot 100 may shake its head when it feels disgust, or turn the eye LEDs red and glare at the other person when it feels ashamed.
  • These types of emotions are distributed around the 6 o'clock position on the emotion map 400.
  • emotion map 400 represents what is going on inside one's mind, while the outside of emotion map 400 represents behavior. Therefore, the further out you go on the emotion map 400, the more visible (more visible in behavior) your emotions become.
  • the robot 100 When listening to someone with a sense of security, which is distributed around the 3 o'clock area of the emotion map 400, the robot 100 may lightly nod its head and say “hmm,” but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.
  • the emotion determination unit 3232 inputs the information analyzed by the sensor module unit 3210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10.
  • This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 3210 and the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 400.
  • this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6.
  • Figure 6 is a diagram showing another example of an emotion map. Figure 6 shows an example in which multiple emotions, such as "peace of mind,” “calm,” and “reassuring,” have similar emotion values.
  • the emotion determination unit 3232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 3232 inputs the information analyzed by the sensor module unit 3210, the state of the user 10 recognized by the user state recognition unit 3230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100.
  • This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 3210, the recognized state of the user 10, and the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 400.
  • the neural network is trained based on learning data indicating that when the robot 100 is recognized as being stroked by the user 10 from the output of the touch sensor 207, the emotional value becomes "happy” at “3," and when the robot 100 is recognized as being hit by the user 10 from the output of the acceleration sensor 205, the emotional value becomes “anger” at “3.” Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6.
  • the emotion determination unit 3232 may also determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 3232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values indicating each emotion with the emotion values indicating each emotion of the current robot 100 to update the emotion of the robot 100. For example, the emotion values indicating each emotion obtained and the emotion values indicating each emotion of the current robot 100 are averaged and integrated.
  • This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values indicating each emotion shown in the emotion map 400.
  • the speech content of the robot 100 "That's great. You're lucky,” is obtained as the behavioral content of the robot 100 generated by the sentence generation model, then when the text representing this speech content is input to the neural network, a high emotion value for the emotion "happy” is obtained, and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes higher.
  • the behavior decision unit 3236 generates the robot's behavior by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the results into a sentence generation model with a dialogue function.
  • the behavior determination unit 3236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 3232, using an emotion table such as that shown in FIG. 25.
  • FIG. 25 is a diagram showing an example of an emotion table.
  • an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.
  • the emotion of the robot 100 determined by the emotion determination unit 3232 corresponds to index number "2"
  • the text "very happy state” is obtained. Note that if the emotions of the robot 100 correspond to multiple index numbers, multiple pieces of text representing the state of the robot 100 are obtained.
  • FIG. 26 is a diagram showing an example of an emotion table.
  • the emotion of the robot 100 is index number "2”
  • the emotion of the user 10 is index number "3”
  • the robot is in a very happy state.
  • the user is in a normally happy state.
  • the user has spoken to you, "It's AAA. How would you respond as the robot?" is input into the sentence generation model, and the content of the robot's action is obtained.
  • the action decision unit 3236 decides the robot's action from this content of the action.
  • the behavior decision unit 3236 may also generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 3222, adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting the resulting text into the dialogue function.
  • This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has a personality, and is encouraged to take actions such as talking to the robot.
  • the history data may also further include the robot's emotions and actions.
  • the plush toy 100N which is equipped with the function of an input/output device for the robot 100, has a detachable smartphone 50 that functions as the control part for the robot 100, and the input/output device is connected to the housed smartphone 50 inside the plush toy 100N.
  • the stuffed toy 100N has been described as having the shape of an animal, but is not limited to this.
  • the stuffed toy 100N may have the shape of a specific character.
  • the smartphone 50 is accommodated in the space 52 from the outside and connected to each input/output device via a USB hub 64 (see FIG. 27(B)) to provide the same functionality as the robot 100 shown in FIG. 18.
  • a non-contact type power receiving plate 66 is also connected to the USB hub 64.
  • a power receiving coil 66A is built into the power receiving plate 66.
  • the power receiving plate 66 is an example of a wireless power receiving unit that receives wireless power.
  • the power receiving plate 66 is located near the base 68 of both feet of the stuffed toy 100N, and is closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70.
  • the mounting base 70 is an example of an external wireless power transmission unit.
  • the stuffed animal 100N placed on this mounting base 70 can be viewed as an ornament in its natural state.
  • this base portion is made thinner than the surface thickness of other parts of the stuffed animal 100N, so that it is held closer to the mounting base 70.
  • the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.
  • a teddy bear 100N is exemplified, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable.
  • the material of the outer skin is not limited to cloth, and may be other materials such as soft vinyl, but a soft material is preferable.
  • a monitor may be attached to the surface of the stuffed toy 100N to add a control object 3252 that provides visual information to the user 10.
  • the eyes 56 may be used as a monitor to express joy, anger, sadness, and happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be seen.
  • the eyes 56 may be used as a projector to express joy, anger, sadness, and happiness by the image projected onto a wall.
  • the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed animal 100N.
  • the smartphone 50 When trying to use wireless charging for the smartphone 50, the smartphone 50 must be placed as far out as possible when viewed from the inside of the stuffed toy 100N, which makes the stuffed toy 100N feel rough when touched from the outside.
  • the smartphone 50 is placed as close to the center of the stuffed toy 100N as possible, and the wireless charging function (receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed toy 100N.
  • the 2D camera 3203, microphone 3201, speaker 60, and smartphone 50 receive wireless power via the receiving plate 66.
  • a collection unit that collects situation information indicating a situation of the user including a family member; an output control unit that controls an electronic device having a sentence generation model to take an action according to the situation information collected by the collection unit;
  • a behavior control system comprising:
  • the collecting unit includes: Acquiring status information of the user over time;
  • the output control unit selects the situation information collected by the collection unit according to the time when the information was collected by the collection unit, and controls the electronic device to perform an action to be recorded for each of the selected situation information.
  • the user's contextual information includes the user's schedule;
  • Fig. 29 is a diagram illustrating an example of a control system 1 according to the present embodiment.
  • the control system 1 includes a plurality of robots 100, a linked device 4400, and a server 300. Each of the plurality of robots 100 is managed by a user.
  • the robot 100 converses with the user and provides the user with video.
  • the robot 100 cooperates with a server 300 or the like with which it can communicate via the communication network 20 to converse with the user and provide the user with video, etc.
  • the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user.
  • the robot 100 also records captured video data of the user, etc. on the server 300, and requests the video data, etc. from the server 300 as necessary to provide it to the user.
  • the robot 100 also has an emotion value that represents the type of emotion it feels.
  • the robot 100 has emotion values that represent the strength of each of the emotions: “happiness,” “anger,” “sorrow,” “pleasure,” “discomfort,” “relief,” “anxiety,” “sorrow,” “excitement,” “worry,” “relief,” “fulfillment,” “emptiness,” and “neutral.”
  • the robot 100 converses with the user when its excitement emotion value is high, for example, it speaks at a fast speed. In this way, the robot 100 can express its emotions through its actions.
  • the robot 100 may be configured to determine the behavior of the robot 100 that corresponds to the emotion of the user 10 by matching a sentence generation model, a so-called AI (Artificial Intelligence) chat engine, with an emotion engine. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 that corresponds to the determined emotion.
  • AI Artificial Intelligence
  • the robot 100 when the robot 100 recognizes the behavior of the user 10, the robot 100 automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model.
  • the sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text.
  • the sentence generation model is publicly known as disclosed in, for example, JP 2018-081444 A and chatGPT (Internet search ⁇ URL: https://openai.com/blog/chatgpt>), so a detailed description thereof will be omitted.
  • Such a sentence generation model is configured by a large language model (LLM: Large Language Model).
  • this embodiment can reflect the emotions of the user 10 and the robot 100 and various linguistic information in the behavior of the robot 100 by combining a large language model and an emotion engine.
  • a synergistic effect can be obtained by combining a sentence generation model and an emotion engine.
  • the robot 100 also has a function of recognizing the user's actions.
  • the robot 100 recognizes the user's actions by analyzing the user's facial image acquired by the camera function and the user's voice acquired by the microphone function.
  • the robot 100 determines the action to be performed by the robot 100 based on the recognized user's actions, etc.
  • the robot 100 stores rules that define the actions that the robot 100 will take based on the user's emotions, the robot's 100 emotions, and the user's actions, and performs various actions according to the rules.
  • the robot 100 has reaction rules for determining the behavior of the robot 100 based on the user's emotions, the robot 100's emotions, and the user's behavior.
  • the reaction rules define the behavior of the robot 100 as “laughing” when the user's behavior is “laughing”.
  • the reaction rules also define the behavior of the robot 100 as "apologizing” when the user's behavior is “anger”.
  • the reaction rules also define the behavior of the robot 100 as "answering” when the user's behavior is "asking a question”.
  • the reaction rules also define the behavior of the robot 100 as "calling out” when the user's behavior is "sad”.
  • the robot 100 When the robot 100 recognizes the user's behavior as “anger” based on the reaction rules, it selects the behavior of "apologizing” defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing”, it performs the motion of "apologizing” and outputs a voice representing the words "apologize”.
  • the robot 100 When the robot 100 recognizes based on the reaction rules that the current emotion of the robot 100 is "normal” and that the user is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100.
  • the robot 100 also selects the action of "calling out” defined in the reaction rules as the action to be performed toward the user. For example, when the robot 100 selects the action of "calling out", it converts the words “What's wrong?", which express concern, into a concerned voice and outputs it.
  • the robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user.
  • the user reaction information includes, for example, the user action of "getting angry,” the robot 100 action of "apologizing,” the fact that the user reaction was positive, and the user's attributes.
  • the server 300 stores the user reaction information received from each robot 100. The server 300 then analyzes the user reaction information from each robot 100 and updates the reaction rules.
  • the robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300.
  • the robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate reaction rules acquired by other robots 100 into its own reaction rules. When the reaction rules are updated, they may be automatically transmitted from the server 300 to the robot 100.
  • the robot 100 cooperates with a terminal device (such as a PC 4400a, a smartphone 4400b, or a tablet 4400c) that is a linked device 4400 to execute various actions for a user.
  • a terminal device such as a PC 4400a, a smartphone 4400b, or a tablet 4400c
  • a linked device 4400 to execute various actions for a user.
  • the robot 100 recognizes the actions of the user, including the family, determines its own actions toward the user based on at least one of the recognized user actions and information about the user, and controls the control target based on its own determined actions. Specifically, the robot 100 executes actions that support the user and the user's family.
  • the robot 100 plans and manages surprise events and parties to coincide with family members' birthdays and anniversaries. That is, the robot 100 recognizes the user's emotions based on the user's actions, and determines the execution of a specified event for the user as an action based on the user's emotions and at least one of the user's actions and information about the user. Specifically, the robot 100 plans a specified event based on one or more of the interests, concerns, hobbies, tastes, inclinations, and specified anniversaries of the user contained in the information about the user, and determines the action to be taken for the planned specified event. The robot 100 recognizes the user's emotions based on the user's actions, and performs an action corresponding to the user's emotions (i.e., in keeping with the user's emotions).
  • the robot 100 can execute an event to heighten the user's emotions based on the user's birthday or other anniversaries that have been registered in advance as information about the user, and on historical data including the user's emotion value.
  • the robot 100 plans and executes a "surprise party" by inviting people other than the user (e.g., family, friends, acquaintances, etc.) to participate in the party and researching and purchasing presents that will heighten the user's emotion value "joy.”
  • the robot 100 supports the user in formulating a long-term household financial plan based on the financial situation and goals of the user's household.
  • the robot 100 also proposes an optimal financial strategy according to the family's life stage, such as education expenses, mortgages, and retirement funds. That is, the robot 100 recognizes the user's emotions based on the user's actions, and determines an action to support the user's household finances based on the user's emotions and at least one of the user's actions and information about the user. Specifically, the robot 100 determines an action to support the household finances by formulating a household financial plan or formulating a financial strategy according to the family's life stage, or both. The robot 100 recognizes the user's emotions based on the user's actions, and performs an action corresponding to the user's emotions (i.e., in tune with the user's emotions).
  • the robot 100 creates a household financial plan and a financial strategy suited to the family's life stage based on the user's educational background, career history, household income, family structure, life plan, etc., contained in information about the user that has been registered in advance, and proposes it to the user.
  • the robot 100 takes action to alleviate the user's feelings of anxiety, doubt, etc., by saying, "If you save money in a planned manner, you'll be fine!”, and to enable the user to think about saving in a positive manner.
  • the robot 100 disclosed herein recognizes the hobbies and interests of each family member, and provides information and suggests activities according to the recognized hobbies and interests. That is, the robot 100 recognizes the user's emotions based on the user's actions, and determines the action to provide predetermined information corresponding to the user's attributes based on the user's emotions and at least one of the user's actions and information about the user. The robot 100 recognizes the user's emotions based on the user's actions, and performs an action corresponding to the user's emotions (i.e., in tune with the user's emotions).
  • the robot 100 determines the behavior of providing predetermined information based on one or more of the user's attributes, such as interests, concerns, hobbies, tastes, and inclinations. For example, if the robot 100 recognizes that the user's hobby is "camping" based on the user's attribute information contained in the information about the user, it will provide information to the user by uttering something like "A campsite called XX is recommended at this time of year!.
  • the robot 100 also determines, as an action, the provision of predetermined information that satisfies predetermined conditions between the user and the user's family based on one or more of the attributes of the user and the user's family, including interests, concerns, hobbies, tastes, and inclinations. For example, when the robot 100 recognizes that a common hobby of the user and the user's family is "swimming in the sea" based on attribute information included in the information about the user and the user's family, it will utter an utterance such as "X-X beach has opened for swimming, so let's go!” and provide information to the user.
  • the predetermined condition here is a condition for determining whether the attribute information of the user and the user's family is common, and may be, for example, a condition such as whether the degree of agreement calculated based on the registered attribute information exceeds a predetermined threshold.
  • the robot 100 accompanies a family when they go out, and provides support while out, such as carrying luggage, taking care of children, providing guidance, interpreting, etc., while being sensitive to their emotions. That is, the robot 100 recognizes the user's emotions based on the user's actions, and determines a predetermined support action for the user who is out, based on the user's emotions and at least one of the user's actions and information about the user. Specifically, the robot 100 accompanies the user when they go out, and determines at least one of the following actions to be performed: carrying luggage, taking care of children, providing guidance, interpreting, talking to someone, and assisting. The robot 100 recognizes the user's emotions based on the user's actions, and performs an action corresponding to the user's emotions (i.e., being sensitive to the user's emotions).
  • the robot 100 accompanies the user and the user's family when they go out, and supports the transportation of products purchased at shopping, etc.
  • the robot 100 also accompanies the user and the user's family when they go out, and takes care of children, such as talking to the children if they have children, playing games, letting them watch videos, and watching over the children.
  • the robot 100 also accompanies the user and the user's family when they go out, and performs navigation to the destination based on a preset destination, guidance of tourist spots and landmarks at the destination, guidance of surrounding facilities, etc.
  • the robot 100 also accompanies the user and the user's family when they go out, and translates the language when it is necessary to use a language other than the user's native language when they are out.
  • the robot 100 also accompanies the user and the user's family when they go out, and asks the user questions such as "It's a nice day today! and "Have you been to this place before?", and becomes a conversation partner for the user.
  • the robot 100 accompanies the user and the user's family when they go out, and if there is someone in the user or the user's family who needs assistance, it will support that person by helping them move around, carrying their luggage, and monitoring their health.
  • the robot 100 disclosed herein supports the family's shopping by researching products desired by the user or the user's family, comparing prices, providing optimal shopping information, and carrying out ordering procedures, etc., in accordance with the preferences and needs of the family. That is, the robot 100 recognizes the user's emotions based on the user's actions, and determines an action to support the user in shopping based on the user's emotions and at least one of the user's actions and information about the user. The robot 100 recognizes the user's emotions based on the user's actions, and performs an action that corresponds to the user's emotions (i.e., that is in tune with the user's emotions).
  • the robot 100 determines the action to take to support the user in shopping based on one or more of the interests, concerns, hobbies, tastes, and inclinations contained in the information about the user. For example, if the robot 100 recognizes that the user's hobby is "camping" based on the user's attribute information contained in the information about the user, it will provide information to the user by saying, "New camping gear has been released and I recommend it! etc.
  • the robot 100 also determines at least one of the following actions to support shopping: researching products, researching product prices, providing information on the results of the product research and product price research, and ordering a product. For example, if the robot 100 recognizes that the user's hobby is "camping" based on the user's attribute information included in the information about the user, it searches for "camping equipment” sold on the Internet, etc., and collects information such as product details and prices. Next, the robot 100 provides the collected information such as product details and prices to the user. At this time, the robot 100 may perform an action to enhance the user's emotions by making an utterance such as "New camping gear (equipment) has been released and is highly recommended!. The robot 100 then accepts an utterance (permission to purchase) from the user such as "Please purchase that product,” and performs an ordering procedure for the product, thereby supporting the user's shopping.
  • the robot 100 may perform an action to enhance the user's emotions by making an utterance such as "New camping gear (equipment) has been released
  • the robot 100 recognizes changes in information about the user based on the user's state and behavior, and can then speak. For example, if the robot 100 recognizes that the user's state is "The user is having a hard time carrying the luggage and is in pain," it can speak, "I'll carry your luggage for you!” and take other actions, such as carrying the luggage on the user's behalf.
  • the robot 100 can perform actions in cooperation with a terminal device (such as a PC, smartphone, or tablet) to support the user or the user's family in a manner that is sensitive to their emotions.
  • a terminal device such as a PC, smartphone, or tablet
  • the robot 100 according to the present disclosure can perform appropriate actions for the user.
  • FIG. 30 is a diagram showing an outline of the functional configuration of the robot 100.
  • the robot 100 is composed of a control unit having a sensor unit 4200, a sensor module unit 4210, a storage unit 4220, a user state recognition unit 4230, an emotion determination unit 4232, a behavior recognition unit 4234, a behavior determination unit 4236, a memory control unit 4238, a behavior control unit 4250, a control target 4252, and a communication processing unit 4280.
  • the control object 4252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc.
  • the posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors.
  • the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100.
  • the display device is provided on the chest of the robot 100.
  • the facial expressions of the robot 100 can also be expressed by controlling the display of the display device.
  • the display device may display the contents of a conversation with a user as text.
  • the posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
  • the sensor unit 4200 includes a microphone 4201, a 3D depth sensor 4202, a 2D camera 4203, a distance sensor 4204, an acceleration sensor 4205, a thermosensor 4206, and a touch sensor 4207.
  • the microphone 4201 continuously detects sound and outputs sound data.
  • the microphone 4201 may be provided on the head of the robot 100 and may have a function of performing binaural recording.
  • the 3D depth sensor 4202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from the infrared images continuously captured by the infrared camera.
  • the 2D camera 4203 is an example of an image sensor. The 2D camera 4203 captures images using visible light and generates visible light video information.
  • the contour of the object may be detected from the video information generated by the 2D camera 4203.
  • the distance sensor 4204 detects the distance to the object by irradiating, for example, a laser or ultrasonic waves.
  • the acceleration sensor 4205 is, for example, a gyro sensor, and detects the acceleration of the robot 100.
  • the thermosensor 4206 detects the temperature around the robot 100.
  • the touch sensor 4207 is a sensor that detects a touch operation by the user, and is disposed, for example, on the head and hands of the robot 100.
  • the sensor unit 4200 may also include a clock, a sensor for motor feedback, etc.
  • the components other than the control object 4252 and the sensor unit 4200 are examples of components of the behavior control system of the robot 100.
  • the behavior control system of the robot 100 controls the control object 4252.
  • the storage unit 4220 includes reaction rules 4221 and history data 4222.
  • the history data 4222 includes the user's past emotion values and behavioral history. The emotion values and behavioral history are recorded for each user, for example, by being associated with the user's identification information.
  • At least a part of the storage unit 4220 is implemented by a storage medium such as a memory. It may include a person DB that stores the user's face image, user attribute information, etc.
  • the user attribute information here may include the user's name, age, sex, interests, concerns, hobbies, tastes, inclinations, lifestyle, personality, educational background, work history, place of residence, income, family structure, specific anniversaries, life plans, etc. Note that the functions of the components of the robot 100 shown in FIG.
  • control target 4252 the sensor unit 4200, and the storage unit 4220
  • the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program operating on the OS.
  • OS operating system
  • the sensor module unit 4210 includes a voice emotion recognition unit 4211, a speech understanding unit 4212, a facial expression recognition unit 4213, and a face recognition unit 4214. Information detected by the sensor unit 4200 is input to the sensor module unit 4210. The sensor module unit 4210 analyzes the information detected by the sensor unit 4200, and outputs the analysis result to the user state recognition unit 4230.
  • the voice emotion recognition unit 4211 of the sensor module unit 4210 analyzes the user's voice detected by the microphone 4201 to recognize the user's emotions. For example, the voice emotion recognition unit 4211 extracts features such as frequency components of the voice, and recognizes the user's emotions based on the extracted features.
  • the speech understanding unit 4212 analyzes the user's voice detected by the microphone 4201, and outputs text information representing the content of the user's utterance. For example, the speech understanding unit 4212 can analyze the content of a question asked by the user to the robot 100, such as "Please purchase that product," and output text information representing the content of the user's utterance.
  • the facial expression recognition unit 4213 recognizes the user's facial expression and emotions from the image of the user captured by the 2D camera 4203. For example, the facial expression recognition unit 4213 recognizes the user's facial expression and emotions based on the shape and positional relationship of the eyes and mouth. For example, the facial expression recognition unit 4213 can recognize the facial expression and emotions when the user is performing a predetermined action or when the user is asking the robot 100 a question.
  • predetermined action refers to actions such as speaking, moving, and physical movements performed by the user, and the type and content of the action are not particularly limited.
  • the face recognition unit 4214 recognizes the user's face.
  • the face recognition unit 4214 recognizes the user by matching a face image stored in a person DB (not shown) with the face image of the user captured by the 2D camera 4203.
  • the user state recognition unit 4230 recognizes the state of the user based on the information analyzed by the sensor module unit 4210. For example, the analysis results of the sensor module unit 4210 are used to mainly perform processing related to perception. For example, the user state recognition unit 4230 generates perceptual information such as "The user has a painful expression on his face” or "The user is moving around carrying luggage” and performs processing to understand the meaning of the generated perceptual information. For example, the user state recognition unit 4230 generates semantic information such as "The user is in pain because it is difficult for him to carry his luggage.”
  • the emotion determination unit 4232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 4210 and the user's state recognized by the user state recognition unit 4230. For example, the information analyzed by the sensor module unit 4210 and the recognized user's state are input to a pre-trained neural network to obtain an emotion value indicating the user's emotion.
  • the emotion value indicating the user's emotion is a value indicating the positive or negative state of the user's emotion; for example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as “joy,” “pleasure,” “comfort,” “relief,” “excitement,” “relief,” and “fulfillment,” it will show a positive value, and the more cheerful the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as “anger,” “sorrow,” “discomfort,” “anxiety,” “sorrow,” “worry,” and “emptiness,” it will show a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value will be. If the user's emotion is none of the above (“normal”), it will show a value of 0.
  • the emotion determination unit 4232 also determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 4210 and the state of the user recognized by the user state recognition unit 4230.
  • the emotion value of the robot 100 includes emotion values for each of a number of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of the emotions “joy,” “anger,” “sorrow,” and “happiness.”
  • the emotion determination unit 4232 determines an emotion value indicating the emotion of the robot 100 according to rules for updating the emotion value of the robot 100 that are determined in association with the information analyzed by the sensor module unit 4210 and the state of the user recognized by the user state recognition unit 4230.
  • the emotion determination unit 4232 increases the "sad” emotion value of the robot 100. Also, if the user state recognition unit 4230 recognizes that the user is smiling, the emotion determination unit 4232 increases the "happy" emotion value of the robot 100.
  • the emotion determination unit 4232 may further consider the state of the robot 100 when determining the emotion value indicating the emotion of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is completely dark, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when a user continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.
  • the behavior recognition unit 4234 recognizes the user's behavior based on the information analyzed by the sensor module unit 4210 and the user's state recognized by the user state recognition unit 4230. For example, the information analyzed by the sensor module unit 4210 and the recognized user's state are input into a pre-trained neural network, the probability of each of a number of predetermined behavioral categories (e.g., "laughing,” “angry,” “asking a question,” “sad”) is obtained, and the behavioral category with the highest probability is recognized as the user's behavior. For example, the behavior recognition unit 4234 recognizes the user's behavior such as "holding a terminal device in hand,” “operating a terminal device,” “performing a specified action,” etc.
  • a number of predetermined behavioral categories e.g., "laughing,” "angry,” “asking a question,” “sad”
  • the robot 100 acquires the user's speech content after identifying the user, but in order to acquire and use the speech content, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user, and the behavior control system of the robot 100 in this embodiment takes into consideration the protection of the user's personal information and privacy.
  • the behavior determination unit 4236 determines an action corresponding to the user's action recognized by the behavior recognition unit 4234 based on the user's current emotion value determined by the emotion determination unit 4232, the history data 4222 of past emotion values determined by the emotion determination unit 4232 before the user's current emotion value was determined, and the emotion value of the robot 100.
  • the behavior determination unit 4236 uses one most recent emotion value included in the history data 4222 as the user's past emotion value, but the disclosed technology is not limited to this aspect.
  • the behavior determination unit 4236 may use a plurality of most recent emotion values as the user's past emotion value, or may use an emotion value from a unit period ago, such as one day ago.
  • the behavior determination unit 4236 may determine an action corresponding to the user's action by further considering not only the current emotion value of the robot 100 but also the history of the robot 100's past emotion values.
  • the behavior determined by the behavior determination unit 4236 includes gestures performed by the robot 100 or the contents of speech uttered by the robot 100.
  • the behavior decision unit 4236 may determine the behavior corresponding to the user's behavior based on the emotion of the robot 100. For example, when the robot 100 is verbally abused by the user or when the user is arrogant (i.e., when the user's reaction is poor), when the surrounding noise is loud and the user's voice cannot be detected, when the battery level of the robot 100 is low, etc., if the emotion value of "anger” or "sadness" of the robot 100 increases, the behavior decision unit 4236 may determine the behavior corresponding to the user's behavior according to the increase in the emotion value of "anger” or "sadness".
  • the behavior decision unit 4236 may determine the behavior corresponding to the user's behavior according to the increase in the emotion value of "joy” or "pleasure”. Furthermore, the behavior determining unit 4236 may determine a behavior for a user who has increased the emotional values of "joy” or “pleasure” of the robot 100 to be different from the behavior for a user who has increased the emotional values of "anger” or “sadness” of the robot 100. In this way, the behavior determining unit 4236 may determine a different behavior depending on the emotion of the robot itself or how the user has changed the emotion of the robot 100 through the user's behavior.
  • the behavior decision unit 4236 decides the behavior of the robot 100 as the behavior corresponding to the user's behavior, based on a combination of the user's past emotion value and current emotion value, the emotion value of the robot 100, the user's behavior, and the reaction rules 4221. For example, when the user's past emotion value is a positive value and the current emotion value is a negative value, the behavior decision unit 4236 decides the behavior corresponding to the user's behavior, which is to change the user's emotion value to a positive value.
  • the reaction rule 4221 defines the behavior of the robot 100 according to a combination of the user's past and current emotion values, the emotion value of the robot 100, and the user's behavior. For example, when the user's past emotion value is a positive value and the current emotion value is a negative value, and the user's behavior is sad, a combination of gestures and speech content when asking a question to encourage the user with gestures is defined as the behavior of the robot 100.
  • the reaction rule 4221 defines the behavior of the robot 100 for all combinations of patterns of the robot 100's emotion values (1296 patterns, which are the fourth power of six values of "joy”, “anger”, “sorrow”, and “pleasure”, from “0” to "5"); combination patterns of the user's past emotion values and current emotion values; and the user's behavior patterns. That is, for each pattern of the robot 100's emotion values, the behavior of the robot 100 is defined according to the user's behavior pattern for each of a plurality of combinations of the user's past emotion values and current emotion values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal, and normal and normal.
  • the behavior determination unit 4236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 4222, for example, when the user makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time.”
  • reaction rules 4221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotion value of the robot 100.
  • reaction rules 4221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotion value of the robot 100.
  • the strength of each gesture included in the behavior of the robot 100 defined in the reaction rules 4221 is predefined.
  • the strength of each utterance included in the behavior of the robot 100 defined in the reaction rules 4221 is predefined.
  • the reaction rules 4221 prescribe the behavior of the robot 100 corresponding to a behavioral pattern such as when holding a terminal device in the hand, operating the terminal device, performing a predetermined action, and utterances related to a user's request.
  • a behavioral pattern such as when holding a terminal device in the hand, operating the terminal device, performing a predetermined action, and utterances related to a user's request.
  • An example of an utterance related to a user's request may be a question to the robot 100 such as "Please buy that product.”
  • the memory control unit 4238 determines whether or not to store data including the user's behavior in the history data 4222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 4236 and the emotion value of the robot 100 determined by the emotion determination unit 4232.
  • the predetermined intensity for the gesture included in the action determined by the action determination unit 4236, and the predetermined intensity for the speech content included in the action determined by the action determination unit 4236 is equal to or greater than a threshold value, it is determined that data including the user's action is to be stored in the history data 4222.
  • the memory control unit 4238 decides to store data including the user's behavior in the history data 4222, it stores in the history data 4222 the behavior determined by the behavior determination unit 4236, information analyzed by the sensor module unit 4210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, and smells of the scene), and the user's state recognized by the user state recognition unit 4230 (e.g., the user's facial expression, emotions, etc.).
  • a certain period of time ago e.g., all peripheral information such as data on the sound, images, and smells of the scene
  • the user's state recognized by the user state recognition unit 4230 e.g., the user's facial expression, emotions, etc.
  • the behavior control unit 4250 controls the control target 4252 based on the behavior determined by the behavior determination unit 4236. For example, when the behavior determination unit 4236 determines an behavior including speaking, the behavior control unit 4250 outputs a voice from a speaker included in the control target 4252. At this time, the behavior control unit 4250 may determine the speech speed of the voice based on the emotion value of the robot 100. For example, the behavior control unit 4250 determines a faster speech speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 4250 determines the execution form of the behavior determined by the behavior determination unit 4236 based on the emotion value determined by the emotion determination unit 4232. Specifically, the behavior control unit 4250 recognizes the behavior of the user, including family members, determines its own behavior toward the user based on at least one of the recognized user behavior or information about the user, and controls the control target based on its own determined behavior.
  • the behavior control unit 4250 recognizes the user's emotions based on the user's actions, and executes a predetermined event for the user as an action based on the user's emotions and at least one of the user's actions and information about the user. Specifically, the behavior control unit 4250 plans a predetermined event based on one or more of the interests, concerns, hobbies, tastes, inclinations, and predetermined anniversaries of the user contained in the information about the user, and executes an action to execute the planned predetermined event.
  • the behavior control unit 4250 also executes actions such as holding events to heighten the user's emotions for the user's birthdays and other anniversaries, which have been registered in advance as information about the user, based on historical data including the user's emotion value. As a specific example, on the user's birthday, the behavior control unit 4250 executes actions such as planning and executing a "surprise party,” inviting people other than the user (e.g., family, friends, acquaintances, etc.) to participate in the party and researching and purchasing presents that will heighten the user's emotion value "joy.”
  • a surprise party inviting people other than the user (e.g., family, friends, acquaintances, etc.) to participate in the party and researching and purchasing presents that will heighten the user's emotion value "joy.”
  • the behavior control unit 4250 also recognizes the user's emotions based on the user's behavior, and performs behavior to support the user's household finances based on the user's emotions and at least one of the user's behavior and information about the user. Specifically, the behavior control unit 4250 performs behavior to support the household finances, such as planning a household financial plan or planning a financial strategy according to the family's life stage, or both. For example, the behavior control unit 4250 performs behavior to propose to the user, planning a household financial plan or a financial strategy according to the family's life stage, based on the user's educational background, career, household income, family structure, life plan, etc., contained in the information about the user that has been registered in advance.
  • the robot 100 when proposing "savings” to the user, the robot 100 performs behavior that alleviates the user's feelings, such as anxiety and doubt, by uttering, for example, "You'll be fine if you save in a planned manner!, thereby enabling the user to think about saving in a positive manner.
  • the behavior control unit 4250 also recognizes the user's emotions based on the user's behavior, and performs a predetermined information provision action based on the user's emotions and at least one of the user's behavior and the information about the user, which corresponds to the user's attributes. Specifically, the behavior control unit 4250 performs a predetermined information provision action based on one or more of the user's attributes, such as interests, concerns, hobbies, tastes, and inclinations. For example, when the behavior control unit 4250 recognizes that the user's hobby is "camping" based on the user's attribute information included in the information about the user, it provides information to the user by uttering an utterance such as "A campsite called XX is recommended at this time of year!.
  • the behavior control unit 4250 also performs a predetermined information provision action that satisfies a predetermined condition between the user and the user's family, based on one or more of the user's attributes, such as interests, concerns, hobbies, tastes, and inclinations. For example, if the behavior control unit 4250 recognizes that a common hobby of the user and the user's family is "swimming in the sea” based on attribute information contained in information about the user and the user's family, it will make an utterance such as "XX beach has opened for swimming, so let's go!” and perform an action such as providing information to the user.
  • the behavior control unit 4250 also recognizes the user's emotions based on the user's actions, and performs a predetermined support action for the user who is out based on the user's emotions and at least one of the user's actions or information about the user. Specifically, the behavior control unit 4250 accompanies the user when he or she goes out, and performs at least one of the following actions: carrying luggage, taking care of children, guidance, interpretation, conversation, and assistance. For example, the behavior control unit 4250 accompanies the user and the user's family when they go out, and performs actions such as supporting the carrying of products purchased at shopping, etc.
  • the behavior control unit 4250 also accompanies the user and the user's family when they go out, and performs actions such as talking to the child if there is a child, playing games, letting the child watch videos, and watching over the child.
  • the behavior control unit 4250 also accompanies the user and the user's family when they go out, and performs actions such as guidance to the destination, guidance to tourist spots and landmarks at the destination, and guidance to surrounding facilities based on a preset destination.
  • the behavior control unit 4250 also accompanies the user and the user's family when they go out, and when it is necessary to use a language other than the user's native language while out, performs language translation to act as an interpreter for the user.
  • the behavior control unit 4250 also accompanies the user and the user's family when they go out, and asks the user questions such as "It's a nice day today! and "Have you been to this place before?", and performs actions such as serving as a conversation partner for the user.
  • the behavior control unit 4250 also accompanies the user and the user's family when they go out, and when the user or the user's family includes someone who needs assistance, performs actions such as supporting the movement of that person, carrying luggage, and monitoring the health of that person.
  • the behavior control unit 4250 recognizes the user's emotions based on the user's behavior, and performs shopping support for the user as an action based on the user's emotions and at least one of the user's behavior and the information about the user. Specifically, the behavior control unit 4250 performs shopping support for the user as an action based on one or more of the interests, concerns, hobbies, tastes, and inclinations contained in the information about the user. For example, when the behavior control unit 4250 recognizes that the user's hobby is "camping" based on the user's attribute information contained in the information about the user, it performs an action such as making an utterance such as "New camping gear (tools) has been released and I recommend it! and providing information to the user.
  • the behavior control unit 4250 recognizes that the user's hobby is "camping" based on the user's attribute information contained in the information about the user, it performs an action such as making an utterance such as "New camping gear (tools) has been released and I recommend it! and providing information to
  • the behavior control unit 4250 performs at least one of the following shopping support actions: product research, product price research, providing information on the results of the product research and product price research, and product ordering procedures. For example, when the behavior control unit 4250 recognizes that the user's hobby is "camping" based on the user's attribute information included in the information about the user, it performs an action such as searching for "camping equipment” sold on the Internet, etc., and collecting information such as product details and prices. Next, the behavior control unit 4250 performs an action such as providing the collected information such as product details and prices to the user.
  • the behavior control unit 4250 may perform an action to enhance the user's emotions by making an utterance such as "New camping gear (equipment) has been released and is highly recommended!. Then, the behavior control unit 4250 accepts an utterance (permission to purchase) from the user such as "Please purchase that product” and performs an order procedure for the product, thereby performing an action such as supporting the user's shopping.
  • the behavior control unit 4250 may recognize a change in the user's emotions in response to the execution of the behavior determined by the behavior determination unit 4236.
  • the change in emotions may be recognized based on the user's voice or facial expression.
  • the change in the user's emotions may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 4200. If an impact is detected by the touch sensor included in the sensor unit 4200, the user's emotions may be recognized as having worsened, and if the detection result of the touch sensor included in the sensor unit 4200 indicates that the user's reaction is laughing or happiness, the user's emotions may be recognized as having improved. Information indicating the user's reaction is output to the communication processing unit 4280.
  • the emotion determination unit 4232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 4232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 4236 being performed on the user in the execution form determined by the behavior control unit 4250 is not bad. In addition, the emotion determination unit 4232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 4236 being performed on the user in the execution form determined by the behavior control unit 4250 is bad.
  • the behavior control unit 4250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 4250 increases the emotion value of the robot 100's "happiness”, it controls the control object 4252 to make the robot 100 perform a happy gesture. Furthermore, when the behavior control unit 4250 increases the emotion value of the robot 100's "sadness", it controls the control object 4252 to make the robot 100 assume a droopy posture.
  • the communication processing unit 4280 is responsible for communication with the server 300. As described above, the communication processing unit 4280 transmits user reaction information to the server 300. The communication processing unit 4280 also receives updated reaction rules from the server 300. When the communication processing unit 4280 receives updated reaction rules from the server 300, it updates the reaction rules 4221. The communication processing unit 4280 can transmit and receive information to and from the linked device 4400.
  • the server 300 communicates between each robot 100 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.
  • FIG. 31 is a diagram showing an example of an operation flow relating to an operation for determining an action in the robot 100.
  • the operation flow shown in FIG. 31 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 4210 is input. Note that "S" in the operation flow indicates the step being executed.
  • the user state recognition unit 4230 recognizes the user's state based on the information analyzed by the sensor module unit 4210. For example, the user state recognition unit 4230 generates perceptual information such as "The user has a painful expression on his face” or "The user is moving around carrying luggage” and performs processing to understand the meaning of the generated perceptual information. For example, the user state recognition unit 4230 generates semantic information such as "The user is in pain because it is difficult for him to carry his luggage.”
  • step S4102 the emotion determination unit 4232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 4210 and the user's state recognized by the user state recognition unit 4230.
  • step S4103 the emotion determination unit 4232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 4210 and the state of the user recognized by the user state recognition unit 4230.
  • the emotion determination unit 4232 adds the determined emotion value of the user to the history data 4222.
  • the behavior recognition unit 4234 recognizes the user's behavior classification based on the information analyzed by the sensor module unit 4210 and the user's state recognized by the user state recognition unit 4230. For example, the behavior recognition unit 4234 recognizes the user's behavior such as "holding a terminal device in hand,” “operating a terminal device,” “performing a specified action,” etc.
  • step S4105 the behavior decision unit 4236 decides the behavior of the robot 100 based on a combination of the user's current emotion value determined in step S4102 and the past emotion values included in the history data 4222, the emotion value of the robot 100, the user's behavior recognized by the behavior recognition unit 4234, and the reaction rules 4221.
  • the behavior control unit 4250 controls the control target 4252 based on the behavior determined by the behavior determination unit 4236.
  • the behavior control unit 4250 recognizes the user's emotions based on the user's behavior, and executes a predetermined event for the user based on the user's emotions and at least one of the user's behavior or information about the user as an action.
  • the behavior control unit 4250 recognizes the user's emotions based on the user's behavior, and executes support for the user's household finances based on the user's emotions and at least one of the user's behavior or information about the user as an action.
  • the behavior control unit 4250 recognizes the user's emotions based on the user's behavior, and executes predetermined information provision corresponding to the user's attributes based on the user's emotions and at least one of the user's behavior or information about the user as an action. For example, the behavior control unit 4250 recognizes the user's emotions based on the user's behavior, and executes predetermined support for the user who is out based on the user's emotions and at least one of the user's behavior or information about the user. For example, the behavior control unit 4250 recognizes the user's emotions based on the user's actions, and performs shopping support for the user as an action based on the user's emotions and at least one of the user's actions and information about the user.
  • step S4107 the memory control unit 4238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 4236 and the emotion value of the robot 100 determined by the emotion determination unit 4232.
  • step S4108 the storage control unit 4238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing data including the user's behavior in the history data 4222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S4109.
  • step S4109 the behavior determined by the behavior determination unit 4236, the information analyzed by the sensor module unit 4210 from the current time up to a certain period of time ago, and the user's state recognized by the user state recognition unit 4230 are stored in the history data 4222.
  • the robot 100 is equipped with a control unit that recognizes the actions of users, including family members, determines its own actions toward the user based on at least one of the recognized user actions and information about the user, and controls the control target based on its own determined actions.
  • the control unit of the robot 100 recognizes the user's emotions based on the user's actions, and determines the execution of a predetermined event for the user as an action based on the user's emotions and at least one of the user's actions and information about the user. Specifically, the control unit of the robot 100 plans a predetermined event based on one or more of the interests, concerns, hobbies, tastes, inclinations, and the user's predetermined anniversaries contained in the information about the user, and determines the action to be taken for the planned predetermined event. This makes it possible for the control unit of the robot 100 to plan and manage surprise events and parties in accordance with family birthdays and anniversaries.
  • control unit of the robot 100 since the control unit of the robot 100 has an emotion engine, it recognizes the user's emotions and enables the execution of an event that is in line with the user. Therefore, the robot 100 provides the effect of taking into account the preferences, interests, or emotions of the user and the user's family and enabling them to create beautiful memories.
  • the control unit of the robot 100 recognizes the user's emotions based on the user's actions, and determines, as an action, support for the user's household finances based on the user's emotions and at least one of the user's actions and information about the user. Specifically, the control unit of the robot 100 determines, as an action, to execute either or both of the planning of a household financial plan and the planning of a financial strategy according to the family's life stage as support for household finances. In this way, the control unit of the robot 100 can support the planning of a long-term household financial plan based on the financial situation and goals of the user's household.
  • control unit of the robot 100 can propose an optimal financial strategy according to the family's life stage, such as education expenses, mortgages, and retirement funds. Furthermore, since the control unit of the robot 100 has an emotion engine, it can recognize the user's emotions and make proposals that are tailored to the user. Therefore, the robot 100 provides the effect of enabling financial support for the user's household to be executed in a tailored manner.
  • the control unit of the robot 100 recognizes the user's emotions based on the user's actions, and determines, as an action, a predetermined information provision corresponding to the user's attributes based on the user's emotions and at least one of the user's actions or information about the user. Specifically, the control unit of the robot 100 determines, as an action, a predetermined information provision based on any one or more of interests, concerns, hobbies, tastes, and inclinations as the user's attributes.
  • the control unit of the robot 100 also determines, as an action, a predetermined information provision that satisfies a predetermined condition for the user and the user's family based on any one or more of interests, concerns, hobbies, tastes, and inclinations as the attributes of the user and the user's family. In this way, the control unit of the robot 100 understands the hobbies and interests of each family member, and is able to provide information and suggest activities accordingly. Furthermore, since the control unit of the robot 100 has an emotion engine, it is able to recognize the user's emotions and provide information that is tailored to the user. Therefore, the robot 100 provides an effect of enabling support for finding a common hobby that the user and the family can enjoy.
  • the control unit of the robot 100 recognizes the user's emotions based on the user's actions, and determines a predetermined support action for the user who is out based on the user's emotions and at least one of the user's actions or information about the user. Specifically, the control unit of the robot 100 determines the action to be to accompany the user when they go out and perform at least one of carrying luggage, taking care of children, guiding, interpreting, talking to someone, and assisting. This makes it possible for the control unit of the robot 100 to provide the support necessary for going out, such as carrying luggage, taking care of children, guiding, interpreting, etc., while being sensitive to the user's emotions when accompanying a family outing. Furthermore, since the control unit of the robot 100 has an emotion engine, it is possible to recognize the user's emotions and provide support that is sensitive to the user. Therefore, the robot 100 provides the effect of being able to provide support that is sensitive to the user when they go out.
  • the control unit of the robot 100 recognizes the user's emotions based on the user's behavior, and determines the behavior of shopping support for the user based on the user's emotions and at least one of the user's behavior and the information about the user. Specifically, the control unit of the robot 100 determines the behavior of shopping support for the user based on one or more of the interests, concerns, hobbies, tastes, and inclinations included in the information about the user. The control unit of the robot 100 also determines the behavior of shopping support to be at least one of product research, product price research, providing information on the results of the product research and product price research, and product ordering procedures.
  • control unit of the robot 100 This enables the control unit of the robot 100 to support family shopping by researching products that the family wants, comparing prices, providing optimal shopping information, ordering procedures, etc., according to the family's preferences and needs. Furthermore, since the control unit of the robot 100 has an emotion engine, it recognizes the user's emotions and enables shopping support that is close to the user. Therefore, the robot 100 provides the effect of enabling support to be provided close to the user during shopping.
  • the robot 100 described above may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.
  • the emotion determination unit 4232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 4232 may determine the user's emotion according to an emotion map (see FIG. 5), which is a specific mapping.
  • emotion map 400 is a diagram showing an emotion map 400 on which multiple emotions are mapped.
  • emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged.
  • Emotions that represent states and actions arising from a state of mind are arranged on the outer sides of the concentric circles. Emotions are a concept that includes emotions and mental states.
  • emotions that are generally generated from reactions that occur in the brain are arranged.
  • emotions that are generally induced by situational judgment are arranged on the upper and lower sides of the concentric circles.
  • emotions of "pleasure” are arranged, and on the lower side, emotions of "discomfort” are arranged.
  • emotion map 400 multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.
  • the frequency of the determination of the reaction action of the robot 100 may be set to at least the same timing as the detection frequency of the emotion engine (100 msec), or may be set to an earlier timing.
  • the detection frequency of the emotion engine may be interpreted as the sampling rate.
  • reaction e.g., a backchannel
  • the reaction is performed according to the directionality and degree (strength) of the mandala in the robot's 100 emotion map 400.
  • the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.
  • the directionality of the emotion and its intensity may be preset in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability, security, etc., the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.
  • emotion map 400 These emotions are distributed in the three o'clock direction on emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness takes precedence over internal sensations, resulting in a sense of calm.
  • the filler "ah” may be inserted before the line, and if the robot 100 feels hurt after receiving harsh words, the filler "ugh! may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh! may be included. These emotions are distributed around the 9 o'clock area of the emotion map 400.
  • the robot 100 When the robot 100 feels an internal sense (reaction) of satisfaction, but also feels a favorable impression in its situational awareness, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh.” In this way, the robot 100 may generate a behavior that shows a balanced favorable impression toward the other person, that is, tolerance and psychology toward the other person.
  • Such emotions are distributed around 12 o'clock on the emotion map 400.
  • the robot 100 may shake its head when it feels disgust, or turn the eye LEDs red and glare at the other person when it feels ashamed.
  • These types of emotions are distributed around the 6 o'clock position on the emotion map 400.
  • emotion map 400 represents what is going on inside one's mind, while the outside of emotion map 400 represents behavior. Therefore, the further out you go on the emotion map 400, the more visible (more visible in behavior) your emotions become.
  • the robot 100 When listening to someone with a sense of security, which is distributed around the 3 o'clock area of the emotion map 400, the robot 100 may lightly nod its head and say “hmm,” but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.
  • the emotion determination unit 4232 inputs the information analyzed by the sensor module unit 4210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10.
  • This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 4210 and the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 400.
  • this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6.
  • Figure 6 is a diagram showing another example of an emotion map. Figure 6 shows an example in which multiple emotions, such as "peace of mind,” “calm,” and “reassuring,” have similar emotion values.
  • the emotion determination unit 4232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 4232 inputs the information analyzed by the sensor module unit 4210, the state of the user 10 recognized by the user state recognition unit 4230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100.
  • This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 4210, the recognized state of the user 10, and the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 400.
  • the neural network is trained based on learning data indicating that when the output of the touch sensor 4207 indicates that the robot 100 is being stroked by the user 10, the emotional value becomes "happy” at “3,” and based on learning data indicating that when the output of the acceleration sensor 4205 indicates that the robot 100 is being hit by the user 10, the emotional value becomes "anger” at “3.” Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6.
  • the emotion determination unit 4232 may also determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 4232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values indicating each emotion with the emotion values indicating each emotion of the current robot 100 to update the emotion of the robot 100. For example, the emotion values indicating each emotion obtained and the emotion values indicating each emotion of the current robot 100 are averaged and integrated.
  • This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values indicating each emotion shown in the emotion map 400.
  • the speech content of the robot 100 "That's great. You're lucky,” is obtained as the behavioral content of the robot 100 generated by the sentence generation model, then when the text representing this speech content is input to the neural network, a high emotion value for the emotion "happy” is obtained, and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes higher.
  • the behavior decision unit 4236 generates the robot's behavior by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the results into a sentence generation model with a dialogue function.
  • the behavior determination unit 4236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 4232, using an emotion table such as that shown in FIG. 25.
  • FIG. 25 is a diagram showing an example of an emotion table.
  • an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.
  • the emotion of the robot 100 determined by the emotion determination unit 4232 corresponds to index number "2"
  • the text "very happy state” is obtained. Note that if the emotions of the robot 100 correspond to multiple index numbers, multiple pieces of text representing the state of the robot 100 are obtained.
  • FIG. 26 is a diagram showing an example of an emotion table.
  • the user's behavior is "talk to AAA”
  • the emotion of the robot 100 is index number "2”
  • the emotion of the user 10 is index number "3”
  • the robot is in a very happy state.
  • the user is in a normal happy state.
  • the user has spoken to you with 'AAA'. How would you respond as the robot?" is input into the sentence generation model to obtain the robot's behavior content.
  • the behavior decision unit 4236 decides on the robot's behavior from this behavior content.
  • “AAA” is the name (nickname) given to the robot 100 by the user.
  • the robot 100 can change its behavior according to the index number that corresponds to the robot's emotions, so the user gets the impression that the robot 100 has a heart, and is encouraged to take actions such as talking to the robot.
  • the behavior decision unit 4236 may also generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 4222, adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function.
  • This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has a personality, and is encouraged to take actions such as talking to the robot.
  • the history data may also further include the robot's emotions and behavior.
  • a robot comprising:
  • the control unit is Recognizing an emotion of the user based on a behavior of the user; determining, as an action, execution of a predetermined event for the user based on the emotion of the user and at least one of a behavior of the user and information about the user; 2.
  • the control unit is Plan the specified event based on one or more of the interests, concerns, hobbies, tastes, and inclinations contained in the information about the user, and a specified anniversary of the user; determining the action to be taken for said predetermined planned events; 3.
  • the control unit is Recognizing an emotion of the user based on a behavior of the user; determining, as an action, support for the user's household finances based on the user's emotions and at least one of the user's behavior and information about the user; 2.
  • the robot of claim 1
  • the control unit is As a support for the household finances, a decision is made to take action to formulate a household financial plan or formulate a financial strategy according to the family's life stage, or to implement both. 5.
  • the control unit is Recognizing an emotion of the user based on a behavior of the user; determining, as an action, a predetermined information provision corresponding to an attribute of the user based on the emotion of the user and at least one of a behavior of the user and information about the user; 2.
  • the control unit is determining, as an action, to provide the predetermined information based on one or more of the user's attributes including interests, concerns, hobbies, preferences, and inclinations; 7.
  • the control unit is Recognizing an emotion of the user based on a behavior of the user; determining, as an action, a predetermined support for the user who is out and about, based on the emotion of the user and at least one of a behavior of the user and information about the user; 2.
  • the robot of claim 1
  • the control unit is Accompanying the user when he/she goes out and determining that at least one of the following actions is to be performed: carrying luggage, taking care of children, guiding, interpreting, being a conversation partner, and providing assistance;
  • the robot according to claim 9.
  • the control unit is Recognizing an emotion of the user based on a behavior of the user; determining, as an action, shopping support for the user based on the emotion of the user and at least one of a behavior of the user and information related to the user; 2.
  • the control unit is determining, as an action, shopping support for the user based on any one or more of interests, concerns, hobbies, preferences, and inclinations included in the information about the user; 12.
  • the control unit is determining at least one of the following actions as the shopping support: researching a product, researching the price of the product, providing information on the results of the product research and the price research, and ordering the product; 13.
  • the robot comprises: It is installed in the stuffed toy or connected wirelessly or by wire to a controlled device installed in the stuffed toy, The robot according to claim 1.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Human Computer Interaction (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Acoustics & Sound (AREA)
  • Strategic Management (AREA)
  • Data Mining & Analysis (AREA)
  • Tourism & Hospitality (AREA)
  • Economics (AREA)
  • Human Resources & Organizations (AREA)
  • Artificial Intelligence (AREA)
  • Multimedia (AREA)
  • Robotics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Mechanical Engineering (AREA)
  • Software Systems (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Development Economics (AREA)
  • Psychiatry (AREA)
  • Signal Processing (AREA)
  • Child & Adolescent Psychology (AREA)
  • Hospice & Palliative Care (AREA)
  • Game Theory and Decision Science (AREA)

Abstract

Ce système de commande d'action comprend : une unité de détermination d'émotion qui détermine l'émotion d'un utilisateur ou l'émotion d'un robot ; et une unité de détermination d'action qui, d'après un modèle de génération de texte ayant une fonction d'interaction qui permet à un utilisateur et à un robot d'interagir, génère une description d'actions du robot par rapport aux actions de l'utilisateur et aux émotions de l'utilisateur ou du robot, puis détermine l'action du robot correspondant à la description de l'action. L'unité de détermination d'action reçoit un énoncé d'une pluralité d'utilisateurs qui sont en train de parler, et lorsque l'énoncé a pris un état prédéterminé, détermine, en tant qu'action du robot, de générer un sujet dont la description est différente de celle de l'énoncé.
PCT/JP2024/014849 2023-04-17 2024-04-12 Système de commande d'action et de robot Ceased WO2024219336A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202480026288.1A CN121127857A (zh) 2023-04-17 2024-04-12 行动控制系统及机器人

Applications Claiming Priority (20)

Application Number Priority Date Filing Date Title
JP2023067477 2023-04-17
JP2023-067477 2023-04-17
JP2023-068069 2023-04-18
JP2023068069 2023-04-18
JP2023-069616 2023-04-20
JP2023069616 2023-04-20
JP2023-071300 2023-04-25
JP2023071300 2023-04-25
JP2023072774 2023-04-26
JP2023-072774 2023-04-26
JP2023-072769 2023-04-26
JP2023072769 2023-04-26
JP2023-073672 2023-04-27
JP2023073672 2023-04-27
JP2023073671 2023-04-27
JP2023-073671 2023-04-27
JP2023-074509 2023-04-28
JP2023074509 2023-04-28
JP2023081665 2023-05-17
JP2023-081665 2023-05-17

Publications (1)

Publication Number Publication Date
WO2024219336A1 true WO2024219336A1 (fr) 2024-10-24

Family

ID=93152466

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/014849 Ceased WO2024219336A1 (fr) 2023-04-17 2024-04-12 Système de commande d'action et de robot

Country Status (2)

Country Link
CN (1) CN121127857A (fr)
WO (1) WO2024219336A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017049427A (ja) * 2015-09-01 2017-03-09 カシオ計算機株式会社 対話制御装置、対話制御方法及びプログラム
WO2019054464A1 (fr) * 2017-09-15 2019-03-21 Ｇｒｏｏｖｅ Ｘ株式会社 Robot à déplacement divertissant et structure associée
KR102299563B1 (ko) * 2020-11-23 2021-09-08 주식회사 보인정보기술 가상 조교 로봇을 이용한 비대면 수업 제공 방법 및 장치
JP7121848B1 (ja) * 2021-11-12 2022-08-18 株式会社ユカリア 情報処理装置、情報処理方法、情報処理プログラムおよび情報処理システム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017049427A (ja) * 2015-09-01 2017-03-09 カシオ計算機株式会社 対話制御装置、対話制御方法及びプログラム
WO2019054464A1 (fr) * 2017-09-15 2019-03-21 Ｇｒｏｏｖｅ Ｘ株式会社 Robot à déplacement divertissant et structure associée
KR102299563B1 (ko) * 2020-11-23 2021-09-08 주식회사 보인정보기술 가상 조교 로봇을 이용한 비대면 수업 제공 방법 및 장치
JP7121848B1 (ja) * 2021-11-12 2022-08-18 株式会社ユカリア 情報処理装置、情報処理方法、情報処理プログラムおよび情報処理システム

Also Published As

Publication number Publication date
CN121127857A (zh) 2025-12-12

Similar Documents

Publication Publication Date Title
JP2024166138A (ja) 行動制御システム
WO2024219336A1 (fr) Système de commande d'action et de robot
JP2025001586A (ja) 行動制御システム
JP2024166165A (ja) 行動制御システム
JP2024166143A (ja) 行動制御システム
JP2025001572A (ja) 行動制御システム
JP2024167086A (ja) 行動制御システム
JP2025001510A (ja) 行動制御システム
JP2025081111A (ja) 制御システム
JP2025022835A (ja) 行動制御システム
JP2025028012A (ja) 行動制御システム
JP2024180168A (ja) 行動制御システム
JP2024163888A (ja) 行動制御システム
WO2025028459A1 (fr) Système de commande d'action
JP2024163069A (ja) 行動制御システム
JP2025000504A (ja) 制御システム
JP2024164826A (ja) 行動制御システム
JP2024163100A (ja) 行動制御システム
JP2025001580A (ja) 制御システム
JP2024163885A (ja) 行動制御システム
JP2024163891A (ja) 行動制御システム
JP2024180165A (ja) 行動制御システム
JP2025022854A (ja) 行動制御システム
JP2025001587A (ja) 情報処理システム
JP2025022853A (ja) 行動制御システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24792611

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE