JP2023042999A - Operation control device, operation control method and program - Google Patents

Abstract

To secure safety of execution of brainwave command by externally notifying the state of sending or reception of the command when a user executes the sending or reception of the command.SOLUTION: An operation control device is provided, comprising: a brain information acquisition unit acquiring brain information of a user; a determination unit determining whether or not the brain information acquired by the brain information acquisition unit is brain information which corresponds to execution command signifying operation content for operating operation target; an execution unit executing processing which corresponds to execution command when the determination unit determines that the brain information acquisition unit acquires brain information which corresponds to the execution command; and a notification unit which when transmitting a first processing signal which corresponds to the execution command to an operation target or when receiving a second processing signal from the operation target, notifies to surroundings of at least a user and an operation target that the execution unit is processing the first or second processing signal.SELECTED DRAWING: Figure 1

Description

The present invention relates to an operation control device, an operation control method and a program.

In recent years, techniques for measuring brain information have been developed, and a brain-machine interface (BMI), which is an interface that directly connects the brain and a machine, has become a reality. As such a technique, for example, there is one described in Patent Document 1. US Pat. No. 6,200,000 obtains a plurality of steady-state visual evoked response potential signals from a user when the user is stimulated to provide system command signals.

JP 2013-117957 A

In the conventional BMI, when the user thinks "forward" to the robot, a forward command for the robot to move forward is recalled, and when the forward command is detected, the robot moves forward. At this time, a person near the robot does not know how the user intends to operate the robot or how the robot operates. Therefore, there is a risk that the robot will operate without notice and come into contact with a nearby person. Also, if the user is spoken to while remembering a command, the user may remember the wrong command.

The present invention has been made in view of the above, and an object of the present invention is to ensure the safety of execution of electroencephalogram commands by informing the outside of the state when a user is executing command transmission or reception. and

In order to solve the above-described problems and achieve the object, an operation control device according to the present invention includes a brain information acquisition unit that acquires brain information of a user; a determination unit that determines whether or not the brain information corresponds to an execution command that means operation content for operating the an execution unit that executes processing corresponding to an execution command; and a first processing signal corresponding to the execution command being transmitted to the operation object or receiving a second processing signal from the operation object. and a notification unit that notifies a surrounding of at least one of the user and the operation target that the first processed signal or the second processed signal is being processed.

An operation control method according to the present invention includes a step of acquiring brain information of a user, and determining whether or not the acquired brain information is brain information corresponding to an execution command indicating operation content for operating an operation target. executing a process corresponding to the execution command when it is determined that the brain information corresponding to the execution command has been acquired; and sending a first processed signal corresponding to the execution command to the operation object. When transmitting or receiving a second processed signal from the operation object, the user and the operation object are notified that the first processing signal or the second processing signal is being processed. and a step of notifying toward at least one of the surroundings.

A program according to the present invention includes a step of acquiring brain information of a user, and determining whether or not the acquired brain information is brain information corresponding to an execution command indicating operation content for operating an operation target. executing a process corresponding to the execution command when it is determined that the brain information corresponding to the execution command has been acquired; and transmitting a first processing signal corresponding to the execution command to the operation object. or receiving a second processed signal from the operation object, at least one of the user and the operation object indicates that the first processed signal or the second processed signal is being processed. and a step of informing the surroundings of the computer.

Advantageous Effects of Invention According to the present invention, it is possible to ensure the safety of execution of an electroencephalogram command by informing the outside of the state when the user is executing command transmission or reception.

FIG. 1 is a block diagram showing a configuration example of an operation control device according to the first embodiment. FIG. 2 is a flow chart showing the flow of transmission processing in the operation control device according to the first embodiment. FIG. 3 is a flow chart showing the flow of reception processing in the operation control device according to the first embodiment. FIG. 4 is a graph showing the amount of brain activity with respect to the measurement position in the brain. FIG. 5 is a graph showing temporal changes in electroencephalograms at specific measurement locations in the brain. FIG. 6 is a block diagram showing a configuration example of an operation control device according to the second embodiment. FIG. 7 is a flow chart showing the flow of transmission processing in the operation control device according to the second embodiment. FIG. 8 is a flow chart showing the flow of reception processing in the operation control device according to the second embodiment.

Embodiments of an operation control device, an operation control method, and a program according to the present invention will be described in detail below with reference to the accompanying drawings. In addition, the present invention is not limited by the following embodiments.

<First embodiment>
[Operation control device]
FIG. 1 is a block diagram showing a configuration example of an operation control device according to the first embodiment.

In the first embodiment, as shown in FIG. 1, the operation control device 10 controls the operation of the operation object 50 based on brain information of the user. The operation control device 10 includes an electroencephalogram acquisition unit (brain information acquisition unit) 11, an electroencephalogram decoder 12, a start command buffer 13, an execution command buffer 14, a storage unit 15, a processing unit 16, and a display control unit 17. , a display unit 18 , a communication unit 19 , and a first notification unit 31 .

The operation target 50 is, for example, a robot, but is not limited to a robot, and may be any device or the like that operates by communication. Although not shown, the operation target 50 is provided with a driving device, a control device for controlling the driving device, and a communication unit capable of receiving control signals and the like. When the communication unit of the operation target 50 receives the control signal, the control device drives the driving device to operate the robot.

The electroencephalogram acquisition unit 11 can be worn on the user's head. The electroencephalogram decoder 12, the processing unit 16, the display control unit 17, the display unit 18, and the communication unit 19 may be worn by the user, or may be placed at a predetermined position without being worn. The electroencephalogram acquisition unit 11, the electroencephalogram decoder 12, the processing unit 16, the display control unit 17, the display unit 18, and the communication unit 19 may be provided integrally or separately.

The electroencephalogram acquisition unit 11 acquires an electroencephalogram, which is brain information of the user. The electroencephalogram acquisition unit 11 has, for example, an electrical sensor (for example, electrodes) that detects electroencephalograms emitted from weak currents flowing in the neural network of the brain. The electroencephalogram acquisition unit 11 detects a weak current potential (electrical signal) when the user is externally stimulated or based on the user's thoughts such as thoughts. Note that the brain information acquisition unit is not limited to the electroencephalogram acquisition unit 11 . The brain information acquisition unit may acquire, for example, the blood flow volume due to brain activity, which is the brain information of the user, by, for example, near-infrared light measurement. The electroencephalogram acquisition unit 11 is connected to the electroencephalogram decoder 12 and transmits electroencephalograms, which are brain information acquired from the user, to the electroencephalogram decoder 12 .

The electroencephalogram decoder 12 restores the electrical signal of the electroencephalogram of the user acquired by the electroencephalogram acquisition unit 11 to the user's thinking information. In this case, the relationship between a plurality of electrical signals of the user's electroencephalogram and the user's thinking information is linked in advance. In this case, for example, machine learning such as deep learning is used to link the relationship between the electrical signal of the electroencephalogram and the thinking information of the user.

The start command buffer 13 temporarily stores the first command determined by the determination unit 21 based on the thinking information of the user. The first command is a command for preparing to start accepting a second command (execution command) for operating the operation target 50. For example, commands such as "start robot operation" and "start input of execution command" be. The start command buffer 13 can transmit and receive various data to and from the processing unit 16 .

The execution command buffer 14 temporarily stores the second command determined by the determination unit 21 based on the thinking information of the user. The second command is an execution command for actuating the operation object 50, and is, for example, a command such as "forward", "backward", "stop", "accelerate", "decelerate", "turn right", "turn left". . In this case, the relationship between the second command and the motion of the operation target 50 is linked in advance. The execution command buffer 14 can transmit and receive various data to and from the processing unit 16 .

It should be noted that the second command is not limited to the concept indicating the action to be instructed, and may be a value or the like that complements the concept indicating the action to be instructed. For example, the electroencephalogram decoder 103 restores the user's thought information based on the electrical signal acquired by the electroencephalogram acquisition unit 11, detects the intensity of the electroencephalogram as a decimal number from 0 to 1, and converts the intensity (numerical value) into the user's thought information. ) may be added to set the second command. In this case, the numerical value of the second command is the count of the maximum value of the torque output of the driving device provided on the manipulation object 50 .

Also, the first command (start command) and the second command (execution command) may be linked and set. That is, the first command may set one common command that is common to different second commands. For example, the first command may be "start" or "start operation", while the second command is "forward" or "backward". Alternatively, a plurality of different individual first commands may be set for each different second command. For example, for the second commands such as "forward" and "accelerate", the first command is "start running", and for the second commands such as "stretch" and "grab", the first command is "Start arm operation". In addition, by setting individual first commands for each of the second commands "forward", "backward", "stop", "accelerate", "decelerate", "turn right", and "turn left", 2 commands can be avoided.

The storage unit 15 stores a plurality of first commands (start commands) and a plurality of second commands (execution commands). In addition, the storage unit 15 may store learning data obtained by machine-learning the relationship between the electric signal of the user's electroencephalogram and thought information. The second command is associated in advance with the first command.

The start command buffer 13 and the execution command buffer 14 are recording units such as semiconductor memory elements such as RAM (Random Access Memory) and flash memory provided in the processing unit 16, for example. The storage unit 15 is configured by a memory card, an SSD (Solid State Drive), an external storage device, or the like.

The processing unit 16 is an arithmetic processing device (control device) including, for example, a CPU (Central Processing Unit). The processing unit 16 loads the stored program into memory and executes the instructions contained in the program. The processing unit 16 includes an internal memory (not shown), and the internal memory is used for temporary storage of data in the processing unit 16 and the like. The processing unit 16 has a determination unit 21, an execution unit 22, and a reception unit 23 as functions. Also, the processing unit 16 transmits the processing result to the display control unit 17 and the communication unit 19 .

The determination unit 21 determines whether or not the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to two different first commands and second commands.

Specifically, the determination unit 21 compares the user's thinking information restored by the electroencephalogram decoder 12 with the first command (start command) stored in the storage unit 15 . At this time, when the user's thought information matches the first command, the determination unit 21 determines that the user's thought information is the first command. store in On the other hand, when the user's thought information does not match the first command, the determination unit 21 determines that the user's thought information is not the first command. At this time, the determination unit 21 does not store the command determined as not the first command in the start command buffer 13 . The judgment by the judging unit 21 of whether the user's thinking information restored by the electroencephalogram decoder 12 matches the first command can be made even if the user's thinking information does not completely match as long as it can be judged that the user's thinking information means the first command. Judged as a match.

The determination unit 21 also compares the user's thinking information restored by the electroencephalogram decoder 12 with the second command (execution command) stored in the storage unit 15 . At this time, when the thought information of the user matches the second command, the determination unit 21 determines that the thought information of the user is the second command. When determining that the thought information of the user is the second command, the determination unit 21 stores the second command in the execution command buffer 14 . On the other hand, when the user's thought information does not match the second command stored in the storage unit 15, the determination unit 21 determines that the user's thought information is not the second command. The determination unit 21 does not store the second command in the execution command buffer 14 when determining that the thought information of the user is not the second command.

At this time, the determination unit 21 determines a first matching rate in which a matching rate between the user's thinking information restored by the electroencephalogram decoder 12 and the first command determination value (start command determination value) stored in the storage unit 15 is set in advance. When the ratio (for example, 80%) or higher, it is determined that the user's thought information is the brain information corresponding to the first command. On the other hand, the determination unit 21 sets a second match rate, which is a preset match rate between the user's thought information restored by the electroencephalogram decoder 12 and the second command determination value (execution command determination value) stored in the storage unit 15. (eg, 90%) or more, it is determined that the user's thought information is brain information corresponding to the first command. Here, the second matching rate is set to a higher matching rate than the first matching rate.

FIG. 4 is a graph showing the amount of brain activity with respect to measurement positions in the brain, and FIG. 5 is a graph showing temporal changes in electroencephalograms at specific measurement positions in the brain.

FIG. 4 represents brain activity at different measurement locations in the brain. By comparing the amount of brain activity of the user at each measurement position with a predetermined judgment value for the amount of brain activity, a first matching rate between the user's thought information and the first command, or a user's thought information and the second A second matching rate with the command is obtained.

FIG. 5 shows temporal changes in electroencephalogram levels at specific measurement positions. By comparing the temporal change in the electroencephalogram at the predetermined time ta with a preset electroencephalogram determination value (bold line in FIG. 5), the first matching rate between the user's thinking information and the first command, or the user's thinking A second match rate between the information and the second command is determined.

By comparing the brain activity amount of the user and the brain activity amount judgment value at each measurement position, and by comparing the temporal change of the brain wave and the brain wave judgment value, the user's thought information and the first command can be compared. 1 matching rate, or user's thinking information and a second matching rate may be obtained.

Further, when the first command (start command) and the second command (execution command) are associated, the determination unit 21 determines whether the first command (start command) stored in the start command buffer 13 and the execution command It is determined whether or not the second command (execution command) stored in the buffer 14 is associated.

When the determination unit 21 determines that the electroencephalogram corresponding to the first command and the electroencephalogram corresponding to the second command have been acquired, the execution unit 22 executes processing corresponding to the second command. Specifically, the execution unit 22 determines that the user's thought information acquired first by the determination unit 21 is the first command, and determines that the user's thought information acquired next by the determination unit 21 is the second command. If determined, the process corresponding to the second command is executed.

However, the execution unit 22 cancels the acquired first command when a preset standby time, such as three seconds, elapses after the determination unit 21 determines acquisition of the first command. Further, when the processing unit 16 acquires a cancel command as the user's thought after the determination unit 21 determines acquisition of the first command, the execution unit 22 cancels the acquired first command. The cancel command is set in advance, and the relationship between the electric signal of the user's electroencephalogram and the user's thought information (cancel command) is linked in advance and stored in the storage unit 15 . The cancel command is, for example, a symbol or concept different from the first command, such as "x" or "cancel".

The receiving unit 23 can receive a processed signal from the operation target 50 . That is, the execution unit 22 transmits a first processing signal corresponding to the second command (execution command) to the operation object 50 through the communication unit 19 in order to execute processing corresponding to the second command. At this time, for example, when the operation object 50 cannot execute the process corresponding to the second command, the operation object 50 processes the second processing signal indicating that the operation object 50 cannot execute the process corresponding to the second command through the communication unit 19. 16. The receiving unit 23 receives the second processed signal from the operation target 50 .

For example, when the second command is "move forward", if a wall is in front of the manipulation object 50, the manipulation object 50 cannot "move forward" because of the wall. At this time, the operation target 50 transmits to the processing unit 16 a second processing signal indicating that the operation target 50 cannot move forward, and the receiving unit 23 receives the second processing signal. Note that when there is a removable obstacle in front of the operation object 50, the operation object 50 may transmit a second processing signal to the processing unit 16 requesting that the obstacle be removed. The same is true when the second command is "retreat", "turn right", "turn left", and the like. Furthermore, when the operation target 50 cannot operate due to insufficient battery charge, the operation target 50 may transmit a second processing signal to the processing unit 16 to charge the battery.

The second processed signal received by the receiving unit 23 may be converted into an electric brain wave signal by an electroencephalogram encoder (not shown) and transmitted to the user as thought information via an electroencephalogram reproduction unit (not shown). It should be noted that the presentation method for presenting various information as the second processed signal received by the receiving unit 23 to the user is not limited to the presentation of electroencephalograms, and may be displayed on the display unit 18, for example.

The display control section 17 is connected to the display section 18 . The display control unit 17 transmits the processing result transmitted from the processing unit 16 to the display unit 18 for display. The display control unit 17 causes the display unit 18 to display the operation state of the operation control device 10 and the thinking instructions to the user, which are the processing results of the processing unit 16 .

The display unit 18 displays the operating state of the operation control device 10 transmitted from the display control unit 17 and the thinking instruction to the user. The display unit 18 presents necessary information to the user. The display unit 18 is a display including, for example, a liquid crystal display (LCD) or an organic EL (organic electro-luminescence) display.

The communication unit 19 is capable of wireless communication between the processing unit 16 and the operation target 50 . The communication unit 19 transmits the processing information processed by the processing unit 16 to the operation target 50 . Specifically, when the determination unit 21 determines that the electroencephalogram corresponding to the first command and the electroencephalogram corresponding to the second command have been acquired, the execution unit 22 outputs a control signal (first process signal) is generated and sent to the communication unit 19 . The communication unit 19 transmits a control signal corresponding to processing of the second command to the operation target 50 . By receiving the second command transmitted by the communication unit 19, the operation target 50 operates corresponding to the processing of the second command.

The communication unit 19 also transmits processing information (second processing signal) indicating the processing status of the operation target 50 to the processing unit 16 . Specifically, when the operation object 50 cannot execute the processing corresponding to the second command, the operation object 50 outputs a control signal (second processing signal) corresponding to the content of the inability to execute the processing of the second command. is generated and sent to the communication unit 19 . The communication unit 19 transmits to the processing unit 16 a control signal corresponding to the content that the processing of the second command cannot be executed. The receiving unit 23 receives a control signal corresponding to the content that the processing of the second command transmitted by the communication unit 19 cannot be executed.

The first notification unit 31 notifies the outside that the processing unit 16 is processing. Specifically, when the processing unit 16 is transmitting the first processing signal corresponding to the second command (execution command) to the operation object 50, the first notification unit 31 is processing the first processing signal. The user and the surroundings of the operation target 50 are notified that the Further, when the processing unit 16 is receiving the second processed signal from the operation target 50, the first notification unit 31 notifies the user or the surroundings of the operation target 50 that the second processing signal is being processed. report to. Further, after the determination unit 21 has determined the first command (start command), the first notification unit 31 may notify the user or the surroundings of the operation object 50 that each processing signal is being processed. good. Note that the first notification unit 31 only needs to be able to notify the user and the surroundings of the operation target 50 that at least one of the first processed signal and the second processed signal is being processed. Further, the first notification unit 31 may notify the surroundings of at least one of the user and the operation target 50 that the first processing signal and the second processing signal are being processed.

The first notification unit 31 is a display including, for example, a liquid crystal display or an organic EL display. In this case, the first notification section 31 may also serve as the display section 18 . Further, the first notification unit 31 may be a visual expression device such as illumination, or an auditory sound device such as a speaker, other than the display. The first notification unit 31 may be attached to the user or the operation target 50 or may be arranged around the user or the operation target 50 . In this case, the first notification unit 31 is set when the processing unit 16 transmits the first processing signal corresponding to the second command to the operation object 50 and when the processing unit 16 receives the second processing signal from the operation object 50 . The method of informing the surroundings of the user and the operation target 50 may be different when the signal is being received.

[Operation control method]
Here, the operation control method by the operation control device 10 will be described. FIG. 2 is a flow chart showing the flow of transmission processing in the operation control device according to the first embodiment.

As shown in FIGS. 1 and 2, in step S11, the processing unit 16 controls the display control unit 17 to instruct the user to recall the first command (start command) on the display unit 18. indicate. The display control unit 17 displays, on the display unit 18, a message for instructing the recall of the first command, which means the content of the operation, such as "Please recall the start command."

In step S12, the determination unit 21 determines whether or not the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thinking information restored by the electroencephalogram decoder 12 with the first command stored in the storage unit 15, and the user's thinking information does not match the first command. is less than the first matching rate (for example, 80%) (No), the user's thinking information is not the first command, and this routine exits. On the other hand, the determination unit 21 determines that the thought information of the user matches the first command stored in the storage unit 15, that is, that the matching rate between the two is equal to or higher than the first matching rate (for example, 80%) (Yes ) Then, assuming that the thought information of the user is the first command, the determination unit 21 stores the first command in the start command buffer 13, and proceeds to step S13.

In step S13, the first notification unit 31 notifies the outside that the processing unit 16 is processing. Specifically, the first notification unit 31 determines that the thinking information of the user is the first command (start command), and when the second command (execution command) is being accepted, the user or the operation Various kinds of information are notified around the object 50 . Here, for example, the processing unit 16 notifies the user and the operation target 50 that the user is recalling the second command (execution command) by the first notification unit 31 . The first notification unit 31 notifies, for example, a message indicating the content of processing, such as "Recalling execution command". At this time, people around the user can determine the state of the user by means of the first notification unit 31, so that the user can easily recall the execution command.

In step S<b>14 , the processing unit 16 controls the display control unit 17 to display an instruction to remind the user of the second command (execution command) on the display unit 18 . The display control unit 17 displays, on the display unit 18, a message for instructing to recall a second command, which means operation content, such as "Please recall an execution command."

In step S<b>15 , the processing unit 16 determines whether or not a timeout has occurred after a preset standby time has elapsed since the determination unit 21 determined acquisition of the first command. When determining that the timeout has occurred (Yes), the processing unit 16 cancels the acquired first command in step S22. Then, in step S<b>23 , the processing unit 16 controls the display control unit 17 to display to the user that the first command has been canceled on the display unit 18 . The display control unit 17 displays, for example, a timed-out message such as “Timed out” on the display unit 18 .

If it is determined that the timeout has not occurred (No), the processing unit 16 determines in step S16 whether or not an electroencephalogram corresponding to the cancel command has been detected. When determining that an electroencephalogram corresponding to the cancel command has been detected (Yes), the processing unit 16 cancels the acquired first command in step S24. Then, in step S25, the processing unit 16 controls the display control unit 17 to display to the user on the display unit 18 that the first command has been canceled. The display control unit 17 displays, on the display unit 18, a message indicating that the first command has been canceled, such as "Cancel the first command."

On the other hand, when the processing unit 16 determines that no electroencephalogram corresponding to the cancel command is detected (No), in step S17, the determination unit 21 determines whether or not the electroencephalogram corresponding to the second command (execution command) has been detected. judge. Here, the determination unit 21 compares the user's thinking information restored by the electroencephalogram decoder 12 with the second command stored in the storage unit 15, and the user's thinking information does not match the second command. is less than the second matching rate (for example, 90%) (No), the process returns to step S15 to continue the process.

On the other hand, the determination unit 21 determines that the thought information of the user matches the second command stored in the storage unit 15, that is, that the matching rate between the two is equal to or higher than the second matching rate (for example, 90%) (Yes ) Then, the determined second command is stored in the execution command buffer 14 assuming that the thought information of the user is the second command. Further, in step S18, the determination unit 21 determines in advance the first command determined in step S12 and stored in the start command buffer 13 and the second command determined in step S17 and stored in the execution command buffer 14. It is determined whether or not the first command and the second command are correlated. When determining that the first command and the second command are not associated (No), the determination unit 21 returns to step S15. When determining that the first command and the second command are associated (Yes), the determination unit 21 proceeds to step S19.

In step S19, the execution unit 22 executes processing corresponding to the second command. That is, the execution unit 22 transmits a control signal corresponding to the processing of the second command to the operation object 50 through the communication unit 19 to operate the operation object 50 . Then, in step S<b>20 , the processing unit 16 controls the display control unit 17 to display to the user that the second command has been executed on the display unit 18 . The display control unit 17 displays, on the display unit 18, a message indicating that the first command has been executed, such as "acceleration command executed". Further, the processing unit 16 notifies the user or the operation object 50 that the operation object 50 executes the first command (execution command) by the first notification unit 31, for example. The first notification unit 31 notifies, for example, a message indicating the content of processing, such as "The robot will operate." Then, in step S21, the first notification unit 31 terminates the notification to the outside.

FIG. 3 is a flow chart showing the flow of reception processing in the operation control device according to the first embodiment.

As shown in FIGS. 1 and 3, in step S31, the processing unit 16 controls the display control unit 17 to instruct the user to recall the first command (start command) on the display unit 18. indicate. The display control unit 17 displays, on the display unit 18, a message for instructing the recall of the first command, which means the content of the operation, such as "Please recall the start command."

In step S32, the determination unit 21 determines whether or not the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thinking information restored by the electroencephalogram decoder 12 with the first command stored in the storage unit 15, and the user's thinking information does not match the first command. is less than the first matching rate (for example, 80%) (No), the user's thinking information is not the first command, and this routine exits. On the other hand, the determination unit 21 determines that the thought information of the user matches the first command stored in the storage unit 15, that is, that the matching rate between the two is equal to or higher than the first matching rate (for example, 80%) (Yes ) Then, assuming that the thought information of the user is the first command, the determination unit 21 stores the first command in the start command buffer 13, and proceeds to step S33.

In step S<b>33 , the determination unit 21 determines whether or not a reception command has been received from the operation target 50 . Here, for example, the determination unit 21 compares the received command with the received command stored in the storage unit 15, and if it determines that the two do not match (No), exits this routine. On the other hand, the determination unit 21 compares the received command with the received command stored in the storage unit 15, and if it determines that the two match (Yes), the process proceeds to step S34.

In step S34, the first notification unit 31 notifies the outside that the processing unit 16 is processing. Specifically, when the receiving unit 23 receives a command from the operation object 50 , the first notification unit 31 notifies the user and the surroundings of the operation object 50 of various types of information. Here, for example, the processing unit 16 notifies the user or the operation object 50 that the reception command is being received from the operation object 50 by the first notification unit 31 . The first notification unit 31 notifies, for example, a message indicating the content of processing, such as "receiving from operation target 50".

Then, in step S35, the reception unit 23 receives the reception command from the operation target 50, and presents the received content to the user. In step S<b>36 , the reception unit 23 determines whether reception of the reception command from the operation target 50 has ended. Here, if the reception unit 23 determines that the reception of the reception command from the operation target 50 has not ended (No), it continues the processing of step S35. On the other hand, when the reception unit 23 determines that the reception of the reception command from the operation target 50 is finished (Yes), the first notification unit 31 finishes notification to the outside in step S37.

In the above description, when the first command (start command) and the received command are detected, the first notification unit 31 notifies the outside, but the process of step S32 may be omitted. That is, even before the detection of the first command (start command), the determination unit 21 may cause the first notification unit 31 to notify the outside when the reception command is detected.

<Second embodiment>
[Operation control device]
FIG. 6 is a block diagram showing a configuration example of an operation control device according to the second embodiment. Members having the same functions as those of the above-described first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

In the second embodiment, as shown in FIG. 6, the operation control device 10 controls the operation of the operation target 50 based on brain information of the user. The operation control device 10 includes an electroencephalogram acquisition unit (brain information acquisition unit) 11, an electroencephalogram decoder 12, a start command buffer 13, an execution command buffer 14, a storage unit 15, a processing unit 16, and a display control unit 17. , a display unit 18 , a communication unit 19 , a first notification unit 31 , and a second notification unit 32 .

The electroencephalogram acquisition unit 11, the electroencephalogram decoder 12, the start command buffer 13, the execution command buffer 14, the storage unit 15, the processing unit 16, the display control unit 17, the display unit 18, the communication unit 19, and the first notification unit 31 are used for the first implementation. Since it is the same as the form, the explanation is omitted.

The second notification unit 32 notifies the user that the processing unit 16 is processing. Specifically, when the processing unit 16 is transmitting the first processing signal corresponding to the second command (execution command) to the operation object 50, the second notification unit 32 is processing the first processing signal. The user is notified that Further, the second notification unit 32 notifies the user that the second processing signal is being processed while the processing unit 16 is receiving the second processing signal from the operation target 50 . Further, the second notification unit 32 may notify the user that each processing signal is being processed after the determination unit 21 has determined the first command (start command). The second notification unit 32 only needs to be able to notify the user that at least one of the first processed signal and the second processed signal is being processed.

The second notification unit 32 is a display including, for example, a liquid crystal display or an organic EL display. In this case, the second notification section 32 may also serve as the display section 18 . Further, the second notification unit 32 may be a visual expression device such as illumination, or an auditory sound device such as a speaker, headphones, or earphones, other than the display. Furthermore, the second notification unit 32 may be notified by the user's tactile sensation. The second notification unit 32 can be visually, auditory, or tactilely confirmed by the user, and may be worn by the user or placed in the vicinity of the user. In this case, the second notification unit 32 is set when the processing unit 16 transmits the first processing signal corresponding to the second command to the operation object 50 and when the processing unit 16 receives the second processing signal from the operation object 50 . The method of informing the surroundings of the user and the operation target 50 may be different when the signal is being received.

[Operation control method]
Here, the operation control method by the operation control device 10 will be described. FIG. 7 is a flow chart showing the flow of transmission processing in the operation control device according to the second embodiment. Note that the operation of the first notification unit 31 is the same as in the first embodiment, and the description is omitted.

As shown in FIGS. 6 and 7, in step S41, the processing unit 16 controls the display control unit 17 to instruct the user to recall the first command (start command) on the display unit 18. indicate. The display control unit 17 displays, on the display unit 18, a message for instructing the recall of the first command, which means the content of the operation, such as "Please recall the start command."

In step S42, the determination unit 21 determines whether or not the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thinking information restored by the electroencephalogram decoder 12 with the first command stored in the storage unit 15, and the user's thinking information does not match the first command. is less than the first matching rate (for example, 80%) (No), the user's thinking information is not the first command, and this routine exits. On the other hand, the determination unit 21 determines that the thought information of the user matches the first command stored in the storage unit 15, that is, that the matching rate between the two is equal to or higher than the first matching rate (for example, 80%) (Yes ) Then, assuming that the thought information of the user is the first command, the determination unit 21 stores the first command in the start command buffer 13, and proceeds to step S13.

In step S43, the second notification unit 32 notifies the inside that the processing unit 16 is processing. Specifically, when the determination unit 21 determines that the thinking information of the user is the first command (start command) and is accepting the second command (execution command), the second notification unit 32 notifies the user of various to notify the information of Here, for example, the processing unit 16 notifies the user that the recall of the second command (execution command) from the user is being received by the second notification unit 32 . The second notification unit 32 notifies, for example, a message indicating the content of processing, such as "receiving recall of execution command". At this time, the second notification unit 32 may restrict the user's vision by darkening the surroundings or darkening the display. In addition, the second notification unit 32 may restrict the user's hearing by making it difficult to hear surrounding sounds or by reducing the volume of the speaker. The second notification unit 32 restricts the user's sense of sight and hearing, making it easier for the user to remember the execution command.

In step S<b>44 , the processing unit 16 controls the display control unit 17 to display an instruction to remind the user of the second command (execution command) on the display unit 18 . The display control unit 17 displays, on the display unit 18, a message for instructing to recall a second command, which means operation content, such as "Please recall an execution command."

In step S<b>45 , the processing unit 16 determines whether or not a timeout has occurred after a preset standby time has elapsed since the determination unit 21 determined acquisition of the first command. When determining that the timeout has occurred (Yes), the processing unit 16 cancels the acquired first command in step S52. Then, in step S<b>53 , the processing unit 16 controls the display control unit 17 to display to the user that the first command has been canceled on the display unit 18 . The display control unit 17 displays, for example, a timed-out message such as “Timed out” on the display unit 18 .

If it is determined that the timeout has not occurred (No), the processing unit 16 determines in step S46 whether or not an electroencephalogram corresponding to the cancel command has been detected. When determining that an electroencephalogram corresponding to the cancel command has been detected (Yes), the processing unit 16 cancels the acquired first command in step S54. Then, in step S<b>55 , the processing unit 16 controls the display control unit 17 to display to the user that the first command has been canceled on the display unit 18 . The display control unit 17 displays, on the display unit 18, a message indicating that the first command has been canceled, such as "Cancel the first command."

On the other hand, when the processing unit 16 determines that no electroencephalogram corresponding to the cancel command is detected (No), in step S47, the determination unit 21 determines whether or not the electroencephalogram corresponding to the second command (execution command) has been detected. judge. Here, the determination unit 21 compares the user's thinking information restored by the electroencephalogram decoder 12 with the second command stored in the storage unit 15, and the user's thinking information does not match the second command. is less than the second matching rate (for example, 90%) (No), the process returns to step S45 to continue the process.

On the other hand, the determination unit 21 determines that the thought information of the user matches the second command stored in the storage unit 15, that is, that the matching rate between the two is equal to or higher than the second matching rate (for example, 90%) (Yes ) Then, the determined second command is stored in the execution command buffer 14 assuming that the thought information of the user is the second command. Further, in step S48, the determination unit 21 determines in advance the first command determined in step S42 and stored in the start command buffer 13 and the second command determined in step S47 and stored in the execution command buffer 14. It is determined whether or not the first command and the second command are correlated. When determining that the first command and the second command are not associated (No), the determination unit 21 returns to step S45. When determining that the first command and the second command are associated (Yes), the determination unit 21 proceeds to step S49.

In step S49, the execution unit 22 executes processing corresponding to the second command. That is, the execution unit 22 transmits a control signal corresponding to the processing of the second command to the operation object 50 through the communication unit 19 to operate the operation object 50 . Then, in step S<b>50 , the processing unit 16 controls the display control unit 17 to display to the user that the second command has been executed on the display unit 18 . The display control unit 17 displays, on the display unit 18, a message indicating that the first command has been executed, such as "acceleration command executed". Further, the processing unit 16 notifies the user or the operation object 50 by the operation object 50 executing the first command (execution command) by the second notification unit 32, for example. The second notification unit 32 notifies, for example, a message indicating the content of processing, such as "The robot will operate." Then, in step S51, the second notification unit 32 terminates the notification to the inside.

FIG. 8 is a flow chart showing the flow of reception processing in the operation control device according to the second embodiment.

As shown in FIGS. 6 and 8, in step S61, the processing unit 16 controls the display control unit 17 to instruct the user to recall the first command (start command) on the display unit 18. indicate. The display control unit 17 displays, on the display unit 18, a message for instructing the recall of the first command, which means the content of the operation, such as "Please recall the start command."

In step S62, the determination unit 21 determines whether or not the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thinking information restored by the electroencephalogram decoder 12 with the first command stored in the storage unit 15, and the user's thinking information does not match the first command. is less than the first matching rate (for example, 80%) (No), the user's thinking information is not the first command, and this routine exits. On the other hand, the determination unit 21 determines that the thought information of the user matches the first command stored in the storage unit 15, that is, that the matching rate between the two is equal to or higher than the first matching rate (for example, 80%) (Yes ) Then, assuming that the thought information of the user is the first command, the determination unit 21 stores the first command in the start command buffer 13, and proceeds to step S63.

In step S<b>63 , the determination unit 21 determines whether or not a reception command has been received from the operation target 50 . Here, for example, the determination unit 21 compares the received command with the received command stored in the storage unit 15, and if it determines that the two do not match (No), exits this routine. On the other hand, the determination unit 21 compares the received command with the received command stored in the storage unit 15, and if it determines that the two match (Yes), the process proceeds to step S64.

In step S64, the second notification unit 32 internally notifies that the processing unit 16 is processing. Specifically, when the reception unit 23 receives the reception command from the operation target 50, the second notification unit 32 notifies the user of various types of information. Here, for example, the processing unit 16 notifies the user that the reception command is being received from the operation object 50 by the second notification unit 32 . The second notification unit 32 notifies, for example, a message indicating the content of processing such as "receiving from operation target 50".

Then, in step S65, the reception unit 23 receives the reception command from the operation object 50, and in step S66, the reception unit 23 determines whether reception of the reception command from the operation object 50 is finished. do. Here, if the reception unit 23 determines that the reception of the reception command from the operation target 50 has not ended (No), it continues the processing of step S65. On the other hand, when the reception unit 23 determines that the reception of the reception command from the operation target 50 is finished (Yes), the second notification unit 32 finishes notification to the inside in step S67.

[effect]
In this embodiment, an electroencephalogram acquisition unit (brain information acquisition unit) 11 that acquires brain information of a user, and an execution command that indicates the operation content for operating the operation target 50 based on the brain information acquired by the electroencephalogram acquisition unit 11 . and an execution unit 22 that executes a process corresponding to the execution command when the determination unit 21 determines that the brain information corresponding to the execution command is obtained. , when transmitting the first processing signal corresponding to the execution command to the operation object 50 or receiving the second processing signal from the operation object 50, the first processing signal or the second processing signal and a first notification unit 31 that notifies the surroundings of at least one of the user and the operation target 50 that processing is being performed.

Therefore, the state of recollection of the user and the operating state of the robot can be notified to the surroundings of the user and the operation target 50 according to the notification contents of the first notification unit 31, and the surrounding people can determine the state of the user. Also, the operation of the robot can be predicted, and the user's recollection of wrong commands due to a nearby person touching the robot or speaking to the user is suppressed. As a result, it is possible to ensure the safety of the execution of the electroencephalogram command by informing the outside of the state when the user is executing command transmission or reception.

Further, in the present embodiment, the determination unit 21 determines whether the brain information acquired by the electroencephalogram acquisition unit 11 is brain information corresponding to a start command that means that an execution command can be accepted, and the first notification unit 31 notifies that processing is in progress when the determination unit 21 determines that the brain information corresponding to the start command has been acquired. As a result, the user has enough thinking time from when he/she remembers the start command to when he/she remembers the execution command, and can remember the appropriate execution command without fail.

Further, in the present embodiment, the determination unit 21 determines the start command when the match rate between the brain information acquired by the electroencephalogram acquisition unit 11 and the brain information of the start command determination value is equal to or higher than a preset first match rate. When the matching rate between the brain information acquired by the electroencephalogram acquisition unit 11 and the brain information of the execution command determination value is equal to or higher than the second matching rate higher than the first matching rate, the execution command is determined to be the corresponding brain information. is determined to be brain information corresponding to . Therefore, by setting the determination degree of the matching rate of the start command to a low level, individual differences in the determination of the start command can be eliminated, and the start command can be shared, thereby simplifying the program.

In the above-described embodiment, when the brain information acquired by the electroencephalogram acquisition unit 11 is determined to be the brain information corresponding to the start command, acceptance of the execution command is started. Acceptance of the execution command may be terminated when it is determined that the acquired brain information corresponds to the end command.

Although the operation control device 10 according to the present invention has been described so far, it may be implemented in various different forms other than the embodiments described above.

Each component of the illustrated operation control device 10 is functionally conceptual and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to the illustrated one, and all or part of it may be functionally or physically distributed or integrated in arbitrary units according to the processing load and usage conditions of each device. may

The configuration of the operation control device 10 is realized by, for example, a program loaded in a memory as software. In the above embodiments, functional blocks realized by cooperation of these hardware or software have been described. That is, these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

The components described above include those that can be easily assumed by those skilled in the art and those that are substantially the same. Furthermore, the above configurations can be combined as appropriate. Also, various omissions, substitutions, or modifications of the configuration are possible without departing from the gist of the present invention.

10 operation control device 11 electroencephalogram acquisition unit (brain information acquisition unit)
REFERENCE SIGNS LIST 12 electroencephalogram decoder 13 start command buffer 14 execution command buffer 15 storage unit 16 processing unit 17 display control unit 18 display unit 19 communication unit 21 determination unit 22 execution unit 23 reception unit 31 first notification unit (notification unit)
32 second notification unit 50 operation target object

Claims (5)

a brain information acquisition unit that acquires brain information of a user;
a determination unit that determines whether or not the brain information acquired by the brain information acquisition unit is brain information corresponding to an execution command that means an operation content for operating an operation target;
an execution unit that executes a process corresponding to the execution command when the determination unit determines that the brain information corresponding to the execution command has been acquired;
the first processing signal or the second processing when transmitting the first processing signal corresponding to the execution command to the operation object or receiving the second processing signal from the operation object; a notification unit that notifies the surroundings of at least one of the user and the operation target that a signal is being processed;
An operation control device comprising: The determination unit determines whether or not the brain information acquired by the brain information acquisition unit is brain information corresponding to a start command indicating that the execution command can be accepted,
The notification unit notifies that processing is in progress when the determination unit determines that the brain information corresponding to the start command has been acquired.
The operation control device according to claim 1. The determination unit is
determining that the brain information corresponds to the start command when a match rate between the brain information acquired by the brain information acquisition unit and the brain information of the start command determination value is equal to or higher than a preset first match rate; ,
The brain information corresponds to the execution command when the matching rate between the brain information acquired by the brain information acquisition unit and the brain information of the execution command determination value is equal to or higher than a second matching rate higher than the first matching rate. judge,
The operation control device according to claim 2. obtaining brain information of a user;
a step of determining whether or not the acquired brain information is brain information corresponding to an execution command indicating operation details for operating an operation target;
executing a process corresponding to the execution command when determining that brain information corresponding to the execution command has been obtained;
the first processing signal or the second processing when transmitting the first processing signal corresponding to the execution command to the operation object or receiving the second processing signal from the operation object; a step of informing the surroundings of at least one of the user and the operation target that the signal is being processed;
Operation control method including; obtaining brain information of a user;
a step of determining whether or not the acquired brain information is brain information corresponding to an execution command indicating operation details for operating an operation target;
executing a process corresponding to the execution command when determining that brain information corresponding to the execution command has been obtained;
the first processing signal or the second processing when transmitting the first processing signal corresponding to the execution command to the operation object or receiving the second processing signal from the operation object; a step of informing the surroundings of at least one of the user and the operation target that the signal is being processed;
A program that makes a computer run

Images