JP2024135097A - Remote operation control system, remote control device, remote control method, and program - Google Patents

Abstract

To provide a remote operation control system by which optimal support function is presented from a difference between a person is good at operation and myself and operation can be easily performed by even a person with low proficiency, a remote control device, a remote control method, and a program.SOLUTION: A remote operation control system, by which an operator remotely controls a remote control object that is a real object or a virtual object in a virtual world, comprises: a score calculation part which calculates a remote control operation score; and a difference presentation part which presents a difference between operations of a first operator and a second operator who has an operation score higher than the first operator.SELECTED DRAWING: Figure 2

Description

The present invention relates to a remote control system, a remote control device, a remote control method, and a program.

In recent years, expectations for remote-controlled robots have been rising due to factors such as labor shortages. For this reason, development of systems that allow robots to perform tasks remotely is underway. Technologies to support such remote control are being developed (for example, see Non-Patent Document 1).

Kazunori Ohno, Naoji Shiroma, "Remote Control Support Technology for Rescue Robots", Journal of the Robotics Society of Japan, Vol 28 No2, pp160-163, 2010

However, with conventional technology, there was a problem in that different operators had different levels of proficiency, and therefore different types of remote support were required.

The present invention has been made in consideration of the above problems, and aims to provide a remote operation control system, a remote control device, a remote control method, and a program that present optimal support functions based on the differences between the user and people who are skilled at the work, making the work easier even for people with low proficiency.

(1) In order to achieve the above object, a remote operation control system according to one aspect of the present invention is a remote operation control system in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world, and the remote operation control system includes a score calculation unit that calculates an operation score of the remote operation, and a difference presentation unit that presents a difference between the operation of the first operator and an operation of a second operator who has a higher operation score than the first operator.

(2) In the remote operation control system of (1), a control unit is configured to estimate an operation intention of an operator from information about the first operator and to perform operation assistance according to the operation intention for a remote operation robot; and an assistance level change unit is configured to change a level of the operation assistance according to the task score.
The above configuration may be adopted.

(3) In the remote operation control system of (1) or (2), the score calculation unit may calculate the score by weighting and adding the time required for the task and the achievement level, which is the quality of each task.

(4) In any one of the remote operation control systems (1) to (3), the difference presentation unit may compare the operation data of the first operator with the operation data of the second operator to determine the difference, generate presentation information to be presented based on the determined difference, and present the generated presentation information.

(5) In the remote operation control system of (4), the presented information may be at least one of an image, text, and audio.

(6) In any one of the remote operation control systems (1) to (3), the difference presentation unit may compare an operation state of the first operator with an operation state of the second operator to determine a difference, and present an improvement suggestion based on the determined difference.

(7) In the remote control system of (4) or (6), the operation data of the second operator may be operation data of a typical pattern.

(8) In order to achieve the above object, a remote control device according to one embodiment of the present invention is a remote control device for a remote operation control system in which an operator remotely operates a remote operation target that is a real object or a virtual object in a virtual world, and the remote control device is equipped with a score calculation unit that calculates an operation score of the remote operation, and a difference presentation unit that presents a difference between the operation of the first operator and an operation of a second operator who has a higher operation score than the first operator.

(9) In order to achieve the above object, a remote control method according to one aspect of the present invention is a remote control method for a remote control device of a remote control system in which an operator remotely controls a remote control target, which is a real object or a virtual object in a virtual world, in which a score calculation unit calculates an operation score of the remote operation, and a difference presentation unit presents a difference between the operation of the first operator and an operation of a second operator who has a higher operation score than the first operator.

(10) In order to achieve the above object, a program according to one aspect of the present invention is a program that causes a computer of a remote control device of a remote operation control system in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world, to calculate an operation score for the remote operation and present the difference between the operation of the first operator and that of a second operator who has a higher operation score than the first operator.

According to (1) to (10), the optimal support functions are suggested based on the differences between you and someone who is skilled at the task, making the task easier even for people with low proficiency.

FIG. 1 is a diagram illustrating an example of an environment of a remote operation control system according to an embodiment. 1 is a diagram illustrating an example of the configuration of a remote operation control system according to an embodiment. FIG. 13 is a diagram for explaining an outline of the auxiliary support. 4 is a diagram showing an example of information stored in a storage unit of the operation control device according to the embodiment; FIG. 4 is a diagram showing an example of information stored in a storage unit of the operation control device according to the embodiment; FIG. 4 is a diagram showing an example of information stored in a storage unit of the operation control device according to the embodiment; FIG. FIG. 13 is a diagram showing an example of a difference image presented to an operator. 13A and 13B are diagrams illustrating other examples of difference images presented to the operator. 10 is a flowchart of an example of a processing procedure performed by a remote control device according to the embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized.
In addition, in all the drawings for explaining the embodiments, the same reference numerals are used for the parts having the same functions, and the repeated explanation is omitted.
In addition, "based on XX" in this application means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is directly used, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

(overview)
A score calculation formula is created so that people with high speed and quality get higher scores. Speed is weighted by time, and quality is weighted by the degree of achievement for each task and then added together. Images and data (hand trajectory, object state (position, posture information)) of people with superior scores are compared with those of the person with the highest score, and the system suggests how to do the task based on the differences, or a human makes a visual judgment. A typical grasping pattern and a manual pattern can be selected, and the most suitable typical pattern is suggested from multiple patterns in a database based on the user's selected pattern and work record (speed and quality). The object to be operated may be a real object, or it may be a virtual object in a virtual world.

(Example of a remote control system environment)
An example of the environment of the remote operation control system will be described.
FIG. 1 is a diagram showing an example of an environment of a remote operation control system according to this embodiment.
As shown in FIG. 1, an operator Us wears, for example, an HMD (head mounted display) as the image display unit 3, and also wears operation detection units 21 (21L, 21R) and the like.
An environmental sensor 4 is installed in the working space. The environmental sensor 4 may be attached to the robot 6. The environmental sensor 4 is, for example, an RGB (red, blue, green) D camera that can also obtain depth information D.
The robot 6 includes, for example, arms 63 (63L, 63R), end effectors 64 (64L, 64R), a body 66, and a head 67. In the following description, the end effector 64 is also referred to as a hand. The robot 6 may include only the arm 63 and the end effector 64. The robot 6 may include one or more arms. The robot 6 includes a communication unit (not shown) and transmits and receives information to and from the remote control device 5.

The operator Us remotely controls the robot 6 by moving the hand or fingers wearing the operation detection unit 21 while viewing the image displayed on the image display unit 3. In the example of FIG. 1, the operator Us remotely controls the robot 6 to grasp a plastic bottle obj on the table Tb. In remote control, the operator Us cannot directly see the movement of the robot 6, for example, but can indirectly see the image of the robot 6 on the image display unit 3. The work content includes, for example, tightening screws, attaching parts, adjustments, etc.

(Example of remote control system configuration)
2 is a diagram showing an example of the configuration of the remote operation control system according to this embodiment. As shown in FIG. 2, the remote operation control system 1 includes, for example, an operation input unit 2 (2-1, 2-2, ...), an image display unit 3 (3-1, 3-2, ...), an environment sensor 4, a remote control device 5, a robot 6 (6-1, 6-2, ...), and a DB 7.
The operation input unit 2 includes, for example, an operation detection unit 21 and a gaze detection unit 22 .
The remote control device 5 includes, for example, an acquisition unit 51 , a score calculation unit 52 , a difference generation unit 53 (difference presentation unit), a provision unit 54 (difference presentation unit), an assistance level change unit 55 , a control unit 56 , and a memory unit 57 .
The robot 6 includes, for example, a sensor 61 , an arm 63 , an end effector 64 , and a drive unit 65 .

DB7 stores, for example, three-dimensional model data of the robot 6, information for estimating the operation intention, information on the object to be operated, etc.

The operation input unit 2 includes a communication unit (not shown) and exchanges information with the remote control device 5 via wired or wireless communication.

The operation detection unit 21 detects operation input by an operator. The operation detection unit 21 is, for example, a data glove or a switch. The operation detection unit 21 outputs the detected operation input information to the remote control device 5.

The gaze detection unit 22 detects the gaze of the operator. The gaze detection unit 22 outputs the detected gaze information to the remote control device 5. Note that the gaze detection unit 22 may be provided in the image display unit 3.

The image display unit 3 is, for example, an HMD, a liquid crystal image display device, an organic EL (Electro Luminescence) image display device, a smartphone, a tablet terminal, or the like, which is capable of displaying images. The image display unit 3 displays an image provided by the remote control device 5. The image display unit 3 includes a communication unit (not shown), and exchanges information with the remote control device 5 in a wired or wireless manner.

The environmental sensor 4 is, for example, an RGBD camera that can also acquire depth information D. The environmental sensor 4 may be a distance sensor using a laser or the like and an RGB camera. There may be two or more environmental sensors 4. The environmental sensor 4 has a communication unit (not shown) and is connected to the remote control device 5 by wire or wirelessly.

The remote control device 5 controls the operation of the robot 6 based on the operation information operated by the operator so as to assist the operator in performing the work. Note that the remote control device 5 may, for example, estimate the operator's operation intention and assist the work. Note that the remote control device 5 may obtain information from multiple operation input units 2 and control the corresponding robot 6. For example, a first operator remotely operates the robot 6-1 using the operation input unit 2-1 and the image display unit 3-1, and a second operator remotely operates the robot 6-2 using the operation input unit 2-2 and the image display unit 3-2.

The acquisition unit 51 acquires operation input information and gaze information from the operation input unit 2, image information from the environmental sensor 4, sensor detection values from the robot 6, etc.

The score calculation unit 52 uses the acquired information to calculate a task score for each operator. The score calculation unit 52 calculates the task score, for example, by weighting and adding the speed (time) of the task and the degree of accomplishment (quality) of each task (each step in a task). Note that the method of calculating the task score is an example and is not limited to this. The score calculation unit 52 stores the calculated task score for each worker and task in the memory unit 57.

The difference generation unit 53 finds the operation difference between operators with a higher task score than the current operator. The difference generation unit 53 generates difference information (e.g., an image) that presents, for example, the task state of a first operator and the task state of a second operator with a task score higher than the first operator in an overlapping manner. Alternatively, the difference generation unit 53 generates difference information (e.g., an image) that presents, for example, the task state of the first operator and the task state of a typical pattern stored in the storage unit 57 in an overlapping manner. The difference generation unit 53 generates a suggestion (text or voice) to the operator based on the task score, the operation data, etc. The operation data is, for example, the trajectory of the hand, the state of an object (position, posture information), etc.

The providing unit 54 causes the difference information generated by the difference generating unit 53 to be presented on the image display unit 3.

The assistance level change unit 55 changes the level of the assistance function according to the task score.

The control unit 56 controls the robot 6 based on the acquired operation information. The control unit 56 controls the robot 6 while estimating the operator's operation intention from the operation information and assisting the robot 6 to perform an action according to the operation intention. The intention estimation method will be described later. The control unit 56 changes the assistance level according to the level changed by the assistance level change unit 55.

The memory unit 57 stores programs, thresholds, etc. used by the remote control device 5. The memory unit 57 stores a task score for each operator and for each task. The memory unit 57 stores operation data for each operator and for each task. The memory unit 57 stores operation data of a typical task pattern for each task. The memory unit 57 stores the position where the environmental sensor 4 is installed. The memory unit 57 may store, for example, three-dimensional model data of the robot 6.

The sensor 61 may be, for example, an encoder attached to a joint, a six-axis sensor attached to the fingertip of the end effector 64, a force sensor, etc.

An end effector 64 is attached to the tip of the arm 63.

The end effector 64 has at least two or more fingers. The end effector 64 may be a gripper or the like, and may have multiple fingers. Furthermore, the end effector 64 of each robot 6 may be different depending on, for example, the work content.

The drive unit 65 includes, for example, a drive circuit and an actuator. The drive unit 65 drives the arm 63 and the end effector 64 in response to control instructions (joint angle instructions, etc.) output by the remote control device 5.

(Assistant support)
Next, an outline of the auxiliary support will be described with reference to Fig. 3.
The grasping action that humans perform casually is a complex process that involves, for example, planning which object to grasp and how to hold it, reaching out and grasping it, checking to see if it is held firmly, and making corrections if it is not.
On the other hand, when trying to do the same thing by remotely controlling a robotic hand, although it is possible to create a rough hand shape, it is difficult to grasp the sense of distance when using camera footage, and there may be a discrepancy between the operator's sense of touch and the actual state of the touch.
Therefore, in remote operation, the remote control device 5 not only controls based on the operator's commands as in FIG. 3, but also supports the operation based on the operator's intention to operate, for example, to achieve smooth grasping.

The control unit 56 determines the contact points of the fingers of the robot 6 with respect to the object that can stably grasp the object without dropping it, for example, based on constraints such as the classification of the selected motion, the object shape, estimated physical parameters such as friction and weight of the object, and the torque that the robot 6 can output. The control unit 56 then assists with the correction motion, for example, using the joint angles calculated from these as target values (see, for example, JP 2022-155623 A).

(Support Level)
Next, the assistance level and changes to the assistance level will be described.
The assistance levels are at least two or more levels, for example, "large" and "small", or "strong", "normal" and "weak", or "level 1", "level 2", "level 3" and "level 4", etc.
When there are three assistance levels, "strong,""normal," and "weak," the control unit 56, for example, sets the initial value to "normal," compares the calculated work score with a preset threshold, and changes the level to "weak" if the work score is equal to or greater than the threshold, or changes the level to "strong" if the work score is less than the threshold.
The above-described assistance levels and assistance level switching are merely examples and are not limiting.

(Estimation of Operation Intention)
Next, an outline of a method for estimating an operational intention will be described.
The control unit 56 estimates the operator's intention using the information acquired by the acquisition unit 51. Note that the control unit 56 estimates the operator's intention using at least one of, for example, gaze information, operator arm information, and head movement information. Note that the control unit 56 may also estimate the intention using a detection value detected by the environmental sensor 4.

For example, when the task includes grasping, the control unit 56 estimates the operator's intention to operate using the GRASP Taxonomy method (see, for example, Reference 1).
The control unit 56 classifies the operator state by classifying the posture of the operator or the robot 6, i.e., the gripping posture, using, for example, a grasptaxonomy method, and estimates the operator's operational intention. The control unit 56, for example, inputs operation information into a trained model stored in the storage unit 57 to estimate the operator's operational intention. Note that other methods may be used to classify the gripping posture. The control unit 56 may also perform an integrated estimation using the gaze and the movement of the arm. In this case, the control unit 56 may input gaze information, hand movement information, and position information of an object on a table into a trained model to estimate the operator's operational intention.

Reference 1: Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77

The control unit 56 may estimate the object to be grasped based on, for example, line-of-sight information. Next, the control unit 56 estimates the posture of the operator's hand based on the estimated object to be grasped. Alternatively, the control unit 56 may estimate the posture of the operator's hand based on, for example, operation information input by the operator. The control unit 56 may estimate in advance the future trajectory of the operator's intended hand based on the operation information and state information of the robot 6. The control unit 56 may estimate the object to be manipulated and its position using the results of image processing of images captured by the environmental sensor 4 (see, for example, Japanese Patent Application No. 2022-006498).

(Example of information stored in the storage unit)
Next, an example of information stored in the storage unit 57 will be described.
4 to 6 are diagrams showing examples of information stored in the storage unit of the operation control device according to the present embodiment.
4, the storage unit 57 stores at least an operation score in association with an operator. The storage unit 57 may also store the operation score in association with the speed and completion of the operation. The storage unit 57 stores such an operation score for each operator for each operation. The information indicating the operator is, for example, a name or an identifier.

As shown in FIG. 5, the memory unit 57 stores operation data for each operator. The operation data includes time-series operation instructions for one task, time-series joint angle information for the robot 6 based on the operation instructions, and motion images (still images or video) of the robot 6 based on the operation instructions.

As shown in FIG. 6, the storage unit 57 stores typical operation data for each operation. The typical operation data is, for example, data for completing a task with exemplary speed, and may be operation data of an expert.

(Example of image)
Next, an example of a difference image presented to the operator will be described.
FIG. 7 is a diagram showing an example of a difference image presented to the operator.
Reference symbol g101 denotes an image of the robot 6 when it operates in accordance with an operation instruction from an operator. Note that the image may be a real image or a simulation image.
The line drawing g102 is a line drawing of the robot 6 when a second worker with a higher task score than the operator performs the same task, or a line drawing of the robot 6 based on typical operation data.
In the example of FIG. 7, a model line drawing is displayed superimposed on the operation image of the operator.

When the image display unit 3 is, for example, a personal computer monitor, a smartphone screen, or a tablet terminal screen, as shown in FIG. 8, an image g121 of the robot 6 being operated by the operator and an image g122 of the robot 6 when a second worker with a higher task score than the operator performs the same task (or an image of the robot 6 based on typical operation data) may be displayed side-by-side or vertically. FIG. 8 shows another example of a difference image presented to the operator.

Note that the presentation images shown in FIG. 7 and FIG. 8 are merely examples, and the present invention is not limited to these.
Note that the difference generation unit 53 may suggest the difference by text or voice, other than presenting the difference by image as in Fig. 7 and Fig. 8. For example, as shown by reference symbol g131 in Fig. 8, the difference generation unit 53 may suggest how to correct the finger position, movement timing, etc. to improve the task score based on a result of comparing the operation data of a first operator with the operation data of a second operator having a higher task score than the first operator, or based on the calculated task score.

(Example of processing procedure)
Next, a description will be given of an example of a processing procedure performed by the remote control device 5. Fig. 9 is a flowchart of an example of a processing procedure performed by the remote control device according to this embodiment.

(Step S1) The acquisition unit 51 acquires the image captured by the environmental sensor 4.

(Step S2) The acquisition unit 51 acquires the sensor information detected by the sensor 61.

(Step S3) The acquisition unit 51 acquires the operation information detected by the operation input unit 2.

(Step S4) The control unit 56 uses the acquired information to estimate the operation intention.

(Step S5) The control unit 56 generates a control command value (joint angle command value, etc.) based on the acquired information and the estimated operation intention.

(Step S6) The control unit 56 drives the robot 6 based on the generated control command value.

(Step S7) The control unit 56 determines whether or not the desired series of tasks has been completed. If the tasks have been completed (Step S7; YES), the control unit 56 proceeds to the process of Step S8. If the tasks have not been completed (Step S7; NO), the control unit 56 returns to the process of Step S1.

(Step S8) The score calculation unit 52 calculates the task score for the completed task.

(Step S9) The difference generation unit 53 obtains operation differences between operators with higher work scores than the current operator, and generates difference information to be presented. The difference generation unit 53 generates a proposal (text or voice) to the operator based on the work score, operation data, etc. The provision unit 54 causes the image display unit 3 to present the difference information and proposal generated by the difference generation unit 53.

(Step S10) The assistance level change unit 55 changes the level of the assistance function according to the task score. The control unit 56 changes the assistance level according to the level changed by the assistance level change unit 55. After processing, the control unit 56 returns to the processing of step S1.

Note that the process contents and process procedures shown in FIG. 9 are merely examples and are not limited to these. For example, the remote control device 5 may perform several processes simultaneously, or may switch between process procedures.

As described above, in this embodiment, the work data of the person with the superior work score is compared with the person's own video and operation data, and the method of work is suggested or visually judged by a human based on the differences. Note that a typical gripping pattern and a manual pattern may also be selected. In such a case, the remote control device 5 may suggest the most suitable typical pattern from multiple patterns in the database based on the pattern selected by the operator and the work record (speed and quality).

As a result, this embodiment suggests optimal support functions based on the differences between you and a skilled person, making work easier even for people with low proficiency.

In addition, a program for implementing all or part of the functions of the remote control device 5 in the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform all or part of the processing performed by the remote control device 5. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a home page providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes those that hold a program for a certain period of time, such as volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

1...Remote control system, 2, 2-1, 2-2...Operation input unit, 3, 3-1, 3-2...Image display unit, 4...Environment sensor, 5...Remote control device, 6, 6-1, 6-2...Robot, 7...DB, 21...Operation detection unit, 22...Gaze detection unit, 51...Acquisition unit, 52...Score calculation unit, 53...Difference generation unit, 54...Provision unit, 55...Assistance level change unit, 56...Control unit, 57...Memory unit, 61...Sensor, 63...Arm, 64...End effector, 65...Drive unit

Claims (10)

A remote control system in which an operator remotely controls a remote control target which is a real object or a virtual object in a virtual world,
a score calculation unit that calculates a task score of the remote operation;
a difference presentation unit that presents a difference between the operation of the first operator and an operation of a second operator having a higher task score than the first operator;
A remote control system comprising: a control unit that estimates an operation intention of the operator from the information of the first operator and performs operation assistance according to the operation intention on the remote-controlled robot;
an assistance level change unit that changes a level of the action assistance in accordance with the task score;
The remote control system according to claim 1 . The score calculation unit calculates the score by weighting and adding the time required for the task and the achievement level, which is the quality of each task.
3. The remote control system according to claim 1 or 2. The difference presentation unit is
comparing the operation data of the first operator with the operation data of the second operator to obtain the difference, generating presentation information to be presented based on the obtained difference, and presenting the generated presentation information;
3. The remote control system according to claim 1 or 2. The presentation information is at least one of an image, a text, and a sound.
The remote control system according to claim 4. The difference presentation unit is
comparing an operation state of the first operator with an operation state of the second operator to obtain a difference, and presenting an improvement proposal based on the obtained difference;
3. The remote control system according to claim 1 or 2. The operation data of the second operator is operation data of a typical pattern.
The remote control system according to claim 4. A remote control device of a remote operation control system in which an operator remotely operates a remote operation target which is a real object or a virtual object in a virtual world,
a score calculation unit that calculates a task score of the remote operation;
a difference presentation unit that presents a difference between the operation of the first operator and an operation of a second operator having a higher task score than the first operator;
A remote control device comprising: A remote control method for a remote control device of a remote operation control system in which an operator remotely controls a remote operation target which is a real object or a virtual object in a virtual world, comprising:
The score calculation unit calculates a task score of the remote operation,
a difference presentation unit presents a difference between the operation of the first operator and an operation of a second operator having a higher task score than the first operator;
Remote control method.
A computer of a remote control device of a remote control system in which an operator remotely controls a remote control target which is a real object or a virtual object in a virtual world,
Calculate the remote operation work score,
a difference between the operation of the first operator and an operation of a second operator having a higher task score than the first operator is displayed;
program.

Images