KR20230154363A - System and method for controlling dangerousness tasks using robot based on xtended reality - Google Patents

Abstract

Disclosed are a system and method for controlling a dangerous task using a robot based on extended reality. According to an aspect of the present invention, the system for controlling a dangerous task using a robot based on extended reality comprises: a server learning a task performed in a dangerous region, one or more scenarios on the sequence and moving path of each task, and an operation model on vision information and movement information of a robot performing each scenario; a robot system having a photographing device, and moving in accordance with an operation control signal, and performing a relevant task; and a user system displaying an image photographed by the photographing device through an extended reality display device worn on the head of a user, and transmitting a control command in accordance with the speech and movement of the user to the server. The server is able to generate an operation control signal by acquiring the vision information or the movement information of the robot in accordance with the control command based on the operation model, and transmit the operation control signal to the robot system. Therefore, a user can be enabled to control and monitor a robot by using the extended reality technology.

Description

System and method for controlling dangerous tasks using extended reality-based robots {SYSTEM AND METHOD FOR CONTROLLING DANGEROUSNESS TASKS USING ROBOT BASED ON XTENDED REALITY}

The present invention relates to a system and method for controlling dangerous work using an extended reality-based robot. More specifically, when a robot is put into dangerous work such as a power plant, the user can control and monitor the robot using extended reality technology. It relates to a hazardous work control system and method using an extended reality-based robot.

In general, people tend to avoid working in sites or dangerous places with poor working environments such as hazardous substances, abnormal temperatures, humidity, noise, and gas, including power plants and small and medium-sized factories, so the supply and demand for securing labor in these industries Relationships are emerging as a big problem.

To solve this problem, industrial robots have been used recently, and they are also used as a means to minimize product defect rates and safety accidents that can easily occur due to boredom and carelessness in monotonous repetitive tasks that humans tend to fall into.

In addition, even in industries such as welding and painting, which have poor working conditions but require skilled labor, robots are used as a means of solving the problem of supply and demand of skilled workers who do not meet the increasing workload each year and the increase in product costs caused by rising labor costs due to manpower shortage. This is in the spotlight.

Robots are auxiliary devices that complement human labor and reduce the human workforce by taking the place of jobs that humans dislike and extreme labor that exceeds human labor capabilities in places with poor working environments and preventing mistakes that can easily occur due to human carelessness. It is being used as.

In this way, it is possible to perform dangerous tasks using a robot without humans directly approaching the dangerous area, but it is only possible to monitor the movements performed by the robot, and it is not possible to control the robot's movements by reflecting the user's movements in real time. There wasn't.

The background technology of the present invention is disclosed in 'Wind power generator maintenance unit and maintenance method using the same' in Republic of Korea Patent Publication No. 10-1592904 (announced on February 18, 2016).

The present invention was brought out to improve the above-mentioned problems, and the purpose of the present invention is to use extended reality technology to enable users to control and monitor robots when putting robots into dangerous work such as power plants. To provide a system and method for controlling hazardous work using robots.

The problem to be solved by the present invention is not limited to the problem(s) mentioned above, and other problem(s) not mentioned will be clearly understood by those skilled in the art from the description below.

A hazardous work control system using an extended reality-based robot according to an aspect of the present invention includes at least one scenario for tasks performed in a hazardous area, the sequence and movement path of each task, and vision information of a robot performing each scenario. and a server that learns a motion model for motion information, a robot system equipped with a photographing device, and moves according to a motion control signal to perform the corresponding task, and the photographing device through an extended reality display device worn on the user's head. It includes a user system that displays a captured image and transmits a control command according to the user's voice and movement to the server, wherein the server provides vision information or movement of the robot according to the control command based on the operation model. Information can be acquired to generate the motion control signal, and the motion control signal can be transmitted to the robot system.

In the present invention, the robot system includes a communication circuit, a photographing device that operates according to the movement of the extended reality display device to capture images, a robot that moves according to the motion control signal and performs a task, and captures images through the photographing device. It includes a control device that transmits the image to the server through the communication circuit, transmits an operation control signal from the server to the robot or imaging device, and causes the robot or imaging device to operate according to the operation control signal. can do.

In the present invention, the user system includes a communication circuit, an extended reality display device worn on the user's head and displaying images captured through the photographing device in extended reality, a voice recognition device that recognizes the user's voice, and Determine the user's intention regarding the movement of the robot from voice data recognized by a haptic sensor that recognizes the user's tactile information and the voice recognition device or tactile information recognized through the haptic sensor, and determine the user's intention for the movement of the robot according to the identified intention. It may include a control device that transmits control commands for controlling the robot to the server and outputs images received from the robot system through the extended reality display device.

In the present invention, the server receives vision information or movement information of the robot performed according to the motion control signal from the robot system, and if the received vision information or movement information of the robot is unlearned information, the server receives the vision information or movement information of the robot performed according to the motion control signal. The motion model can be updated by learning the vision information or movement information of one robot.

A method of controlling hazardous work using an extended reality-based robot according to another aspect of the present invention is a state in which the user system displays images captured through the imaging device of the robot system through the extended reality display device worn on the user's head. When the user's voice data or movement information is obtained, transmitting a control command including the voice data or movement information to a server, the server configuring the robot according to the control command based on a previously learned motion model. It includes obtaining the coordinates of elements to generate a motion control signal, transmitting the generated motion control signal to a robot system, and performing a task by moving the robot system according to the motion control signal.

The present invention further includes the step of the server learning a motion model for tasks performed in a dangerous area, at least one scenario for the sequence and movement path of each task, and vision information and movement information of a robot performing each scenario. And after the step of performing the task, the server receives vision information and movement information performed by the robot according to the task performance from the robot system, and the received vision information and movement information of the robot are not learned. If the information is not available, the step of updating the motion model by learning the received vision information and movement information of the robot may be further included.

In addition to this, other methods for implementing the present invention, other systems, and computer programs for executing the methods may be further provided.

The system and method for controlling dangerous work using an extended reality-based robot according to an embodiment of the present invention allows the user to control and monitor the robot using extended reality technology when a robot is put into dangerous work such as a power plant. .

Meanwhile, the effects of the present invention are not limited to the effects mentioned above, and various effects may be included within the range apparent to those skilled in the art from the contents described below.

Figure 1 is a diagram for explaining a hazardous work control system using an extended reality-based robot according to an embodiment of the present invention.
Figure 2 is a block diagram schematically showing the configuration of a user system according to an embodiment of the present invention.
Figure 3 is an example diagram illustrating a screen displayed through an extended reality display device according to an embodiment of the present invention.
Figure 4 is a block diagram schematically showing the configuration of a robot system according to an embodiment of the present invention.
Figure 5 is a diagram for explaining a method of controlling dangerous work using an extended reality-based robot according to an embodiment of the present invention.

Hereinafter, a hazardous work control system and method using an extended reality-based robot according to an embodiment of the present invention will be described with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawing may be exaggerated for clarity and convenience of explanation.

Additionally, implementations described herein may be implemented as, for example, a method or process, device, software program, data stream, or signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the features discussed may also be implemented in other forms (eg, devices or programs). The device may be implemented with appropriate hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device that includes a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

Additionally, the terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by these terms. The above terms are used only for the purpose of distinguishing one component from another.

Throughout the specification, “extended reality (XR)” refers to virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR technology provides objects and backgrounds in the real world only as CG images, AR technology provides virtual CG images on top of images of real objects, and MR technology provides computer technology that mixes and combines virtual objects in the real world. It is a graphic technology. MR technology is similar to AR technology in that it shows real objects and virtual objects together. However, in AR technology, virtual objects are used to complement real objects, whereas in MR technology, virtual objects and real objects are used equally.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components are assigned the same drawing numbers and duplicate descriptions thereof are omitted. I decided to do it.

FIG. 1 is a diagram illustrating a hazardous work control system using an extended reality-based robot according to an embodiment of the present invention. FIG. 2 is a block diagram schematically showing the configuration of a user system according to an embodiment of the present invention. FIG. 3 is an example diagram for explaining a screen displayed through an extended reality display device according to an embodiment of the present invention, and FIG. 4 is a block diagram schematically showing the configuration of a robot system according to an embodiment of the present invention.

Referring to FIG. 1, a hazardous work control system using an extended reality-based robot according to an embodiment of the present invention includes a server 100, a user system 200, and a robot system 300.

The server 100 may learn a motion model for tasks performed in a power plant, etc., at least one scenario for the sequence and movement path of each task, and vision information and movement information of the robot 330 performing each scenario. Here, the motion model may include movement information such as grabbing, pushing, and moving the item, and the motion information may be related to the components constituting the robot 330 (e.g., driving unit, body, link, end effector, etc.). It may include rotational force, rotation range, driving range, gripping force, adsorption force, etc. At this time, the server 100 can learn a motion model composed of an artificial neural network using data (expert data or big data) in which an expert performs a given motion.

The server 100 receives data (vision, movement information, more than 100 for each scenario) about the movements of experts performing dangerous tasks in an extended reality environment and provides vision information (vision information) so that the actual robot 330 can perform the task. A motion model (artificial neural network model) including CNN), motion information (NN), etc. can be trained, and the learned motion model can be used as an input to the robot 330.

The server 100 acquires movement information that repeats task performance actions in an extended reality environment, and infers arm movements and both arm movement movements based on the hand movements of the user equipped with the extended reality display device 220, so that the user It is possible to determine the movement intention and learn a motion model that causes the robot 330 to operate according to the determined intention. At this time, the server 100 can learn the motion model using a model-based inverse reinforcement learning algorithm.

In order to perform a task (grasping, pushing, etc.) using the force/tactile information detected through the haptic sensor 240 of the user system 200, it must be used together with the kinematic information of the robot 330 to ensure stable interaction with the item. For this, the balance of power must be maintained. Accordingly, the server 100 can utilize the balance of power relationship in the policy and objective function model using the behavior cloning/inverse reinforcement learning algorithm.

The server 100 receives vision information and movement information performed by the robot 330 according to task performance from the robot system 300, and when the received vision information and movement information of the robot 330 are unlearned information. The motion model can be updated by learning the received vision information and movement information of the robot 330.

When receiving a control command from the user system 200, the server 100 acquires vision information and movement information of the robot 330 according to the control command based on the motion model, generates a motion control signal, and sends the motion control signal to the robot. It can be transmitted to system 300. Here, the motion control signal may include vision information and movement information of the robot 330.

Since the control command includes the user's vision information and movement information, and the vision information and movement information include coordinate values, the server 100 converts the user's vision information and movement information into the vision information and movement information of the robot 330. It can also be converted and transmitted to the robot system 300.

This server 100 may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a learned artificial neural network.

Server 100 may be a computing device including at least one processor. Server 100 may also mean a cloud server. In one embodiment, the server 100 may be used interchangeably with a platform.

The user system 200 displays the image captured by the photographing device 320 through the extended reality display device 220 worn on the user's head, and transmits control commands according to the user's voice and movement to the server 100. You can.

The user system 200 can receive images captured through the imaging device 320 of the robot system 300 as VR/AR/MR images.

The user system 200 can transmit control commands for manipulating the movement/motion/movement, camera angle, etc. of the robot system 300 to the robot system 300 through the server 100, and the robot system 300 Movement/motion/movement is controlled according to the operation control signal corresponding to the control command received from the user system 200, and extended reality images can be transmitted in real time.

As shown in FIG. 2, this user system 200 may include a communication circuit 210, an extended reality display device 220, a voice recognition device 230, a haptic sensor 240, and a control device 250. there is.

The communication circuit 210 can provide a communication interface necessary to provide transmission and reception signals between the user system 200 and the server 100 in the form of packet data in conjunction with a communication network. In particular, the communication circuit 210 may receive images captured in the robot system 300 from the server 100 and transmit control commands for controlling the robot 330 to the server 100. . Additionally, the communication circuit 210 may be a device that includes hardware and software necessary to transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices. Additionally, the communication circuit 210 may be implemented in various forms such as a short-distance communication module, wireless communication module, mobile communication module, and wired communication module.

The extended reality display device 220 is worn on the user's head and can display images captured through the imaging device 320 of the robot system 300 in extended reality.

The extended reality display device 220 may be implemented in the form of a head mounted display (HMD) at a location corresponding to both eyes of the user. Additionally, the extended reality display device 220 may be implemented in the form of a virtual reality headset, HoloLens, AR/MR glasses, etc.

The extended reality display device 220 is equipped with at least one direction/gyro/angle/acceleration sensor to sense the direction/angle of the user's head, and transmits the sensed results to the robot system 300 through the control device 250. You can. Then, the imaging device 320 of the robot system 300 may operate in a direction/angle corresponding to the direction/angle of the extended reality display device 220. The extended reality display device 220 can display the image transmitted from the robot system 300 as an extended reality image.

Accordingly, the user can perform the corresponding task while checking image information captured by the imaging device 320 of the robot system 300 in real time through the extended reality display device 220.

The voice recognition device 230 can recognize the user's voice and transmit it to the control device 250.

The voice recognition device 230 may be detachably provided on one side of the extended reality display device 220 or may be implemented in the form of a microphone integrally provided in the extended reality display device 220.

The haptic sensor 240 can recognize the user's force/tactile sensation and transmit the recognized tactile information to the control device 250.

The control device 250 determines the user's intention regarding the movement of the robot 330 from voice data recognized by the voice recognition device 230 or tactile information recognized through the haptic sensor 240, and performs the operation according to the identified intention. Control commands for controlling the robot 330 can be generated and transmitted to the server 100.

Additionally, the control device 250 may output the image received from the robot system 300 through the extended reality display device 220. That is, the control device 250 can render the image received from the robot system 300 into a VR/AR/MR image and output it through the extended reality display device 220. At this time, the control device 250 can configure a plurality of virtual screens on the extended reality display device 220 to control and monitor the work process performed by the robot 330.

For example, the control device 250 can display five virtual screens on the extended reality display device 220 as shown in FIG. 3. When the user looks straight ahead, the front screen is displayed, when the user turns his head to the left, the left screen is displayed, when the user turns his head to the right, the right screen is displayed, and when the user looks up, the top screen is displayed. It can be displayed, and the bottom screen can be displayed when the user looks down.

In this way, the control device 250 can link the environment (visual information) seen by the robot 330 with the extended reality environment used by the user.

In addition, the control device 250 acquires the end hand movements of the user equipped with the extended reality display device 220 through the haptic sensor 240, and performs arm movements and both arm movement actions based on the acquired hand movements of the user. Inferring, the user's intention can be determined based on the inferred arm movement and both arm movements, and a control command according to the identified user's intention can be generated and transmitted to the server 100.

The robot system 300 is equipped with an imaging device 320 and can perform dangerous tasks according to operation control signals from the server 100.

This robot system 300 may include a communication circuit 310, an imaging device 320, a robot 330, and a control device 340, as shown in FIG. 4.

The communication circuit 310 can provide a communication interface necessary to provide transmission and reception signals between the robot system 300 and the server 100 in the form of packet data in conjunction with a communication network. In particular, the communication circuit 310 may receive an operation control signal from the server 100 and transmit an image captured through the photographing device 320 to the server 100. Additionally, the communication circuit 310 may be a device that includes hardware and software necessary to transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices. Additionally, the communication circuit 310 may be implemented in various forms such as a short-distance communication module, wireless communication module, mobile communication module, and wired communication module.

The photographing device 320 can capture images by operating according to the movement of the user (i.e., the extended reality display device 220). In other words, the imaging device 320 can link the vision information seen by the robot 330 with the extended reality environment used by the user. This imaging device 320 may be implemented as, for example, a PTZ camera.

The robot 330 can move and perform work in accordance with the motion control signal from the server 100. In other words, the robot 330 can perform work while moving according to the movement path included in the motion control signal.

The robot 330 is equipped with a driving unit (not shown) including an actuator or motor and can perform various physical movements, such as moving the joints of the robot 330. In addition, the movable robot 330 includes wheels, brakes, propellers, etc. in the driving part, and can travel on the ground or fly in the air through the driving part.

The robot 330 may be equipped with a smart sensor (not shown). The smart sensor is mounted on the arm of the robot 330 and can link the tactile information of the robot 330 with the user's tactile information.

Meanwhile, although the embodiment of the present invention is described as a robot 330, the robot 330 may be a robot arm. Since the robot 330 and the robot arm have the same configuration as the conventional one, their description will be omitted.

The control device 340 transmits the image captured through the imaging device 320 to the server 100 through the communication circuit 310, and transmits the operation control signal from the server 100 to the robot 330 and the imaging device ( 320), the robot 330 and the imaging device 320 can be controlled to operate according to the operation control signal.

Figure 5 is a diagram for explaining a method of controlling dangerous work using an extended reality-based robot according to an embodiment of the present invention.

Referring to FIG. 5, when the user's voice data or motion information is acquired (S510), the user system 200 transmits a control command including the voice data or motion information to the server 100 (S520).

When step S520 is performed, the server 100 acquires the coordinates of robot components according to the control command based on the motion model, generates a motion control signal (S530), and sends the generated motion control signal to the robot system 300. Transmit (S540). Here, the control command includes the user's vision information and movement information, and the vision information and movement information include coordinate values, so the server 100 combines the user's vision information and movement information with the vision information and movement of the robot 330. It can be converted into information, and a motion control signal including the converted vision information and movement information can be transmitted to the robot system 300.

When step S540 is performed, the robot system 300 moves according to the motion control signal and performs the task (S550).

Afterwards, the robot system 300 transmits vision information and movement information to the server 100 when the robot 330 performs a task, and the server 100 transmits the vision information and movement information of the robot 330 to information that has not been learned. In this case, the motion model can be updated by learning the received vision information and movement information of the robot 330.

As described above, the system and method for controlling hazardous work using an extended reality-based robot according to an embodiment of the present invention allows the user to control and control the robot using extended reality technology when putting a robot into dangerous work such as a power plant. It can be monitored.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will recognize that various modifications and other equivalent embodiments are possible therefrom. You will understand. Therefore, the true technical protection scope of the present invention should be determined by the scope of the patent claims below.

100: Server
200: user system
210, 310: Communication circuit
220: Extended reality display device
230: Voice recognition device
240: Haptic sensor
250, 340: Control device
300: Robot system
320: filming device
330: robot

Claims (6)

A server that learns a motion model for tasks performed in a dangerous area, at least one scenario for the sequence and movement path of each task, and vision information and movement information of a robot performing each scenario;
A robot system equipped with an imaging device and moving according to motion control signals to perform the corresponding task; and
It includes a user system that displays images captured by the photographing device through an extended reality display device worn on the user's head and transmits control commands according to the user's voice and movement to the server,
The server is,
Based on the motion model, the vision information or movement information of the robot according to the control command is acquired, the motion control signal is generated, and the motion control signal is transmitted to the robot system. Hazardous work control system used.
According to paragraph 1,
The robot system is,
communication circuit;
a photographing device that operates according to the movement of the extended reality display device to capture images;
A robot that moves and performs work according to the motion control signal; and
The image captured through the imaging device is transmitted to the server through the communication circuit, and an operation control signal from the server is transmitted to the robot or imaging device, so that the robot or imaging device operates according to the operation control signal. A hazardous work control system using an extended reality-based robot, characterized in that it includes a control device to do so.
According to paragraph 1,
The user system is,
communication circuit;
an extended reality display device worn on the user's head and displaying images captured through the photographing device in extended reality;
A voice recognition device that recognizes the user's voice;
A haptic sensor that recognizes the user's tactile information; and
The server determines the user's intention for the movement of the robot from voice data recognized by the voice recognition device or tactile information recognized through the haptic sensor, and sends a control command to control the robot according to the identified intention. A hazardous work control system using an extended reality-based robot, comprising a control device that transmits the image received from the robot system and outputs the image received from the robot system through the extended reality display device.
According to paragraph 1,
The server is,
Vision information or movement information of the robot performed according to the motion control signal is received from the robot system, and if the received vision information or movement information of the robot is unlearned information, the received vision information or movement information of the robot A hazardous work control system using an extended reality-based robot, characterized in that the motion model is updated by learning information.
When the user's voice data or motion information is acquired while the user system is displaying an image captured through the imaging device of the robot system through the extended reality display device worn on the user's head, the voice data or motion information is Transmitting a control command including a control command to a server;
The server generates a motion control signal by acquiring coordinates of robot components according to the control command based on a previously learned motion model, and transmitting the generated motion control signal to the robot system; and
A step in which the robot system moves according to a motion control signal and performs work.
A method of controlling dangerous work using an extended reality-based robot including.
According to clause 5,
The server further includes a step of learning a motion model for tasks performed in a dangerous area, at least one scenario for the sequence and movement path of each task, and vision information and movement information of a robot performing each scenario,
After performing the above operations,
The server receives vision information and movement information performed by the robot according to the task performance from the robot system, and when the received vision information and movement information of the robot are unlearned information, the received vision information of the robot And a hazardous task control method using an extended reality-based robot, further comprising the step of updating the motion model by learning motion information.