JP2003311661A - Remote control device - Google Patents

Abstract

(57) [Summary] [PROBLEMS] As a remote control device configured to allow a robot to perform necessary work while remotely controlling a robot in a work environment, a burden on an operator who manually operates the robot. And a remote control device that can be applied to a wider range of maintenance targets and work contents than before. SOLUTION: A camera 3 for capturing an image of a work environment, an environment model storage means 5 for storing, as an environment model, information relating to the position and orientation of objects 12 and 13 in the work environment and positioning necessary for operation of the robot 1, and a camera Image generating means 8 for generating a synthesized image by synthesizing the image captured by the camera and the image obtained by graphically displaying the information on the positioning obtained from the environment model. When the robot is manually operated, the synthesized image is generated. It is displayed on the display means 9 so that a manual operation can be guided.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a remote control device used, for example, to remotely maintain a device or the like located in a place where maintenance personnel cannot easily enter it by operating a robot.

[0002]

2. Description of the Related Art Equipment that is in an environment where humans cannot easily enter, such as a radiation environment or a space environment, or equipment that is located in a remote place and where maintenance personnel are required to visit it at a time and costly place. In order to perform maintenance on a regular basis, it is desirable to work remotely from a remote location using a work robot. At this time, if the work to be performed in the maintenance is known, or if the contents of the work in the periodic maintenance are the same each time, the robot operation is programmed in advance and the operation is automatically performed without human operation. It is efficient to be able to work while driving.
(In this specification, the term "robot" refers to a general mechanical device including a manipulator and the like, which allows remote operation.)

However, even if the same work is performed on a regular basis, an unexpected situation may occur, such as the presence of obstacles that did not exist the previous time or the breakage of some of the devices to be maintained. It is possible that In the current robot technology, it is difficult to correctly interpret such an unexpected situation and avoid it automatically by a program, so conventionally, a master-slave manipulator that conveys the hand movement of a person in a remote place to the manipulator. Maintenance work is often performed manually with a device such as. Alternatively, automatic operation can be performed by limiting the maintenance target and work content that can be detected in the event of an unexpected situation, or by providing a special device to the device to be maintained so that an unexpected situation can be detected. The maintenance work of is realized.

[0004]

In the method using the master-slave manipulator in the above conventional technique,
Since a TV camera is installed at the work site of the manipulator and the user works while watching the image captured by this camera, there is a problem that the work is difficult to do and it takes time. Also,
In the method of operating the master arm in such a situation, there is a problem that operator's fatigue is significant and it is difficult to work for a long time.

On the other hand, the above-mentioned conventional automatic driving method has a problem that applicable maintenance targets and work contents are significantly limited. Even if unexpected situations do not occur, there are still many tasks that require sufficient reliability, such as attachment / detachment of parts, that require subtle positioning and force adjustment, and cannot be automated with sufficient reliability. There is a problem that

As a remote control device using a robot, an example disclosed in Japanese Patent Laid-Open No. 9-254065 is known. However, JP-A-9-2540
The technique disclosed in Japanese Patent No. 65 is intended to improve the efficiency of displaying a robot and its working environment on a display device for remote control of the robot. Does not bring a solution.

Therefore, an object of the present invention is to provide a remote operation control device which can reduce the burden on the operator and can be applied to a wider range of maintenance targets and work contents than before. is there.

[0008]

To achieve the above object, the present invention allows a robot installed in a work environment to be remotely operated and controlled by an operation control system while allowing the robot to perform a necessary work. In the remote control device, the operating means for manually operating the robot, the camera for capturing an image of the work environment, the position and orientation of the object to be worked by the robot in the work environment, and the operation of the robot An environment model storage means for storing information about positioning in a necessary work environment as an environment model, an image captured by the camera and an image displayed by graphically displaying the positioning information obtained from the environment model storage means. A composite image generating means for generating the composite image, and a display means for displaying the composite image generated by the composite image generating means, The serial operation unit when performing manual operation of the robot, is characterized in that it is so capable of inducing manual operation to display the composite image on the display unit (first invention).

To achieve the above object, the present invention allows the robot installed in the work environment to be remotely controlled by an operation control system while allowing the robot to perform necessary work in the work environment. In the remote control device, the environment of the camera that captures the image of the work environment and the position and orientation of the object that is the target of the work by the robot in the work environment and the positioning information in the work environment that is necessary for the operation of the robot The robot includes an environment model storage unit for storing as a model, an operation command generation unit for generating an operation command for automatic operation of the robot, and an operation unit for manually operating the robot, and an automatic operation of the robot. The manual operation can be selected, and the position and orientation of the object in the environment model and the object in the working environment captured by the camera When it is detected that there is a certain amount of deviation between the two by comparing with the position and orientation described in (1), the automatic operation is stopped and the manual operation is automatically performed ( Second invention).

Further, according to the present invention, in the remote control device according to the second aspect of the present invention, a composite image obtained by synthesizing a video captured by a camera and an image displayed by graphically displaying positioning information obtained from the environment model storage means is displayed. Providing a composite image generating means for generating and a display means for displaying the composite image generated by the composite image generating means, when the manual operation is started, the composite image can be displayed on the display means to guide the manual operation. (Third invention).

Further, according to the present invention, in the remote operation control device according to any one of the first to third inventions, the position and orientation of the object in the environment model and the position and orientation of the object in the working environment captured by the camera are determined. When a deviation occurs, an environment model correction means for correcting the environment model is provided so as to eliminate the deviation (fourth invention).

Further, in the present invention, in the remote operation control device according to any one of the first to fourth inventions, a working robot or an object being worked by the robot is in a working environment for its working purpose. The image in which the mark of the position and orientation corresponding to the positioning point to be positioned is displayed is used as the graphic image of the information regarding the positioning (fifth invention).

Further, according to the present invention, in the remote operation control device according to any one of the first to fifth inventions, a robot being worked or an object undergoing work by the robot is in a working environment for its working purpose. Information regarding positioning of an image displaying a first mark of a position and orientation corresponding to a positioning point to be positioned by and a second mark of a position and orientation corresponding to the position and orientation changed by the robot or the object as the work progresses. (6th invention).

Further, according to the present invention, in the remote operation control device according to any one of the first to sixth inventions, an object in a work environment undergoing work by a robot is in the work environment for its work purpose. An image displaying a figure of the appearance shape of the object that will be visible after positioning at the positioning point to be positioned is used as a graphic image of information regarding positioning (seventh invention).

Further, according to the present invention, in the remote operation control device according to any one of the first to seventh inventions, when the robot is made to perform an operation of combining an object to be positioned with another target object in the work environment. , Using an image displaying a figure of the external shape of the target object that would be visible when the target object is combined with the object to be positioned in the state of being operated by the robot, as a graphic image of information regarding positioning (Eighth invention).

Further, according to the present invention, in the remote operation control device according to the eighth aspect of the present invention, a camera capable of capturing an object to be positioned from a gripping portion of the robot while the robot grips an object to be positioned. Is provided, and a graphic image of information regarding positioning is superimposed on the image captured by the camera to generate a composite image (ninth invention).

Further, in the present invention, the seventh invention or the ninth invention is provided.
In the remote operation control device according to the invention described above, the image of the work environment captured by the camera and the graphic image of the information regarding positioning are overlapped in consideration of the depth to generate a composite image (the tenth invention). ).

Further, according to the present invention, in the remote control device according to any one of the first to tenth inventions, the input means for inputting the position on the screen of the display means and the position input by this input means are related. Target object selecting means for selecting one object in the work environment from information as a target object is provided, and when the position is input on the screen of the display means by the input means, the target object selecting means selects A graphic image of information on positioning of the target object is generated (the eleventh invention).

Further, in the present invention, the remote operation control device according to any one of the first to eighth inventions or the tenth to eleventh inventions is provided with a plurality of cameras for capturing an image of a work environment. The captured image of the work environment can be switched and displayed on the display means, and the method of displaying the graphic image of the information regarding positioning can also be switched according to the switching of the displayed image (twelfth). Invention).

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below. FIG. 1 is a block diagram showing the configuration of a remote control device according to an embodiment of the present invention. This remote operation control device comprises a robot 1 installed at a work site for performing work such as maintenance of devices and equipment, and an operation control system for remotely operating and controlling the robot 1. The operation control system includes operation means 2 used for manually operating the robot 1, a camera 3 for capturing an image of a work environment in which the robot 1 operates,
An environment model correction unit 4 that recognizes the position and orientation of an object included in the work environment and corrects the environment model, and stores the position and orientation information of the object included in the work environment and information regarding positioning necessary for robot operation as an environment model. The environment model storage means 5, the operation command generation means 6 for generating an operation command related to an automatic operation for the robot 1, the operation command generated by the operation command generation means 6 and the operation command input from the operation means 2 are selected. Then, the image is displayed by combining the image captured by the camera 3 with the switching means 7 for transferring to the robot 1 and the information stored in the environment model storage means 5 to display the information relating to the positioning necessary for robot operation. The composite image generation means 8 for generating the image, the composite image display means 9, the position on the image captured by the camera 3 and various commands are input to the remote control device. And a target object selecting means 11 for selecting an object in the work environment based on the input unit 10, and the position information inputted from the input means 10 for. In the example of FIG. 1, the object 12 and the object 1 are placed in the work environment.
3, there is shown a situation in which the robot 1 inserts the object 12 into the hole 14 of the object 13.

The operation control system including the environment model correction means 4 to the target object selection means 11 may be configured as a computer system. In such a case, the environment model correction means 4, the action command generation means 6, the switching means 7, the composite image generation means 8 and the target object selection means 11 are used as software on the computer system, and the environment model storage means. Reference numeral 5 denotes a storage device of a computer system, and a display means 9 for displaying a composite image.
Will be configured as a display device of the computer system, and the input means 10 will be configured as an input device of the computer system.

The details of each component will be described below. The robot 1 according to the present embodiment is an arm type robot having a rotary joint and a gripping mechanism for gripping an object at the tip, and the position and orientation of the gripping mechanism is changed by changing the angle of the joint by a driving device. Is able to. In the present invention, in addition to such an arm type robot, a traveling type robot having a moving mechanism such as a leg mechanism or a wheel may be used. Further, the robot does not necessarily have to have an object gripping mechanism, and even an arm type robot does not need to have a rotary joint, and may have a direct acting joint.

The operation means 2 is a device for inputting the position / orientation to be taken by the gripping mechanism of the arm type robot 1 in this embodiment. On the other hand, when the traveling type robot is used, the operating means is a device for inputting its own position and orientation to the traveling type robot.
As a specific configuration, for example, a master arm type device configured as an arm type multi-joint mechanism having the same shape as the robot 1 and operated by holding a place corresponding to a gripping mechanism in a hand may be used. Alternatively, a device such as a joystick or a computer mouse for inputting front, rear, left and right positions may be used.

The environment model storage means 5 stores the arrangement state of the objects in the actual work environment as illustrated in FIG. 2 in the form as shown in FIG. In the example of FIG. 2, in the work environment, an object 100 named floor (meaning the floor of the work environment), an object 101 named body (body) and an object 1 named cap (lid).
02, and the object 102 has a hole 103.

Regarding this work environment, coordinate systems 104 to 110 for representing the position and orientation of the object and the positioning points of the object
Is set. The coordinate system 104 is a coordinate system (world coordinate system) that serves as a reference for the entire environment model, and the positions and orientations of all objects and positioning points included in the environment model are expressed as coordinate values for this coordinate system. Coordinate system 105, 1
Reference numeral 06 is a coordinate system that represents the position and orientation of floor and body, respectively, and 107 is a coordinate system that represents the positioning point of cap with respect to body and at the same time is a coordinate system that represents the position and orientation of cap. That is, FIG. 2 shows that the cap is positioned at the body positioning point. The coordinate system 108 is a coordinate system that represents a positioning point of the gripping mechanism when the robot grips the cap. 1
Reference numeral 09 is a coordinate system representing a positioning point of the object when the object is inserted into the hole 103 provided in the cap. Reference numeral 110 is a coordinate system that represents a point to be positioned immediately before the insertion of an object into the hole 103. In addition,
"Position and orientation" is a concept that includes the position and orientation (orientation) of an object or the like.

Here, the positioning point 109 (coordinate system 109
And the positioning point 110 (coordinate system 1)
The relationship between the positioning points 10) will be described with reference to FIG. When an object such as a bolt 111 is inserted into the hole 103 of the cap, first, FIG.
Set the bolt 111 to the positioning point 11 just above the hole as shown in.
It is desirable to position the hole so that the central axis of the hole and the central axis of the bolt coincide with each other, and then insert straight up to a predetermined positioning point 109 as shown in FIG. 3B.
Similarly, when removing the bolt 111, the positioning point 1
It is desirable to take it to 10 and then remove it. By performing such an insertion / removal operation, it is possible to prevent the occurrence of a trouble in which the bolt is caught on the side wall of the hole.

As described above, the positioning point 109 is a coordinate system that represents the position and orientation of the bolt that should be taken in the state of being inserted into the hole, whereas the positioning point 110 is where the bolt is positioned during insertion or removal. It is a coordinate system that represents the power position and orientation. Hereinafter, a positioning point such as 109 will be referred to as an attach point, while a positioning point such as 110 will be referred to as an approach point. Further, a positioning point for positioning the gripping mechanism of the robot, such as 108, will be referred to as a grasp point.

The environment model storage means 5 shows the situation of the work environment as described above as structure variables 120 to 1 as shown in FIG.
It is stored as 26. Information of the object name 120a, the connection destination 120b, and the coordinate conversion parameter 120c is stored in each structure variable (structure). The object name 120a represents the name of the object or positioning point represented by the structure, the connection destination 120b represents the object or positioning point to which the structure represented by the structure or the positioning point is connected, and the coordinate conversion parameter 120c is the connection destination object or It represents the relative position and orientation with respect to the positioning point. In FIG. 4, the structure 120,
121 and 123 are the object 100 (fl shown in FIG. 2 respectively.
oor), 101 (body), and 102 (cap), and the structures 122, 124, 125, and 126 correspond to the positioning points 107, 108, 109, and 110.

The coordinate conversion parameters are expressed by the following 4 × 4 matrix, for example. Rotation movement conversion matrix around the X-axis, Y-axis, and Z-axis, respectively

[Equation 1] And a three-dimensional vector that represents the translation

[Equation 2] Then, the coordinate conversion parameter A is given by the following formula.

[Equation 3]

By using the coordinate conversion parameters as described above, the position and orientation of an object or a positioning point with respect to the world coordinate system starts from the coordinate conversion parameter of the structure corresponding to the object or the positioning point and its connection destination. It is obtained by multiplying all coordinate transformation parameters up to the structure whose connection destination is the root. For example, the position / orientation of the positioning point 110 sequentially traces the connection destination from the coordinate conversion parameter A7 of the corresponding structure 126 and the connection target structure 125 to the structure 120 having the connection destination as root. It is obtained by multiplying all the coordinate conversion parameters A6, A4, A3, A2 and A1.

Similarly, the position and orientation of an object or a positioning point B with respect to a coordinate system of a certain object or a positioning point A starts from the coordinate conversion parameter of the structure corresponding to B and its connection destination, and the connection destination is A. It is obtained by multiplying all the coordinate conversion parameters up to a certain structure.

In addition to the above, the structure does not have to include the above-mentioned information for identifying whether the structure represents an object or a positioning point, or an object. For example, various information such as information on the shape of the object and information indicating the type of the object is stored.

Next, the details of the environment model correction means 4 will be described. The environment model correction means 4 recognizes the position and orientation of the object by the procedure shown in FIG. 5, and updates the position and orientation information of the object recorded in the environment model. First, an edge image of an object whose position and orientation is to be recognized is acquired from the image of the work environment captured by the camera 3 (s10). The edge image is an image obtained by performing processing such as extraction of a portion of the image in which the brightness greatly changes. This is done by obtaining the difference in brightness (brightness) between each pixel of the captured image and the adjacent pixel, and leaving the pixels whose difference is larger than a preset threshold value. Obtainable. The image thus obtained becomes an image of a pattern such as the image 133 shown in FIG. 6, for example. This image 133 is an image that includes many contour lines of the object, but does not necessarily include all contours.

Next, regarding the object whose position and orientation is to be recognized, the position and orientation of the object are extracted from the information stored in the environment model storage means 5, and the position and orientation are visible when viewed from the position of the camera 3. An expected contour image (reference image) is generated (s11). The position / orientation of the object is obtained by calculating the position / orientation with respect to the world coordinate system by the procedure already described for the corresponding structure in the environment model. Further, the shape data of the object is obtained as information stored in the corresponding structure in the environment model. From these data and the position / orientation data of the camera, a contour image 134 of the object as shown in FIG. 6, for example, is generated by means similar to the procedure for generating a computer graphic (CG).

Next, the edge image and the reference image (contour image)
The virtual attractive force that acts during the period is calculated (s12). This is because, for example, the edge image pattern is positively charged,
It is calculated as a two-dimensional attractive force that acts between two patterns, assuming that the pattern of the reference image is negatively charged. This process corresponds to approximately obtaining the amount of deviation between the edge image representing the situation of the object in the actual work environment and the reference image representing the situation of the object recorded in the environment model.

Next, the position and orientation of the object are corrected so that the above two patterns are overlapped (s13). This converts the direction in which the attractive force acts and the direction of the moment acting on the reference pattern by the attractive force into the parallel movement direction and the rotational movement direction in the original three-dimensional space, and moves / rotates the position and orientation of the object in these directions. Meanwhile, the pattern of the reference image and the pattern of the edge image are superimposed. At this time, the correction result of the position and orientation of the object is reflected in the coordinate conversion parameter of the corresponding structure in the environment model.

Next, the degree of coincidence between the two patterns is calculated (s14). This is calculated by, for example, the amount of correction of the position and orientation of the object in step s13. If the amount of correction is small, the degree of coincidence is large (the two patterns are sufficiently overlapped), and if the amount of correction is large, the degree of coincidence is small (two. The two patterns are not yet sufficiently overlapped).

Finally, it is determined whether or not the superposition processing of the patterns indicated by s11 to s14 is in a converged state. If the determination result is a convergent state, the processing is terminated, and if it is not converged, s
Return to 11 (s15). The determination of convergence / non-convergence is made by, for example, determining the convergence state if the degree of coincidence calculated in s14 is equal to or higher than a threshold value, or determining non-convergence otherwise.

Next, the details of the action command generation means 6 will be described. The motion command generation means 6 stores motion command data for automatically driving the robot, and the motion of the robot is controlled by the procedure shown in FIG. 7 based on this motion command data. Here, the operation command data is, for example, command data for instructing to move the gripping mechanism of the robot to a certain position / posture, command data for instructing opening / closing of a finger provided in the gripping mechanism, or automatic driving to manual driving. It is composed of command data for instructing switching of the.

First, with T set to 1, the first operation command is read (s20, s21), and if the operation command is an instruction to switch to manual operation, manual operation is executed, and if not, automatic operation is performed. Execute (s22, s23, s2
4). Then, T is incremented by 1, the next operation command is read, and the same processing is repeated until all the operation commands are executed (s25, s26).

In step s24 of executing the automatic operation in the procedure shown in FIG. 7, the switching means 7 is set to a state in which the operation command generated by the motion command generating means 6 is transferred to the robot 1. Then, when the operation command is a command to move the gripping mechanism of the robot to a certain position and orientation, the control data of the joint angle corresponding thereto is transferred to the robot. If the operation command is a command to open or close the gripping mechanism of the robot, the finger control data of the gripping mechanism is transferred to the robot. Execution of the automatic operation of s24 ends when the operation command of the current step is completed.

In the step s23 of executing the manual operation in the procedure shown in FIG. 7, the switching means 7 is set to the state in which the operation command input from the operating means 2 is transferred to the robot 1. Then, the control data of the joint angle and the control data of the finger of the gripping mechanism corresponding to the operation command input from the operation means 2 are transferred to the robot. In s23, once the execution of the manual operation is started, the state of the manual operation by the operating means 2 is continued. Then, when there is an input from the input device 10 instructing the end of the manual operation, the process ends.

In step s23, the composite image generating means 8 described below displays on the display device 9 a graphic image of the information regarding the positioning of the robot and the object to be positioned. The positioning point to be displayed at this time is specified in the command data for instructing switching to the manual operation.

The composite image generating means 8 generates a composite image by the procedure shown in FIG. First, when switching to the manual operation, the object (target object) to which the positioning point designated by the operation command data is connected is specified (s30). this is,
In the environment model, it is performed by sequentially tracing the connection destinations of the structures corresponding to the positioning points, and finding the structure representing the object that reaches first. For example, when the designated positioning point is approach_point1 (126), the target object is cap (123). Next, the environment model correction means 4 recognizes the position and orientation of the target object (s3
1) The environment model is updated (s32). This is performed by the procedure shown in FIG. 5 already described. Next, the position and orientation of the target object is acquired from the updated environment model, and based on this, the position and orientation coordinates when the mark for manual operation guidance is displayed as a reference figure are calculated (s33). Then draw a reference figure in the form of a mark in that position and orientation,
The image of the actual work environment captured by the camera 3 is displayed in an overlapping manner (s34).

In the above procedure, the following method can be considered as a preferable display method of the reference graphic. For example, as shown in FIG. 9, the hole 14 provided on the object 142
3 approach point (14 when inserting the object 141 into
When displaying the positioning information regarding 4), for example, at the same time as displaying the reference graphic (second mark) 145 on the position and orientation of the gripping mechanism of the robot (or the position and orientation of the object 141 gripped by the gripping mechanism), Object 141 is appr
When the gripping mechanism (or the object 141 gripped by the gripping mechanism) of the robot is positioned at the position of the oach point (144), the reference pattern 145 (first mark) that is paired with the reference pattern 145 is also present in the position and orientation to come. ) Is displayed. Then, the image 14 obtained by the camera 3 capturing these reference patterns
The object 141 can be positioned at a predetermined positioning point by moving the gripping mechanism of the robot by the operation means 2 while viewing the image superimposed on 0 so that the two reference figures overlap.

In this case, the position / orientation at which the reference graphic 145 should be displayed is the position / orientation of the gripping mechanism of the robot. This is calculated by the following procedure. First, for the robot, have a model similar to that shown in FIG. 4,
The position and orientation information of the model is constantly updated based on the joint angle information of the robot so as to match the actual state. Then, it can be obtained by calculating the position and orientation of the gripping mechanism in the model by the procedure already described. On the other hand, for the position and orientation in which the reference graphic 146 should be displayed, the relative position and orientation of the grasp point of the object 141 with respect to the coordinate system of the object 141 is obtained, and this and approach point (14
It can be obtained by multiplying the position and orientation of 4).

The above example is effective when the position and orientation of the gripping mechanism calculated from the model of the robot and the actual position and orientation match, but in general, this is not always the case. For example, when the robot is gripping a heavy object, it is possible that the position and orientation of the gripping mechanism deviate from the position and orientation calculated from the joint angle or the like due to bending or the like. In such a case, the following method is used, for example.

As shown in FIG. 10, the external shape that the object 141 to be positioned will look like when it is positioned is generated as a reference graphic 147, and this is displayed on the image 140 of the actual work environment captured by the camera 3. Display on top of each other. At this time, the reference graphic 147 is displayed by a line drawing such as a wire frame in order to make it easy to understand the degree of overlap with the actual image of the object 141. By moving the gripping mechanism of the robot by the operation means 2 while observing this image and bringing it so that the actual image of the object 141 overlaps the reference pattern 147, the object 141 can be positioned at a predetermined positioning point.

In the above example, the actual position / orientation of the gripping mechanism of the robot is unknown, and the actual position / orientation of the target object is known by the function of the environment model correction means 4. The position and orientation may be known and the position and orientation of the target object may be unknown. This corresponds, for example, to the situation where the environmental model modifying means does not work well. In such a case, the following method is used, for example.

As shown in FIG. 11, the object 141 is positioned and oriented so that the relative positional relationship between the object 141 to be positioned and the target object 142 is the same as the relative positional relationship between the positioning point and the target object. Refer to the external shape of the object 142 Figure 1
The image is generated as 48, and this is displayed by being superimposed on the image 140 of the actual work environment captured by the camera 3. At this time, the reference graphic 148 is displayed by a line drawing such as a wire frame in order to make it easy to understand the degree of overlap with the actual image of the object 142. The object 141 can be positioned at a predetermined positioning point by moving the gripping mechanism of the robot by the operation means 2 while watching this image so that the reference pattern 148 overlaps with the actual image of the object 142.

In the example shown in FIGS. 10 and 11, the reference figure is drawn as a line drawing like a wire frame, but instead of this, a solid having three-dimensional depth information as shown in FIG. The image 149 may be displayed as a reference graphic. In this case, the solid image 149
Is drawn by the following procedure.

First, a distance image of the work environment viewed from the position of the camera 3 is obtained using a device such as a laser scanner. A distance image is an image in which depth information at that point is added to a pixel on the image. Next, the reference figure is drawn on the distance image while calculating the depth, and the pixels of the reference figure are written for the pixels whose depth is before the depth of the distance image, and when the pixel is in the depth, it is captured from the camera 3. Then, write the pixels of the image. In the image obtained as a result of the above, the degree of three-dimensional overlap between the reference figure and the actual object becomes clear, so that the operation of superimposing the image of the reference figure and the actual object becomes easy.

Next, details of the target object selecting means 11 will be described. The target object selecting unit 11 selects the target object selected by the operator from the position information on the screen of the display device 9 input by the operator of the robot using the input device 10 as shown in FIG.
It is specified by the procedure shown in. Here, the input device 10 is
For example, a device such as a mouse or a joystick which is an input device of a computer may be used as a device for inputting a position.

First, the position information input by the input device 10 is read and converted into a position on the screen (s1).
Then, when i is 1 and the object i = 1 is viewed from the position of the camera 3, a region to be drawn is calculated (s2, s3).
Then, it is determined whether or not the position calculated in s1 is included in the region calculated in s3 (s4). If it is included, it is determined that the target object is i = 1, and if not, i is i +
The process proceeds to the next object as 1 (s5), and the same investigation is performed until the target object is specified or until all the objects included in the environment model are completed (s6). If s1
If the positions calculated in step 1 are not included in the drawing areas of all objects, it is determined that there is no target object.

For example, when the scene viewed from the camera 3 is as shown in FIG. 14, the areas in which the objects 130 and 131 or the holes 132 provided on the object 131 are drawn are indicated by thick lines in the figure. It becomes the area surrounded by. At this time, in the object 130, the finger 13 of the gripping mechanism of the robot
The portion hidden by 3 does not become the drawing area of the object 130. Similarly, in the object 131, the part hidden by the object 130 and the part overlapping with the area of the hole 132 are the objects 13 and 13.
It does not become the drawing area of 1. In this situation,
When the position 134 indicated by X is input by the input device 10, the target object is 131.

Next, the overall flow of remote maintenance work using the remote control device according to the present invention will be described. First, the operation procedure of the robot in the maintenance work is created as operation command data and stored in the operation command generation means 6. At this time, for a task that is difficult to automate or a task that requires delicate positioning, such as a task of inserting a bolt into a bolt hole, a command to switch to a manual operation is input at the time of entering the task. With respect to the deviation between the actual work environment and the environment model, for example, an appropriate threshold value for the deviation amount is set, and the actual work environment is recognized by the position and orientation recognition of the object as described above by the environment model correction means 4. If there is a deviation of more than the threshold value between the environment model and the environment model, a command to automatically switch to manual operation is also entered on the condition. The reason for doing this is as follows. That is, in a situation where the magnitude is larger than a certain level, it is predicted that there is an obstacle that may hinder the execution of the automatic driving of the robot even if the deviation can be corrected by the environment model correcting means 4. is there. In other words, by taking such measures, it becomes possible to perform the work by the robot more safely, and in turn, it becomes possible to apply the remote control device to a wider range of maintenance targets and work contents. .

When the robot is set at a predetermined position and the preparation for starting the work is completed, an instruction to start the work is input from the input device 10. Then, the motion command generation means 6 controls the robot 1 based on the motion command data stored in advance to proceed with the work. At this time, the operator monitors the work status of the robot by the video of the work environment captured by the camera 3 displayed on the display device 9.

When the execution step of the operation command data comes to the instruction to switch to the manual operation, the operation command generating means 6
Is in the manual operation mode, and the switching means 7 is the operation means 2
The input is switched to transfer to the robot 1. Further, the composite image generating means 8 superimposes the reference graphic relating to the positioning point designated by the command for switching to the manual operation on the image of the camera 3 and displays it on the display device 9.

Then, the operator operates the robot while looking at this reference pattern to carry out a predetermined work. For example, in the work of inserting a bolt into a bolt hole, positioning information regarding the approach point when inserting the bolt into the bolt hole is displayed in the form of a reference graphic. The operator operates the robot so that the reference pattern and the actual bolt image overlap,
You can take the bolt to the correct position. Therefore, even in the subsequent insertion operation, the work can proceed without hooking the bolt on the side wall of the bolt hole.

When the work by the manual operation is completed, the fact is input from the input device 10, and the manual operation mode is completed. Then, the operation command generation means 6 is again in the automatic operation mode, and restarts execution of the subsequent operation command data.

As described above, the operation which is difficult to automate in the maintenance work or the delicate work is manually operated and the other parts are automatically operated, whereby the operator's burden can be reduced. Further, even in manual operation, the operation guide information that can guide visually the positioning point in the operation is displayed in the form of a reference graphic or the like to facilitate the positioning of the operation target object, so that the manual operation can be performed easily. It can be carried out.

The operator constantly monitors the work status of the robot in the automatic operation on the screen of the display device 9. At this time, if it is determined that there is a possibility that the robot interferes with an unexpected obstacle that is not recorded in the environment model, the input device 10 gives an instruction to stop the automatic operation. Then, the operation command generation means 6 is switched to the manual operation mode similarly to when the operation command generation means 6 receives the instruction to switch to the manual operation. Then, the work is manually operated until the influence of the obstacle disappears, and the operation is switched to the automatic operation again.
At this time, the operation command data is instructed to skip the part that is manually operated.

As described above, when the manual operation is performed while the automatic operation is interrupted, the positioning point where the reference graphic should be displayed is not automatically designated. Therefore, the target object is selected by the target object selection means 11 and the information of the positioning point to be displayed is specified. For example, when the work is interrupted immediately before the work of inserting the bolt into the bolt hole, the object having the bolt hole is selected by the input device 10. As a result, it is possible to display information on the positioning point at which the bolt should be positioned.

In this way, when there is an unexpected obstacle in the work environment, by switching to manual operation as appropriate,
It is possible to deal flexibly. Also in such a case, since information that facilitates positioning of the target object is displayed, a manual operation can be easily performed.

Here, the method of recognizing the position and orientation of the object by the environment model correcting means 4 is not necessarily limited to the method described in the above embodiment, and other methods generally known as the image recognizing method. Can be used.

Further, in the above embodiment, the operation command generating means 6 is provided, but this is not always necessary for the purpose of the present invention. That is, the present invention makes it possible to reduce the burden on the operator during manual operation by displaying a reference pattern or the like. Therefore, even if the operator successively designates the target object and displays all the positioning information regarding the object and all the work is manually performed, the burden on the operator can be reduced.

In the above embodiment, only one camera is provided for capturing the image of the work environment. However, a plurality of cameras may be provided if necessary. When a plurality of cameras are provided, the cameras that capture the images to be displayed on the display device 9 are switched so that when the images of a certain camera are displayed, the reference graphic is displayed with the viewpoint position / orientation corresponding thereto. Due to
Since the object can be viewed from various directions, the operability can be further enhanced. In addition, by inputting images from various directions of the object using a plurality of cameras, it is possible to improve the calculation accuracy of the position and orientation of the object. That is, when the position and orientation of the object are obtained by the procedure shown in FIG. 5, the information regarding the depth direction of the viewpoint appears as a change in the size of the object. Since the change in the size of the object on the image with respect to the change in the position in the depth direction is smaller than the change in the position in the vertical and horizontal directions of the image, the position accuracy in the depth direction is poor in the image from one camera. There is a problem of becoming. On the other hand, when there is a second camera that captures an object from a direction perpendicular to the first camera, the position information in the depth direction in the image of the first camera is moved up and down from the image of the second camera.・ It can be obtained as left-right information. Therefore, by combining these pieces of information, the position and orientation of the object can be specified with higher accuracy than in the case where there is only one camera.

Further, in the above embodiment, the camera is provided at a position distant from the robot 1, but instead of this, the camera may be provided in the gripping mechanism portion of the robot 1. In this case, the reference graphic displayed by the composite image generating means 8 may be as follows. For example, when the positioning information about the approach point (144) when inserting the object 141 into the hole 143 provided on the object 142 as shown in FIG.
The object 142 that should be visible from the camera provided in the gripping mechanism, with 1 being positioned at the positioning point 144.
Alternatively, the shape of the hole 143 is displayed as the reference pattern 150 in the shape shown in FIG. As a result, the actual object 1
When the gripping mechanism is moved so that the images of the holes 42 and the holes 143 overlap the reference pattern, the object 141 is positioned at a predetermined position. In this case, the position / orientation data of the target object and the position / orientation data of the gripping mechanism of the robot are unnecessary, and only the work environment model data is sufficient.

[0069]

As described above, according to the present invention, information relating to the positioning of a robot or an object in a work environment is displayed in a graphic form, and the displayed graphic can be used to guide the manual operation of the robot. Therefore, it becomes easier to operate the robot manually. Therefore, according to the present invention, it is possible to reduce the burden on a person who operates the remote control device. Further, in the present invention, regarding the combination of the automatic operation and the manual operation of the robot, when the environment model which is the basis of the automatic operation of the robot deviates from the actual work environment by a certain amount or more, it is detected and the automatic operation is manually performed. Since the operation can be automatically switched, it is possible to perform the work by the robot more safely. Therefore, according to the present invention, it becomes possible to apply the remote control device to a wider range of maintenance targets and work contents.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a remote control device according to an embodiment.

FIG. 2 is a diagram illustrating a positional relationship between an object and a positioning point in a work environment.

FIG. 3 is a diagram illustrating a relationship between positioning points in an object inserting operation.

FIG. 4 is a diagram illustrating information stored in a work environment model.

FIG. 5 is a diagram illustrating an operation procedure of environment model correction means.

FIG. 6 is a diagram showing an example of an edge image and a reference image used in the environment model correction means.

FIG. 7 is a diagram illustrating an operation procedure of an operation command generation unit.

FIG. 8 is a diagram illustrating an operation procedure of a composite image generation unit.

FIG. 9 is a diagram showing an example of an image generated by a composite image generating means.

FIG. 10 is a diagram showing an example of an image generated by a composite image generating means.

FIG. 11 is a diagram showing an example of an image generated by a composite image generating means.

FIG. 12 is a diagram showing an example of an image generated by a composite image generating means.

FIG. 13 is a diagram illustrating an operation procedure of a target object selecting unit.

FIG. 14 is a diagram showing an example of a drawing area of an object.

FIG. 15 is a diagram showing an example of an image generated by a composite image generating means.

[Explanation of symbols]

1 robot 2 Operation means 3 cameras 4 Environmental model correction means 5 Environmental model 6 Action command generation means 7 Switching means 8 Synthetic image generation means 9 Display device 10 Input device 11 Target object selection means

Claims (12)

[Claims] 1. A remote operation control device for remotely operating and controlling a robot installed in a work environment by an operation control system while allowing the robot to perform a required work by manually operating the robot. Operating means for operating,
A camera that captures a video of the work environment, an environment model memory that stores, as an environment model, information about the position and orientation of an object that is a target of work by the robot in the work environment, information about positioning in the work environment necessary for operating the robot, and the like. Means, a synthetic image generating means for generating a synthetic image by synthesizing an image displayed by visualizing the image captured by the camera and the positioning information obtained from the environment model storing means, and the synthetic image generating means. A composite image is displayed on the display means when the robot is manually operated by the operation means, and the manual operation can be guided. Remote control device. 2. A remote operation control device, wherein a robot installed in a work environment is remotely controlled by an operation control system to allow the robot to perform a necessary work in the work environment. An environment model storage unit that stores, as an environment model, a camera that captures an image of the environment and the position and orientation of an object that is a target of work by the robot in the work environment, information about positioning in the work environment necessary for operating the robot, and the like. With the operation command generation means for generating an operation command for the automatic operation of the robot and the operation means for manually operating the robot, the automatic operation and the manual operation of the robot can be selected. And by comparing the position and orientation of the object in the environment model with the position and orientation of the object in the work environment captured by the camera, A remote operation control device characterized in that, when it is detected that a deviation of a certain amount or more has occurred, the automatic operation is stopped and a manual operation is automatically started. 3. A composite image generation means for generating a composite image by combining a video image captured by a camera and an image displayed by visualizing information on positioning obtained from the environment model storage means, and the composite image generation means. Equipped with a display means for displaying the combined image, and when shifting to manual operation,
The remote operation control device according to claim 2, wherein the composite image is displayed on the display unit so that a manual operation can be guided. 4. An environment model that corrects the environment model so that if there is a deviation between the position and orientation of the object in the environment model and the position and orientation of the object in the work environment captured by the camera, the deviation is eliminated. The remote operation control device according to any one of claims 1 to 3, further comprising a correction unit. 5. An image showing a position / orientation mark corresponding to a positioning point to be positioned in a work environment by a robot being worked or an object being worked by the robot for the purpose of the work The remote operation control device according to any one of claims 1 to 4, which is adapted to be used as a graphic image. 6. A first mark having a position and orientation corresponding to a positioning point to be positioned in a work environment for a working robot or an object being worked by the robot, and the robot or the object. 5. An image displaying a second mark of a position and orientation corresponding to a position and orientation that changes with the progress of work is used as a graphic image of information relating to positioning. A remote control device according to the item. 7. An external shape graphic of an object in a work environment undergoing work by a robot, which would be visible when positioned at a positioning point to be positioned in the work environment for the purpose of the work, is displayed. The remote operation control device according to any one of claims 1 to 4, wherein the image is used as a graphic image of information regarding positioning. 8. The target object is combined with the object to be positioned in the state of being operated by the robot when the robot is made to combine the object to be positioned in the work environment with another target object. The remote operation according to any one of claims 1 to 4, wherein an image displaying a figure of an external shape of the target object that may be seen when used is used as a graphic image of information regarding positioning. Control device. 9. A camera is provided which enables a robot to grasp an object to be positioned while grasping the object to be positioned, and a graphic of information relating to positioning in an image captured by the camera. 9. The remote operation control device according to claim 8, wherein the remote control device is configured to superimpose the digitized images to generate a composite image. 10. The remote according to claim 7, wherein the image of the work environment captured by the camera and the graphic image of information on positioning are overlapped in consideration of the depth to generate a composite image. Operation control device. 11. An input means for inputting a position on a screen of a display means, and a target object selecting means for selecting one object in a work environment as a target object from information on the position input by the input means, When the position is input on the screen of the display unit by the input unit, a graphic image of information regarding positioning regarding the target object selected by the target object selection unit is generated. The remote control device according to claim 10. 12. A plurality of cameras for capturing an image of a work environment are provided, the images of the work environment captured by these cameras can be switched and displayed on a display unit, and positioning can be performed according to the switching of the display image. The remote operation control device according to any one of claims 1 to 8 or 10 to 11, wherein a method of displaying a graphic image of information regarding the information can be switched.