JP2007049490A - Remote indication system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote indication system in which the operability of a remote terminal is improved. <P>SOLUTION: The remote indication system is for transmitting a picked up image of a subject 200 by a video camera 20 to the remote terminal 100 and having a projector 40 project an annotation image following the indication given by the remote terminal 100 to the subject 200, and is provided with a reference image accumulation means for extracting a reference image for reference in the remote terminal 100 from the picked up images of the video camera 20 to accumulate it in a database 30. Thus, the operability of the remote terminal can be improved. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a remote instruction system and method capable of remotely instructing an image.

For example, in a remote repair system, a remote maintenance system, a telemedicine system, a remote conference system, etc., it is necessary to give various instructions such as instructions for work procedures from the remote terminal side to the real object side. As such a remote instruction system capable of instructing from the remote terminal side to the real object side, for example, while capturing a subject existing on the real object side with a video camera, the captured image is transmitted to the remote terminal, A technique is known in which an annotation image instructed based on a captured image at a remote terminal is projected onto a subject by a projector on the real object side (see, for example, Patent Document 1).
US Patent Publication No. 2004/0070674

By the way, in the remote instruction system as described above, in order to change the imaging range of the video camera according to an instruction from the remote terminal, an instruction to change the imaging position of the subject is given to an operator or an observer existing on the subject side. There is a need.
For this reason, it takes time to obtain a captured image after changing the imaging position of the subject on the remote terminal side, and it has been difficult to perform a smooth instruction operation.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a remote instruction system with improved operability of a remote terminal.

A remote instruction system according to the present invention is a remote instruction system that transmits a captured image of a subject by an imaging unit to a remote terminal and projects an annotation image according to an instruction given by the remote terminal onto the subject by a projecting unit. Reference image storage means for extracting a reference image for reference at a remote terminal from the captured image of the means and storing it in the storage means is provided.
According to this configuration, when it is desired to change the imaging position of the subject, an image captured in the past by the imaging unit can be used at each remote terminal as alternative image information for the actual captured image.

In the above configuration, the reference image may include a captured image of the subject on which the annotation image is projected.
According to this configuration, when an annotation image projected in the past is desired to be confirmed again, it is possible to refer to the annotation image using the stored reference image.

The above configuration further includes a relative position changing unit that is controlled in accordance with an instruction from the remote terminal to change the relative position of the subject with respect to the projection unit and the imaging unit, and the image storage unit changes the relative position of the subject. It is possible to adopt a configuration in which a reference image is extracted and stored in a storage unit every time.
According to this configuration, the imaging position of the subject can be changed from the remote terminal side, and a reference image at each relative position can be formed.

In the above configuration, the image storage means may be configured to store the reference image after the remote terminal issues a command to change the relative position of the subject and before the relative position of the subject is changed.
According to this configuration, since the reference image of the subject is automatically accumulated, it is possible to reduce the operation burden on the person who operates the remote terminal.

In the above configuration, the image storage means may employ a configuration in which the relative position information of the subject is added to the reference image and stored.
According to this configuration, the captured reference image can be searched based on the side surface position information.

In the above configuration, the image storage means can adopt a configuration in which imaging time information is added to the reference image and stored.
According to this configuration, it is possible to refer to images accumulated along the imaging history.

In the above configuration, the image storage means can adopt a configuration in which information relating to the remote terminal that has instructed to project the annotation image is added to the reference image and stored.
According to this configuration, for example, when a plurality of remote terminals access a server through the Internet, it is possible to search for a reference image stored from a domain name or the like.

In the above configuration, the image storage means can adopt a configuration in which a reference image is extracted and stored every time an annotation image is projected.
According to this configuration, a history of annotation images can be formed, and a desired annotation image can be searched and referenced.

In the above configuration, the image storage means may be configured to store the reference image according to an instruction from the remote terminal.
According to this configuration, it is possible to selectively accumulate only the images necessary for the person who operates the remote terminal.

The remote instruction method according to the present invention transmits a captured image of a subject by an imaging unit to a remote terminal and performs remote instruction by projecting an annotation image according to the instruction made by the remote terminal onto the subject by a projecting unit. A method is characterized in that a reference image for reference at a remote terminal is extracted from a captured image of an imaging unit and stored in a storage unit.
According to this configuration, when it is desired to change the imaging position of the subject, an image captured in the past by the imaging unit can be used at each remote terminal as alternative image information for the actual captured image.

  According to the present invention, it is possible to refer to stored images captured in the past from the remote terminal, and thus the operability of the remote terminal can be improved.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, exemplary embodiments of the invention will be described with reference to the accompanying drawings.
FIG. 1 is a block diagram showing an embodiment of a remote instruction system of the present invention, and shows a case where a database is arranged on the server side.
As shown in FIG. 1, the remote instruction system includes a subject-side device 10, a remote terminal 100, and the like, and the subject-side device 10 and the remote terminal 100 are connected to each other via a network 300 so that they can communicate with each other. Although only one remote terminal 100 is shown in FIG. 1, a plurality of remote terminals 100 can be connected to a later-described server 50 of the subject side apparatus 10 through the network 300.

  The subject side apparatus 10 includes a video camera 20 as an imaging unit, a database 30, a projector 40 as a projection unit, a server 50, a half mirror 60, a rotary table 70 as a relative position changing unit, and the like (hereinafter simply referred to as a “positioning unit”) It is described as table 70).

  The video camera 20 is composed of, for example, a CCD camera and is arranged so as to be able to capture an image of the subject 200 placed on the table 70, and the captured image is taken into the server 50. Note that the video camera 20 captures an image transmitted through the half mirror 60.

  The database 30 is composed of, for example, a relational database, and the server 50 captures an image captured by the video camera 20, stores the captured image as a reference image, and the stored image designated by the remote terminal 100 is stored. Is transmitted to the remote terminal 100.

  The projector 40 is composed of a liquid crystal projector or the like, and is arranged so that the optical principal points of the video camera 20 are substantially matched by the half mirror 60. The projector 40 projects the annotation image transmitted from the server 50 onto the subject 200 via the optical system and the half mirror 60. The annotation image is reflected by the half mirror 60 and projected onto the subject 200.

  The table 70 is controlled by the server 50 to change the position of the subject 200 relative to the video camera 20 and the projector 40 by rotating (moving) the subject 200 placed on the upper surface thereof.

The server 50 controls the operations of the video camera 20, the database 30, the projector 40, and the table 70, and exchanges various information with the remote terminal 100 via the network 300. In addition, the server 50 transmits the captured image of the video camera 20 to the remote terminal 100 and causes the projector 40 to project an annotation image corresponding to an instruction based on the captured image made by the remote terminal 100 onto the subject 200.
Further, the server 50 performs control to change the projection position and imaging position of the annotation image with respect to the subject 200 of the projector 40 in accordance with the rotation of the subject 200 by the table 70.
Further, the server 50 accumulates the reference image in the database 30 every time it is instructed by the remote terminal 100 to project the annotation image, rotate the table 70, and accumulate the reference image.

  The remote terminal 100 includes a liquid crystal display device, a display device 110 such as a CRT, a computer 120 connected to the network 300, a mouse 130 connected to the computer 120, and the like.

  The display device 110 displays an image transmitted from the subject side device 10 on the display screen.

  The mouse 130 operates various buttons and the like on the display screen 111 on which the reference image is displayed, thereby forming instructions regarding the annotation image to be projected onto the subject 200 and rotating the subject 200 by rotating the table 70. It is used to instruct to move, to instruct to display the reference image stored in the database 30, and to instruct the database 30 to store the captured image as a reference image.

Next, the operation of the remote instruction system configured as described above will be described with reference to the flowcharts shown in FIGS. 2 and 3 and FIG.
Here, FIG. 2 is a flowchart showing an example of processing of the server, FIG. 3 is a flowchart showing an example of processing in the computer, and FIG. 4 is a diagram for explaining an example of operation in the remote terminal.

  As shown in FIG. 2, the server 50 on the subject apparatus 10 side projects a preset initial projection image by the projector 40, starts capturing the captured image of the video camera 20, and sends it to the computer 120 on the remote terminal 100 side. The regular transmission of the captured image is started (step ST1).

  Then, the command sent by the computer 120 on the remote terminal 100 side is received (step ST2). Thereafter, it is determined whether the received command is an instruction to draw an annotation image (step ST3). If the instruction is to instruct drawing, step ST4 is executed; otherwise, step ST6 is executed.

If the server 50 determines in step ST3 that the received command is a drawing command, the server 50 executes processing for projecting the annotation image onto the subject 200 in accordance with the drawing command (step ST4). Next, the imaging is stored in the database 30 as a reference image (step ST5).
By acquiring information of the remote terminal 100 that has instructed to project an annotation image from a drawing command or the like, it is possible to add the information and accumulate a reference image in step ST5. Further, by using the system time of the server 50 or the like, it is possible to add the imaging time information and accumulate the reference image. The position information of the table 70 can be added. Further, each time an annotation image is projected onto the subject 200, the captured image of the subject 200 can be stored in the database 30 as a reference image.

  Next, it is determined whether or not there is a termination request (step ST6). If there is a termination request, the process is terminated. If not, the process returns to step ST2 to repeat the above process.

  If the server 50 determines in step ST3 that the received command is not a drawing command, the server 50 determines whether the received command is a table control command (step ST6). If the received command is a table control command, step ST7 is executed. Otherwise, step ST9 is executed.

Next, when it is determined in step ST6 that the received command is a table control command, the imaging of the subject 200 is stored in the database 30 as a reference image (step ST7). Then, the rotation process of the table 70 according to a control command is performed (step ST8), and the process of step ST6 is performed.
Thus, each time the relative position of the subject 200 is changed and after the remote terminal 100 issues a command for changing the relative position, the reference image is stored before the relative position of the subject 200 is changed. it can. Further, as shown in the description of step ST5, it is possible to add information on the remote terminal 100 that has instructed to project the annotation image, imaging time information, and position information on the table 70.

If the server 50 determines in step ST6 that the received command is not a table control command, the command is an accumulation command, and the captured image is accumulated in the database 30 as a reference image (step ST9). .
Thereby, it is possible to create a projection history of an annotation image in which only images necessary for a person who operates the remote terminal 100 are selectively accumulated. Thereafter, step ST6 is executed.

Next, processing in the computer 120 shown in FIG. 3 will be described.
When the operator operates various buttons of the display device 110 with the mouse 130 or the like, the computer 120 receives a command (step ST21).
Specifically, the operator of the remote terminal 100 operates while viewing the image on the display screen 111 of the display device 110 as shown in FIG.

When projecting the annotation image onto the subject 200, the operator operates the annotation designation button BT formed on the display screen 111 to draw the annotation image to be projected. Further, when changing the imaging position of the subject 200, the rotation direction designation button R and the rotation amount designation frame AG are operated to instruct the rotation direction and the rotation amount of the table 70. In order to confirm whether the stored image to be referred to exists in the database 30, the history display button H is operated.
The operator operates the accumulation button S formed on the display screen 111 in order to selectively accumulate only necessary images. By this operation, unlike the above, only images necessary for the operator can be selectively accumulated as a history.
In addition, by these operator operations, drawing, table control, and storage commands are respectively input to the computer 120 in step ST21.

If there is a reference image 114 to be displayed on the main screen 113 among the images stored in the database 30, the pointer Pt on the display screen 111 is moved by operating the pointing device 130. Thereby, the reference image 114 can be selected. The computer 120 displays the selected reference image 114 on the main screen 113.
Since the reference image 114 can be used as alternative image information of the actual captured image, the necessity of rotating the table 70 is reduced. In particular, when a plurality of remote terminals 100 are connected to the server 50 of the subject side device 10, the occurrence of contention for the control right of the table 70 is reduced.

Here, on the real-time screen 112, the state of operation by the remote terminal 100 designating the current imaging area is displayed. Further, when the state of the operation displayed on the real-time screen 112 is displayed on the main screen 113, the pointer Pt is moved onto the real-time screen 112 by operating the pointing device 130. As a result, the real-time screen 112 disappears, and the state of the operation displayed on the real-time screen 112 can be displayed on the main screen 113.
Further, in the remote terminal 100 designating the imaging region, the real-time screen 112 is not displayed on the display screen 111, and the state of operation by the remote terminal 100 is displayed on the main screen 113.

  The computer 120 determines whether the input command is a reference image display command (step ST22). In the case of a reference image display command, the process of step 23 is executed. Otherwise, the process of step ST26 is executed.

Next, when it is determined that the input command is a reference image display command, a reference image is inquired and acquired (step ST23). Thereafter, the acquired reference image is drawn (step ST24).
Thereby, it is possible to display the reference images accumulated in the past.

  Thereafter, it is determined whether or not there is a termination request (step ST25). If there is a termination request, the process is terminated. If not, the process returns to step ST21 to repeat the above process.

  If the computer 120 determines in step ST22 that the input command is not a reference image display command, the computer 120 determines whether the input command is a drawing command (step ST26). If it is a drawing command, step ST27 is executed. If not, there is a table control or accumulation command, and step ST28 is executed.

  Next, when it is determined in step ST26 that the input command is a drawing command, image drawing processing is performed (step ST27).

  The computer 120 performs processing for sending a command to the server 50 (step ST28). At this time, by adding information for the remote terminal 100 and sending the information to the server 50, the information of the remote terminal 100 instructed to project the annotation image can be added and the reference image can be stored in the database 30. .

  As described above, according to the present embodiment, the camera 20 captures the subject 200 on which the annotation image is projected, and the server 50 extracts the reference image from the captured image and stores it in the database 30. It is possible to confirm the annotation projected on the screen.

  Since the server 50 operates the table 70 in accordance with the table control command received from the remote terminal 100, the imaging position of the subject 200 can be changed from the remote terminal side 100. Further, a reference image can be referred to instead of actually changing the imaging position.

  Since the server 50 automatically accumulates the reference image in the database 30 every time the table 70 is rotated and each time the annotation image is projected, the operation burden on the person who operates the remote terminal 100 can be reduced.

  Furthermore, since the reference image is accumulated by adding the domain name, time, and relative position of the table, the domain name, the time range, and the relative position can be specified and the reference image can be referred to according to the search or history. .

  Further, since the operator operates the accumulation button S to input a command for accumulating the reference image in the database 30 to the computer 120, only the image necessary for the operator can be selectively accumulated as a history.

  FIG. 5 is a block diagram showing another embodiment of the present invention. 5 and 6, the same reference numerals are used for the same components as those in the above configuration. This configuration differs from the system shown in FIG. 1 in that the database 140 is arranged in a computer on the remote terminal 100 side.

Next, an example of the operation of the remote instruction system having the above configuration will be described with reference to the flowchart shown in FIG.
Here, FIG. 6 is a flowchart showing an example of processing in the server, and FIG. 7 is a flowchart showing an example of processing in the computer.

  As shown in FIG. 6, the server 50 projects a preset initial projection image by the projector 40, starts capturing the captured image of the video camera 20, and constantly transmits the captured image to the computer 120 on the remote terminal 100 side. Start (step ST31).

  Then, server 50 receives the command transmitted by computer 120 on remote terminal 100 side (step ST32). It is determined whether the received command is a command for instructing rotation of the table 70 (step ST33). If the command is an instruction to turn, the process of step ST34 is executed, and if not, the process of step ST36 is executed.

Next, when it is determined in step ST33 that the received command is a command for instructing the rotation of the table 70, the table 70 is rotated in accordance with the control command (step ST34).
Next, it is determined whether or not there is an end request (step ST35). If there is an end request, the process ends. If not, the process returns to step ST32 to repeat the above process.

  If the server 50 determines in step ST33 that the received command is not a command for instructing rotation of the table 70, the received command is a command for instructing drawing of the annotation image, and the annotation image in accordance with the drawing command is displayed. Projection processing is performed (step ST36). Thereafter, a drawing synchronization command is sent to the remote terminal (step ST37). Here, the drawing synchronization command is a command sent from the server 50 to the remote terminal 100, and is used to notify the remote terminal side that the projection processing of the annotation image by the server 50 has been completed. In addition, by adding the information of the remote terminal 100 that has instructed to project the annotation image to the drawing synchronization command or the like, the information can be added at the command receiving destination and the reference image can be accumulated. Then, the process of step ST35 is performed.

Next, processing in the computer shown in FIG. 7 will be described.
The operator of the remote terminal 100 inputs a command from the display device 110 to the computer 120 with the mouse 130 or the like, or the computer 120 receives the command sent from the server 50 (step ST41).

Next, the computer 120 determines whether the received command is a reference image display command (step ST42). If it is a reference image display command, the process of step ST43 is performed. Otherwise, the process of step ST45 is performed.

  Next, if it is determined in step ST42 that the received command is a reference image display command, a process of drawing the reference image stored in the database 140 is executed (step ST43). Thereby, the reference images accumulated in the past can be displayed.

  Next, it is determined whether or not there is an end request (step ST44). If there is an end request, the process is terminated. If not, the process returns to step ST41 and the above process is repeated.

  If the computer 120 determines in step ST42 that the received command is not a reference image display command, the computer 120 determines whether the received command is a command for controlling the table 70 (step ST45). If it is a table control command, the process of step ST46 is performed, and if not, the process of step ST48 is performed.

Next, when it is determined in step ST45 that the received command is a table control command, the imaging of the subject 200 received on the remote terminal 100 side is stored in the database 140 as a reference image (step ST46).
Thereby, the reference image of the subject 200 can be stored in the database 140 in accordance with an instruction from the remote terminal 100. It is also possible to add information related to the remote terminal 100 that has instructed to project an annotation image, imaging time information, and position information of the table 70 as described above.
Next, the command is sent to the server 50 (step ST47). Thereafter, step ST44 is executed.

  If the computer 120 determines in step ST45 that the received command is not a table control command, the computer 120 determines whether the received command is a drawing command (step ST48). If it is a drawing command, the process of step ST49 is performed. Otherwise, the process of step ST50 is performed.

  Next, when it is determined in step ST48 that the received command is a drawing command, an image drawing process is executed (step ST49). Next, the process of step ST47 is executed.

  If it is determined in step ST48 that the received command is not a drawing command, it is determined whether the received command is a drawing synchronization command (step ST50). If it is a drawing synchronization command, the process of step ST51 is performed, and if not, the process of step ST53 is performed.

If the computer 120 determines in step ST50 that the received command is a drawing synchronization command, the computer 120 executes synchronization processing (step ST51). Here, it is confirmed that the remote terminal 100 has received the captured image of the subject 200 onto which the annotation image has been projected. Thereafter, the captured image is stored in the database 140 as a reference image (step ST52). Thus, each time an annotation image is projected onto the subject 200, the reference image can be stored in the database 140. Thereafter, step ST44 is executed.
In step ST52, by acquiring the information of the remote terminal 100 that has instructed to project the annotation image from the drawing synchronization command or the like, it is possible to add the information and store the reference image. Further, by using the system time of the computer 120 or the like, it is possible to add the imaging time information and accumulate the reference image. It is also possible to add position information of the table 70.

If the computer 120 determines in step ST50 that the received command is not a drawing synchronization command, the received command is an accumulation command, and the captured image is accumulated in the database 140 as a reference image (step ST53).
Thereby, it is possible to create a projection history of an annotation image in which only images necessary for a person who operates the remote terminal 100 are selectively accumulated. It is also possible to add information on the remote terminal 100 that has instructed to project an annotation image, imaging time information, and position information on the table 70 as described above.

As described above, according to the present embodiment, since the database 140 is connected to the computer 120, the time required for data exchange between the computer 120 and the database 140 is affected by the line quality and traffic volume of the network 300. There is no.
In addition, the difference between the image displayed on the display device 110 when the operator operates the accumulation button S and the reference image accumulated in the database 140 is unlikely to occur.
Further, the same effect as that of the above embodiment can be obtained.

  In the above embodiment, the case where a projector is used as the projecting unit has been described. However, the present invention is not limited to this. For example, an image can be formed by irradiating a subject with light such as laser light.

  In the above-described embodiment, the case where a CCD camera is used as the imaging unit has been described. However, the present invention is not limited to this. For example, a camera using a CMOS image sensor or the like can be employed.

  In the above embodiment, the case where the rotation direction designation button is formed on the display screen 111 in order to instruct the rotation of the turntable 70 has been described. However, the present invention is not limited to this, and a keyboard may be used. Various methods can be employed.

  Although the case where the remote terminal 100 is connected to the server 50 via the network 300 has been described in the above embodiment, the present invention is not limited to this, and the remote terminal 100 can be connected to the server 50 by other means. In addition, the remote terminal 100 can be arranged on the subject 200 side.

  In the above-described embodiment, the case where the subject side is rotated by the table 70 has been described. However, the present invention is not limited to this. For example, a robot, an XY table, or the like can be used as the relative position changing unit.

  In the above embodiment, an example in which the subject is rotated by the table 70 as the relative position changing unit has been described. However, the present invention is not limited to this, and the video camera 20, the projector 40, and the half mirror 60 rotate about the subject. A configuration is also possible.

  In the above embodiment, the remote terminal 100 issues a command to change the relative position of the subject 200 every time the relative position of the subject 200 is changed, every time the relative position of the subject 200 is changed, and each time an annotation image is projected onto the subject. Although the description has been made before the relative position of the subject is changed after issuance, the present invention is not limited to this, and an image can be periodically accumulated.

It is a block diagram of the remote instruction | indication system which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the process in a server. It is a flowchart which shows an example of the process in a computer. It is a figure for demonstrating an example of operation in a remote terminal. It is a block diagram of the remote instruction | indication system which concerns on other embodiment of this invention. It is a flowchart which shows an example of the process in a server. It is a flowchart which shows an example of the process in a computer.

Explanation of symbols

10 ... Subject side device 20 ... Video camera (imaging means)
30 ... Database 40 ... Projector (projection means)
50 ... Server 60 ... Half mirror 70 ... Table (relative position changing means)
100 ... Remote terminal 110 ... Display device 111 ... Display screen 112 ... Real time screen 113 ... Main screen 114 ... Reference image 120 ... Computer 130 ... Mouse 140 ... Database 200 ... Subject 300 ... Network DR ... Annotation image Pt ... Pointer BT ... Annotation Designation button R ... Rotation direction designation button AG ... Rotation amount designation frame H ... History display button S ... Accumulation button

Claims (10)

A remote instruction system that transmits a captured image of a subject by an imaging unit to a remote terminal and projects an annotation image according to an instruction made at the remote terminal onto the subject by a projecting unit,
Reference image storage means for extracting a reference image for reference at the remote terminal from the captured image of the imaging means and storing it in the storage means,
A remote indication system characterized by that. The reference image includes a captured image of the subject onto which the annotation image is projected.
The remote indication system according to claim 1. A relative position changing unit that is controlled according to an instruction from the remote terminal and changes a relative position of the subject with respect to the projection unit and the imaging unit;
3. The remote instruction system according to claim 1, wherein the image storage unit extracts the reference image and stores it in the storage unit every time the relative position of the subject is changed. The image storage means stores the reference image after the remote terminal issues a command to change the relative position of the subject and before the relative position of the subject is changed.
The remote instruction system according to claim 3. The image storage means adds the relative position information of the subject to the reference image and stores the information;
The remote indication system according to claim 3 or 4, wherein The image storage means adds imaging time information to the reference image and stores it.
The remote indication system according to any one of claims 1 to 5, wherein The image storage means adds information related to the remote terminal that has instructed to project the annotation image to the reference image and stores the information;
The remote indication system according to any one of claims 2 to 6, wherein The image storage means extracts and stores the reference image for each projection of the annotation image;
The remote indication system according to any one of claims 1 to 7. The image storage means stores the reference image in accordance with an instruction from a remote terminal;
The remote indication system according to any one of claims 1 to 7. A remote instruction method for transmitting a captured image of a subject by an imaging unit to a remote terminal and performing a remote instruction by projecting an annotation image according to an instruction made at the remote terminal onto the subject by a projecting unit,
Extracting a reference image for reference in the remote terminal from the captured image of the imaging unit and storing it in the storage unit;
A remote instruction method characterized by that.

Images