JP6451511B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  In order to support a worker who performs work on the site, there is a system in which a supporter such as an expert can transmit a voice instruction to the worker while browsing a moving image of the work site at a remote place.

  Since both hands are required depending on the work contents, the operator needs to keep both hands free. In such a case, the camera for capturing an image of the work site is attached to a part of the worker's body. For example, in order to make the direction of the operator's line of sight and the imaging direction of the camera coincide as much as possible, a camera is mounted next to the eyes of the operator.

JP 2000-231625 A JP 2004-236020 A

  However, when the camera is mounted next to the eyes of the worker, the position of the lens of the camera has a deviation from the viewpoint of the worker. Therefore, for example, when the worker points at an object that is out of the reach of the worker, the supporter may not be able to correctly recognize the object pointed by the worker from the image. There is.

  FIG. 1 is a diagram for explaining a problem in the case where the position of the camera has a deviation from the operator's viewpoint. As shown in (1) of FIG. 1, in the situation where the camera is mounted next to the operator's eyes, when the operator points to an object o1 located far away, the operator's finger is From the direction, it is located in the direction indicated by the dashed arrow. As a result, the operator's finger is displayed at a position shifted from the object o1 on the screen viewed by the supporter as shown in (2).

  In view of the above, an object of one aspect is to make it possible to specify an indication position intended by an instructor on an image obtained by imaging an indication object that points away from the object and indicates the object.

  In one plan, the information processing apparatus includes: a receiving unit that receives an image of an object and an instruction object that points away from the object and points to the object; The relative position in the real space of the specifying unit for specifying the position in the image, the viewpoint of the instructor who instructs the target using the pointing object, the imaging unit, the pointing object, and the target Based on information indicating a relationship, a calculation unit that calculates a deviation of the pointing object in the image with respect to the target, and a position having the deviation with respect to the position of the pointing object with respect to the image And an output unit that outputs an image to which a predetermined symbol is added within a predetermined range.

  According to one aspect, it is possible to specify an indication position intended by an instructor on an image obtained by imaging an indication object that points away from the object and indicates the object.

It is a figure for demonstrating a problem in case the position of a camera has a shift | offset | difference with respect to an operator's viewpoint. It is a figure which shows the structural example of the work assistance system in embodiment of this invention. It is a figure which shows the hardware structural example of the work assistance server in embodiment of this invention. It is a flowchart for demonstrating an example of the process sequence which a work assistance server performs. It is a figure which shows the structural example of a parameter memory | storage part. It is a figure for demonstrating each parameter. It is a figure for demonstrating an example of the calculation method of an error. It is a flowchart for demonstrating an example of the process sequence which a supporter terminal performs. It is a figure which shows the example of a display of the image to which the range circle was added.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 2 is a diagram illustrating a configuration example of the work support system according to the embodiment of the present invention. In FIG. 2, the work support system 1 includes a worker terminal 20, a work support server 10, a supporter terminal 30, and the like. The work support server 10 is communicably connected to the worker terminal 20 and the supporter terminal 30 via a network.

  The worker terminal 20 is a terminal worn by the worker. An operator means a person who performs work at a work site. In the present embodiment, an example in which the worker terminal 20 is fixed to the worker's head will be described. However, the worker terminal 20 may be attached to another part of the worker's body.

  The worker terminal 20 includes a distance measuring unit 21, an imaging unit 22, a work image transmitting unit 23, and the like. The imaging unit 22 is a camera that captures a moving image including an object (hereinafter, referred to as “target object”) that the operator is paying attention to. The distance measuring unit 21 is a distance measuring sensor that measures the distance between the object and the operator terminal 20 (that is, the imaging unit 22) using, for example, infrared rays. The distance between the object and the imaging unit 22 refers to the distance between the two points when the two points of the object and the imaging unit are projected on a straight line connecting the object and the viewpoint of the worker.

  The work image transmission unit 23 includes an image captured by the imaging unit 22, information indicating a distance measured by the distance measurement unit 21 (hereinafter referred to as “distance information”), and identification information of the worker terminal 20. It transmits to the work support server 10. In the present embodiment, an example in which the image captured by the imaging unit 22 is a moving image will be described, but the image may be a still image. Further, the identification information of the worker terminal 20 can be used as worker identification information. Therefore, in the present embodiment, the identification information is referred to as “worker ID”. The worker ID is stored in the worker terminal 20, for example.

  The work support server 10 is one or more computers that relay a moving image captured by the imaging unit 22 of the worker terminal 20 to the supporter terminal 30. The work support server 10 includes a work image receiving unit 11, an instruction position specifying unit 12, an error calculating unit 13, an image relay unit 14, a correction value receiving unit 15, and the like. Each of these units is realized by processing that one or more programs installed in the work support server 10 cause the CPU of the work support server 10 to execute. The work support server 10 also uses the parameter storage unit 16. The parameter storage unit 16 can be realized using, for example, an auxiliary storage device of the work support server 10 or a storage device that can be connected to the work support server 10 via a network.

  The work image receiving unit 11 receives a moving image and distance information transmitted from the worker terminal 20. The work image receiving unit 11 stores the received distance information in the parameter storage unit 16. The designated position specifying unit 12 extracts the designated object of the target object from the moving image from the worker terminal 20 and identifies the position of the designated object in the moving image. The designated position specifying unit 12 inputs a coordinate value indicating the specified position to the error calculating unit 13.

  Based on the parameter group stored in the parameter storage unit 16 and the coordinate value input from the indicated position specifying unit 12, the error calculating unit 13 determines the position of the indicated object on the moving image. An error from the designated position (that is, a deviation from the position of the object on the moving image) is calculated. The parameter storage unit 16 stores a parameter group that can specify the relative positional relationship among the operator's viewpoint, the imaging unit 22, the pointing object, and the target in real space.

  The image relay unit 14 transfers the moving image received from the worker terminal 20 to the supporter terminal 30. The image relay unit 14 does not simply transfer the moving image to the worker terminal 20, but applies the worker to the moving image received from the worker terminal 20 based on the error calculated by the error calculation unit 13. A symbol indicating the intended pointing position is added.

  The correction value receiving unit 15 receives a correction value input by the supporter from the supporter terminal 30 for the validity of the error calculation result. The correction value is a parameter for correcting the position of the added symbol based on the error.

  The supporter terminal 30 is a terminal used by the supporter. The supporter refers to a person who remotely supports the work performed by the worker while browsing the moving image transferred from the work support server 10. The supporter terminal 30 may be a PC (Personal Computer), a tablet terminal, a smartphone, or the like. In FIG. 2, the supporter terminal 30 includes a display control unit 31, a correction value receiving unit 32, a correction value transmission unit 33, and the like. Each of these units is realized by a process in which one or more programs installed in the supporter terminal 30 are executed by the CPU of the supporter terminal 30.

  The display control unit 31 controls the display of the supporter terminal 30 on the display device for the moving image transferred from the work support server 10. The correction value receiving unit 32 receives an input of a correction value for the position of the symbol indicating the designated position intended by the operator, which is added to the moving image, from the supporter. For example, the supporter can grasp the designated position intended by the worker based on communication (for example, conversation) using voice or the like with the worker. The supporter inputs a correction value for displaying a symbol at the indicated position grasped in this way. The correction value transmission unit 33 transmits the correction value received by the correction value reception unit 32 to the work support server 10.

  In FIG. 2, a mechanism for relaying voice from the supporter terminal 30 to the worker terminal 20 is omitted, but instructions to the worker by the supporter are transmitted to the worker terminal 20 by voice. Is done. The voice transmission may be relayed by the work support server 10 or may be relayed by a telephone communication network. Further, an image (moving image) instead of a voice may be transmitted from the supporter terminal 30 to the worker terminal 20.

  The worker support system 1 may include a plurality of worker terminals 20 and a plurality of supporter terminals 30. Each supporter terminal 30 may have the function of the work support server 10.

  FIG. 3 is a diagram illustrating a hardware configuration example of the work support server according to the embodiment of the present invention. The work support server 10 in FIG. 3 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, and the like that are mutually connected by a bus B.

  A program that realizes processing in the work support server 10 is provided by the recording medium 101. When the recording medium 101 on which the program is recorded is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program need not be installed from the recording medium 101 and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 executes a function related to the work support server 10 according to a program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

  An example of the recording medium 101 is a portable recording medium such as a CD-ROM, a DVD disk, or a USB memory. An example of the auxiliary storage device 102 is an HDD (Hard Disk Drive) or a flash memory. Both the recording medium 101 and the auxiliary storage device 102 correspond to computer-readable recording media.

  Hereinafter, a processing procedure executed in the work support system 1 will be described. FIG. 4 is a flowchart for explaining an example of a processing procedure executed by the work support server.

  In step S <b> 101, the work image receiving unit 11 receives the worker ID, the moving image, and the distance information transmitted from the worker terminal 20. Subsequently, the work image receiving unit 11 stores the distance information in the parameter storage unit 16 in association with the worker ID (S102). Subsequently, the designated position specifying unit 12 tries to detect a finger from the moving image (S103). In the present embodiment, an example in which the pointing object of the target object is the operator's finger will be described. However, an object other than a finger, such as a pointing stick, may be the pointing object. The detection of the pointing object in the moving image may be performed using a known image processing technique.

  When a finger is detected (Yes in S103), the designated position specifying unit 12 specifies the position of the finger on the moving image, and inputs a coordinate value indicating the specified position to the error calculating unit 13 (S104). ).

  Subsequently, the error calculation unit 13 indicates a relative group of parameters stored in the parameter storage unit 16 (the worker's viewpoint in real space, the imaging unit 22, the pointing object, and the target object). Based on (information), an error between the coordinate value of the finger position on the moving image and the designated position intended by the operator is calculated (S105).

  FIG. 5 is a diagram illustrating a configuration example of the parameter storage unit. In FIG. 5, the parameter storage unit 16 stores a name, parameters p1 to p5, a correction value, and the like for each actor ID.

  The name is the name of the worker corresponding to the worker ID. The parameters p1 to p5 will be described with reference to FIG.

  FIG. 6 is a diagram for explaining each parameter. In FIG. 6, the names (p1 to p5) of the parameters are assigned to the symbols representing the parameters.

  The parameter p1 indicates the distance from the operator's viewpoint to the fingertip. For example, the parameter p1 may be obtained by measuring the length of the operator's arm. The parameter p2 is the distance between the centers of the operator's eyes (pupils). The parameter p3 is the distance between the reference position of the imaging unit 22 (for example, the center of the lens (optical axis) of the imaging unit 22) and the center of the eye (pupil) closer to the imaging unit 22. The distance is, for example, on a plane in which two points of the reference position of the imaging unit 22 and the center of the eye (pupil) closer to the imaging unit 22 are orthogonal to a straight line connecting the operator's viewpoint and the object o2. This is the distance between the two points when projected.

  The parameter p4 is a distance between the worker's viewpoint and the reference position of the imaging unit 22. In the form of the main character S, the center between the eyebrows of the worker is regarded as the worker's viewpoint. Therefore, the relationship p4 = p2 ÷ 2 + p3 is established. Therefore, parameter p4 does not need to be set when parameters p2 and p3 are set. On the other hand, when the parameter p4 is set, the parameters p2 and p3 may not be set.

  The parameter p5 is a distance from the imaging unit 22 to the object o2 (a distance between two points when the two points of the object and the imaging unit are projected on a straight line connecting the object and the viewpoint of the worker). . That is, the parameter p5 is a value stored by the work image receiving unit 11 in step S102. The work image receiving unit 11 stores the received distance information in the parameter storage unit 16 as the value of the parameter p5 of the record corresponding to the received worker ID.

  Parameters p1 to p4 are preset values. The parameters p1 to p4 may be set by measuring a corresponding part of the worker. Alternatively, general values may be set for the parameters p1 to p4 as common values for each worker. The general value may be acquired from statistical information or the like, for example. Further, when the mounting position of the imaging unit 22 is not fixed, statistical information may be set for the parameters p3 and p4. However, the parameters p1 to p4 may also be measured in real time using various sensors or the like, similarly to the parameter p5. On the other hand, when the value of the parameter p5 is stable to some extent, the value of the parameter p5 may be set in advance.

  For example, the error calculation unit 13 calculates the error as follows using the parameters p1 to p5 associated with the worker ID received together with the moving image. FIG. 7 is a diagram for explaining an example of an error calculation method. In FIG. 7, the parameter of the record whose worker ID is 0001 among the records shown in FIG. 5 is applied. The error is indicated by x.

In the present embodiment, the following state is assumed for convenience.
(1) It is assumed that the object o2, the imaging unit 22 (the lens thereof), and the operator's eyes are on the same horizontal plane.
(2) It is assumed that the fingertip pointing to the object o2 is on a straight line connecting the center of the operator's eyebrows and the object.

Then, the following relationship is established based on the similarity ratio between the right triangle ABC and the right triangle DBE shown in FIG.
70: 130 = 6: x
Solving the above equation yields x≈11. Therefore, in this case, 11 cm is calculated as an error.

  It should be noted that the above calculation example regarding the error is not necessarily intended to be effective only when the above (1) and (2) are satisfied. Even if (1) and (2) are not strictly satisfied, the above calculation example may be applied. In this case, the accuracy of the error slightly deteriorates. However, as described later, in this embodiment, the accuracy of the error is allowed to be deteriorated to some extent, and the error accuracy deterioration is corrected (corrected). It is also possible to do.

  Subsequently, the image relay unit 14 draws (adds) a circle centered on a position shifted by an error from the position of the finger in the moving image (S106). Specifically, a circle centered at a position shifted by 11 cm in the direction opposite to the vertex D is drawn starting from the vertex E in FIG. The circle is a circle for estimating the range of the arrangement position of the object pointed by the worker. Hereinafter, the circle is referred to as a “range circle”. Note that the conversion between the distance in the real space and the distance on the moving image may be appropriately performed using a known technique.

  Subsequently, the image relay unit 14 transfers (outputs) the moving image to which the range circle is added, together with the worker ID related to the moving image, to the supporter terminal 30 (S107).

  Subsequently, a processing procedure executed by the supporter terminal 30 will be described. FIG. 8 is a flowchart for explaining an example of a processing procedure executed by the supporter terminal.

  In step S <b> 301, the display control unit 31 receives the moving image transferred from the work support server 10. Subsequently, the display control unit 31 displays the moving image (S302).

  FIG. 9 is a diagram illustrating a display example of an image to which a range circle is added. In FIG. 9, the range circle c1 is displayed so as to surround the object o2. As a result, the supporter can recognize that the object designated by the worker is not the object o3 but the object o2.

  The range circle may be replaced by a symbol other than a circle. For example, the range of the object may be indicated by a polygon, an ellipse, or the like. In this case, the symbol used is preferably a symbol that can indicate a range. When such a symbol has a range, statistical information is used when the above conditions (1) and (2) are not completely satisfied, or when the parameters p1 to p4 are not accurate or the parameters p1 to p4 are used. This is because it is possible to allow the accuracy of the error calculation result to be deteriorated in such a case. For example, in the present embodiment, the center of the operator's eyebrows is treated as the operator's viewpoint (a stance point for viewing the object), but in reality, the operator's viewpoint depends on the operator's dominant eye. , With a slight deviation from the center. Such a fact may have some influence on the accuracy of the error calculation result. Therefore, for example, when the accuracy of the parameters p1 to p4 is low, the size of the range circle (for example, the radius of the range circle) may be increased. Further, it is conceivable that the accuracy of the error calculation result decreases as the distance from the imaging unit 22 to the object increases. Therefore, the image relay unit 14 may change the size of the range circle according to the size of the parameter p5. Specifically, the larger the parameter p5, the larger the range circle.

  The supporter can confirm whether or not the position of the range circle is displayed at the position designated by the worker by communicating with the worker while referring to the moving image. As a result, when it is determined that the range circle is not displayed at the position designated by the operator, the supporter inputs a correction value for the position of the range circle. The correction value is a value including, for example, the direction of deviation and the length of deviation.

  When the correction value receiving unit 32 receives the input of the correction value (Yes in S203), the correction value transmitting unit 33 sends the correction value and the worker ID received in step S201 to the work support server 10. Transmit (S204).

  When the correction value receiving unit 15 of the work support server 10 receives the correction value and the worker ID, the parameter storage unit 16 stores the correction value for the record corresponding to the worker ID. .

  In step S105 in FIG. 4, when the correction value for the worker ID received together with the moving image is stored in the parameter storage unit 16, the error calculation unit 13 sets the calculated error to the correction. Correct by value. For example, when the correction value is “right 30”, the position of the range circle specified based on the error is shifted by 30 cm to the right. As a result, the accuracy of the display position of the range circle can be improved.

  In the above description, the example in which the imaging unit 22 is mounted substantially horizontally on the operator's eyes has been described. However, when the imaging unit 22 is mounted on the operator's head, the vertical direction is used. The calculation method described with reference to FIG.

  As described above, according to the present embodiment, based on the relative viewpoints of the worker's viewpoint in real space, the imaging unit 22, the pointing object, and the target object, the pointing object on the image is displayed. For the position, a deviation (error) with respect to the object is calculated. Further, based on the deviation, a range circle is drawn in the estimated range of the indicated position. Therefore, it is possible to specify the designated position intended by the instructor (operator) on the image obtained by imaging the designated object that points away from the object and points to the object.

  The imaging unit 22 may not be attached to a part of the worker's body. For example, when the location of the worker is fixed, the imaging unit 22 may be arranged at a predetermined position other than the worker's body.

  In the present embodiment, the work support server 10 is an example of an information processing apparatus. The designated position specifying unit 12 is an example of a specifying unit. The error calculation unit 13 is an example of a calculation unit. The image relay unit 14 is an example of an output unit. The correction value receiving unit 15 is an example of a receiving unit.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
A receiving unit that receives an image of an object and an instruction object that points away from the object and points to the object;
For the pointing object, a specifying unit for specifying a position in the image;
Based on information indicating a relative positional relationship in real space between the viewpoint of the instructor who instructs the target using the pointing object, the imaging unit, the pointing object, and the target. A calculation unit that calculates a deviation from the target with respect to the position of the pointing object in
An output unit for outputting an image to which a predetermined symbol is added within a predetermined range from a position having the deviation from the position of the pointing object with respect to the image;
An information processing apparatus comprising:
(Appendix 2)
A reception unit that receives an input of a correction value for the deviation;
The output unit outputs an image to which the predetermined symbol is added within a predetermined range from a position specified by applying the correction value to the shift;
The information processing apparatus according to appendix 1, wherein
(Appendix 3)
The output unit changes a size of the predetermined symbol according to a distance between the imaging unit and the pointing object;
The information processing apparatus according to supplementary note 1 or 2, characterized in that:
(Appendix 4)
Computer
Receiving an image of the target object and the pointing object that points away from the target object and points to the target object;
Identifying the position of the pointing object in the image,
Based on information indicating a relative positional relationship in real space between the viewpoint of the instructor who instructs the target using the pointing object, the imaging unit, the pointing object, and the target. Calculating a deviation from the object with respect to the position of the pointing object in
Outputting an image to which a predetermined symbol is added within a predetermined range from a position having the deviation relative to the position of the pointing object with respect to the image;
An information processing method characterized by executing processing.
(Appendix 5)
The computer executes a process of receiving an input of a correction value for the deviation,
The output processing outputs an image in which the predetermined symbol is added within a predetermined range from a position specified by applying the correction value to the deviation.
The information processing method according to appendix 4, characterized in that:
(Appendix 6)
The output processing changes the size of the predetermined symbol according to the distance between the imaging unit and the pointing object.
The information processing method according to appendix 4 or 5, characterized in that:
(Appendix 7)
On the computer,
Receiving an image of the target object and the pointing object that points away from the target object and points to the target object;
Identifying the position of the pointing object in the image,
Based on information indicating a relative positional relationship in real space between the viewpoint of the instructor who instructs the target using the pointing object, the imaging unit, the pointing object, and the target. Calculating a deviation from the object with respect to the position of the pointing object in
Outputting an image to which a predetermined symbol is added within a predetermined range from a position having the deviation relative to the position of the pointing object with respect to the image;
A program characterized by causing processing to be executed.
(Appendix 8)
Causing the computer to execute a process of receiving an input of a correction value for the deviation;
The output processing outputs an image in which the predetermined symbol is added within a predetermined range from a position specified by applying the correction value to the deviation.
The program according to appendix 7, characterized by:
(Appendix 9)
The output processing changes the size of the predetermined symbol according to the distance between the imaging unit and the pointing object.
The program according to appendix 7 or 8, characterized by the above.

DESCRIPTION OF SYMBOLS 1 Work support system 10 Work support server 11 Work image receiving part 12 Pointed position specific | specification part 13 Error calculation part 14 Image relay part 15 Correction value receiving part 16 Parameter memory | storage part 20 Worker terminal 21 Distance measuring part 22 Imaging part 23 Work image transmission Unit 30 Supporter Terminal 31 Display Control Unit 32 Correction Value Accepting Unit 33 Correction Value Transmitting Unit 100 Drive Device 101 Recording Medium 102 Auxiliary Storage Device 103 Memory Device 104 CPU
105 Interface device B bus

Claims (5)

A receiving unit that receives an image of an object and an instruction object that points away from the object and points to the object;
For the pointing object, a specifying unit for specifying a position in the image;
Based on information indicating a relative positional relationship in real space between the viewpoint of the instructor who instructs the target using the pointing object, the imaging unit, the pointing object, and the target. A calculation unit that calculates a deviation from the target with respect to the position of the pointing object in
An output unit for outputting an image to which a predetermined symbol is added within a predetermined range from a position having the deviation from the position of the pointing object with respect to the image;
An information processing apparatus comprising: A reception unit that receives an input of a correction value for the deviation;
The output unit outputs an image to which the predetermined symbol is added within a predetermined range from a position specified by applying the correction value to the shift;
The information processing apparatus according to claim 1. The output unit changes a size of the predetermined symbol according to a distance between the imaging unit and the pointing object;
The information processing apparatus according to claim 1 or 2. Computer
Receiving an image of the target object and the pointing object that points away from the target object and points to the target object;
Identifying the position of the pointing object in the image,
Based on information indicating a relative positional relationship in real space between the viewpoint of the instructor who instructs the target using the pointing object, the imaging unit, the pointing object, and the target. Calculating a deviation from the object with respect to the position of the pointing object in
Outputting an image to which a predetermined symbol is added within a predetermined range from a position having the deviation relative to the position of the pointing object with respect to the image;
An information processing method characterized by executing processing. On the computer,
Receiving an image of the target object and the pointing object that points away from the target object and points to the target object;
Identifying the position of the pointing object in the image,
Based on information indicating a relative positional relationship in real space between the viewpoint of the instructor who instructs the target using the pointing object, the imaging unit, the pointing object, and the target. Calculating a deviation from the object with respect to the position of the pointing object in
Outputting an image to which a predetermined symbol is added within a predetermined range from a position having the deviation relative to the position of the pointing object with respect to the image;
A program characterized by causing processing to be executed.

Images