JP2005069734A - Teaching method of spatial position of object by pointing action - Google Patents

Abstract

Provided is a method for teaching a spatial position of an object by a pointing action that can be performed naturally and easily.
In a teaching method for determining the position and orientation of an object placed in a work space of a computer on a surface basis, fingers of a human body are extracted, indices are set at at least two positions of the fingers, and the fingers The spatial coordinates of the two points are calculated, the spatial coordinates are calculated by the stereo method, the corresponding points in stereo vision are searched based on the index set on the finger, and a three-dimensional straight line is found in the spatial coordinates of the two points. Fitting is performed, and an intersection point between the straight line and an object plane whose three-dimensional coordinates are known is taught as an object position.
[Selection] Figure 1

Description

  The present invention relates to a method for teaching a spatial position of an object by a pointing action.

  In recent years, research on recognizing various human movements has been conducted with the aim of practical use of man-machine interfaces. For example, although a result is obtained by a system in which a sensor, a camera, or the like is mounted on a human body (Non-Patent Documents 1 and 2 below), the subject needs to get used to the experimental environment or gives a burden. Therefore, research has been conducted to convey his intention to a computer in a free state as much as possible by a gesture (Non-Patent Documents 3 to 5 below).

  However, the gesture must be easy for the user to understand. In connection with this, researches have been conducted to operate various home appliances in the work space with gestures (Non-Patent Documents 3 and 4 below). The former assumes only rest for disabled people who need care, while the latter communicates intentions with moving gestures. In the latter case, the target is selected based on the number of fingers extending when selecting one of a plurality of home appliances, but when the number of target objects increases, the target object is pointed by some method. The number of gestures that the user should learn can be reduced by giving instructions.

  In addition, voice conversation (Non-Patent Document 6 below) is also possible, but for example, when a specific book is selected from a plurality of books, it is difficult to convey the intention only by voice. There must be.

  The interaction between humans and robots in daily life is not complicated, and natural ones are desirable.

For example, the utterance of “take that book on the desk” while pointing at the object is more natural than “take a book in how many centimeters from the left end of the desk” (Non-patent Document 7, below) 8). As described above, the most natural method for indicating an object is the pointing.
Mitsuyoshi Yoshimura, Yumiko Sakai, Tamiko Kai, Isao Yoshimura: Attempt to extract the “wager” part of Japanese dance and quantify its characteristics, IEICE Transactions D-II Vol. j84-D-II, 12, pp. 2644-2653 (2001) E. Hori, H .; Saito: 3D Input Interface via Hand Motion Motion Stereo Camera System for Wearable PC Environment, MVA2002 IAPR Workshop on Machine Vision. 285-289 (2002) Takuya Kawano, Kazuhiko Yamamoto, Kunito Kato, Hitoshi Hongo, Yoshinori Niwa: Improvement of home appliance control support system for care support using pose recognition, IEICE technical report, PRMU2001-114 HIP2001-17 MVE2001-76, pp. 9-14 (2001) Kota Irie, Kazunobu Umeda: Construction of intelligent room based on gesture recognition, Proc. 62-67 (2003) Teruhisa Murashima, Yoshinori Kuno, Nobutaka Shimada, Yoshiaki Shirai: Understanding and learning gestures through human-machine interaction, Journal of the Robotics Society of Japan, 18, 4, pp. 590-599 (2000) Toshihiro Murai, Hideki Aso, John Fly, Taichi Asano, Yoichi Motomura, Isao Hara, Takio Kurita, Satoru Hayami, Nobuyuki Yamazaki: Spoken Dialogue System for Office Mobile Robot Jiro-2, Journal of the Robotics Society of Japan, 18, 2, pp. 300-307 (2000) Osamu Morikawa, Yukio Fukui, Juri Yamashita, Shigeru Sato: Pointing action in human-friendly hypermirror dialogue, Transactions of Information Processing Society of Japan, 41, 5, pp. 1290-1297 (2000) Yuri Yoshida, Hideo Saito: Automatic shooting system for hand-directed objects, 2001 IEICE General Information Proceedings Information System 277 (2001)

  In the present invention, as described above, focusing on the fact that the pointing action is more effective as the pointing means than the voice dialogue, the teaching of the spatial position of the object by the pointing action that can be performed naturally and easily. It aims to provide a method.

In order to achieve the above object, the present invention provides
[1] In a teaching method for determining the position / orientation of an object placed in the work space of a computer on a surface basis, fingers of a human body are extracted, indices are set at at least two positions of the fingers, Two spatial coordinates are calculated, the spatial coordinates are calculated by the stereo method, the corresponding points in stereo vision are searched based on the index set on the finger, and a three-dimensional straight line is applied to the spatial coordinates of the two points. The intersection of the straight line and the object plane whose three-dimensional coordinates are known is taught as the position of the object.

  [2] In the method for teaching the spatial position of an object by the pointing action described in [1] above, the finger of the human body is extracted by moving only the hand in a state where there is no movement in the background, and the difference between frames of the image is calculated. The moving region is defined as a hand presence candidate region, the skin color portion of the hand presence candidate region is extracted as a hand region, the image is binarized, and then the finger extends from the palm of the hand It is characterized by extracting the palm area where only the finger is cut, extracting the palm from the hand area, and taking the difference.

  [3] In the method for teaching the spatial position of an object by a pointing action as described in [1] above, the index is a marker.

  [4] In the method for teaching the spatial position of an object by a pointing action described in [1] above, the index is a characteristic point unique to the finger.

  [5] In the method for teaching the spatial position of an object by a pointing action as described in [4] above, the characteristic points unique to the finger are the tip and joint of the finger.

  [6] In the method for teaching the spatial position of an object by a pointing action as described in [1] above, the robot is taught the position of the object.

  According to the present invention, the following effects can be achieved.

  The pointing action can realize a natural and easy-to-understand dialogue, and the position and orientation of an object placed in the computer work space can be determined on a surface basis.

  In addition, various applications are possible, such as applying a finger to the teaching of various object surfaces.

  In the teaching method that determines the position and orientation of an object placed in the work space of a computer on a surface basis, the target object is displayed by pointing the target with a finger so that a natural and easy-to-understand dialogue can be realized. To the computer.

  Hereinafter, embodiments of the present invention will be described in detail.

  Here, a method for teaching the spatial position of an object by a pointing action on a spatial straight line will be described.

(1) Pointer vector A process for pointing a remote object to the computer with a finger and bringing it to the computer with a robot arm will be described. In order to recognize an object located away from the finger, it is necessary to know the spatial positional relationship between them. Therefore, the images used in the experiment were taken with a stereo digital video camera. If the finger is definitely pointing in the direction of the intended object, a straight line extending from the finger and a certain plane constituting the object should intersect. Based on this idea, the position of the object is estimated.

  First, a spatial finger position is searched to apply a three-dimensional straight line to the finger. For estimation of three-dimensional coordinates, the depth is calculated based on the principle of triangulation. In order to facilitate searching for corresponding points in the left and right images of the finger, two markers are provided on the finger (minimum number for determining the direction of a straight line). Note that a feature point of a finger (a tip of a finger, a joint, a wrinkle, etc.) may be used instead of the marker.

  In order to find a corresponding point between the marker template and the marker in the image, matching processing using the following correlation coefficient is performed.

物体の対応点については、現時点では手動で求めている。なお、対象とする物体には立方体を用いている。

At this time, the corresponding points of the object are obtained manually. Note that a cube is used as a target object.

A straight line is fitted to the three-dimensional coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the two positions of the finger obtained by the stereo measurement process, and the intersection of the straight line and the object plane is obtained. . If the target plane is represented by the normal vector ρ ₀ and the distance between the space coordinate origin and the plane is represented by ρ, the plane equation is represented as follows.

また、指の直線ベクトルは、

The finger linear vector is

と表される（図１参照）。これを指さしベクトルと呼ぶ。上記式（２），（３）より、

(See FIG. 1). This is called a pointing vector. From the above formulas (2) and (3),

これらの処理による結果を以下に示す。

The results of these processes are shown below.

  FIG. 3 shows a situation where markers 1 and 2 are extracted by searching. For these points, three-dimensional coordinates are obtained by stereo measurement, and a straight line equation passing through the two points is obtained. As shown in FIG. 4, since the line segment stops at the position where it intersects with the object 3, the direction and position of the object 3 at a distant place are estimated. From this, it is considered that the direction of the object 3 to be reconstructed is estimated.

(2) Accuracy of spatial coordinates An experiment was conducted to confirm whether the three-dimensional coordinates that determine the direction of the finger were correctly reproduced. In the experiment, a rod-like object with a marker that looks like a finger is fixed to a tripod, rotated by 5 ° in the vertical direction and by 10 ° in the horizontal direction, and the measured angle value and the calculated angle value match. I checked to see if. Use images taken with a stereo camera. Corresponding points of binocular stereopsis are searched using a marker, and the angle of the finger is calculated from the reproduced three-dimensional coordinates (X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ ). Ask. The vertical angle is

水平方向の角度は、

The horizontal angle is

によって計算する。実測値と計算値を比較したグラフを図５，６と表１，２に示す。これらの図において、ａは実測値、ｂは計算値（理想値）を示している。

Calculate by Graphs comparing measured values and calculated values are shown in FIGS. In these drawings, a represents an actual measurement value, and b represents a calculated value (ideal value).

この実験から、垂直方向の角度及び水平方向の角度ともに実測値と計算値は極めて近似していることがわかる。

From this experiment, it can be seen that the measured value and the calculated value are very close to each other in the vertical angle and the horizontal angle.

(3) Automatic Detection of Finger Corresponding Points FIG. 7 is an explanatory diagram of a finger corresponding point automatic detection method according to an embodiment of the present invention. 7A is an original image (before hand shaking), FIG. 7B is an original image (after hand shaking), and FIG. 7C is an inter-frame difference (difference from an image several frames before). In this example, the difference from the image one frame before is output as an output, Fig. 7 (d) is a hand presence candidate region, Fig. 7 (e) is a hand region extraction, and Fig. 7 (f) is a hand. Region binarization (in a digital image (monochrome image), white is given if the density value is above a certain threshold value, and black values are given otherwise), FIG. 7 (g) is palm region extraction, and FIG. 7 (h). Indicates the extraction of the finger area, respectively.

  First, in order to search in the image where the hand region is in the space, the skin color portion is extracted. At this time, if other skin parts such as the face are reflected, or if there is a similar color such as cardboard in the background, it will be extracted together. Therefore, a part considered to have a hand in advance is extracted. Only the hand is moved on the assumption that there is no movement in the background [FIG. 7 (b)], the difference between frames [FIG. 7 (c)], and the part where the moving area is large is set as the hand existence candidate area [FIG. d)]. Further, the skin color portion of the hand presence candidate region is extracted as the hand region [FIG. 7 (e)].

  Next, only the fingers are extracted from the hand area.

  First, after binarizing the image [FIG. 7 (f)], it is considered that the finger is a thin stick-like shape extending from the palm, and when the contraction / expansion processing is applied, the palm is cut and only the finger is cut out. [FIG. 7 (g)]. When the difference from the palm of FIG. 7G showing the hand region is extracted from FIG. 7G, only the finger region remains [FIG. 7H], and only the finger region is extracted from the hand region. be able to.

  A preliminary experimental diagram is shown below.

  Only one finger is required for the actual pointing action. Corresponding points are automatically obtained by such a method.

(4) Experiment The validity of the pointing vector obtained in the section (1) above is shown by obtaining the intersection with the object plane. First, three points on the target plane are indicated on the image, and these three-dimensional coordinates are obtained by the stereo method. Further, a unit normal vector ρ ₀ is calculated by the following equation (8).

また、平面方程式を下記式（９）のように表す。

Further, the plane equation is expressed as the following formula (9).

  By substituting the above equation (9) for such a plane into the equation (3), t can be calculated for the intersection.

  As described above, an object placed in front of a stereo camera is pointed with a finger, an intersection with the object plane is obtained, and an experiment is performed in which the object responds to something. I was able to distinguish and indicate objects.

  According to the present invention, an object pointed by a finger can be recognized, and the computer can respond about what it is. However, even if you intend to correctly point to the object you want, you may actually be misaligned and it is difficult to pinpoint. It is thought that various factors such as the influence of the deviation of the finger and the line of sight position and the finger being warped influence, but in order to avoid this, first of all, not only teaching straight pinpoint, but also the target object Move your finger to specify a range. Also, instead of unidirectionally instructing an object from a human, it is thought that it can be solved by confirming “Is this?” From the machine and approaching the actual conversation while interacting with each other .

  It is important in the operation of the robot to know in what position and in what posture the object is placed. The teaching of the object itself in the work environment can be done by collating it with a model stored in advance in a computer using a CAD model or the like, but in order to determine the posture, which surface of the model is located in the work space I have to tell you if there is. Here, it is conceivable to apply a pointing device as a means for teaching a surface. In this way, various applications to a system that can be pointed as freely as possible without requiring a special device are conceivable.

  In the above embodiment, the stereo method is used for detecting the three-dimensional position of the marker to be attached to the finger. However, it is also possible to use light projection type position detection or laser distance measurement (eg, ERIM sensor).

  The present invention is not limited to the above-described embodiments, and various modifications are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

  According to the teaching method of the spatial position of the object by the pointing action of the present invention, the position / posture of the object placed in the work space of the computer is determined by the surface base, and the pointing is applied to the teaching of the surface. In particular, it is suitable for a teaching method to the robot of various object spatial positions.

It is a figure which shows the intersection of the straight line by the pointing and the plane of an object concerning this invention. It is explanatory drawing of the normal vector concerning this invention. It is a figure which shows the example of a detection of the corresponding point concerning this invention. It is a figure which shows the example of an intersection search of the straight line concerning this invention and an object plane. It is a figure which shows the measured value and calculated value of the angle of the vertical direction concerning this invention. It is a figure which shows the measured value and calculated value of the angle of the horizontal direction concerning this invention. It is explanatory drawing of the automatic detection method of the corresponding point of the finger | toe which shows the Example of this invention.

Explanation of symbols

1, 2 Marker 3 Object

Claims (6)

In a teaching method for determining the position / orientation of an object placed in the work space of a computer on a surface basis,
(A) Extract the finger of the human hand,
(B) set an index on at least two positions of the finger;
(C) calculating two spatial coordinates of the finger;
(D) The spatial coordinates are calculated by the stereo method,
(E) searching for corresponding points in stereo vision based on the index set on the finger;
(F) A space of an object by a pointing action characterized by applying a three-dimensional straight line to the spatial coordinates of the two points, and teaching the intersection of the straight line and an object plane whose three-dimensional coordinates are known as the position of the object. Position teaching method.   2. The method of teaching the spatial position of an object by a pointing action according to claim 1, wherein the finger of the human hand is extracted by moving only the hand with no movement in the background, taking the difference between frames of the image, The area where the hand is large is defined as the hand candidate area, the skin color part of the hand candidate area is extracted as the hand area, the image is binarized, and the finger is regarded as a thin stick extending from the palm. Then, the spatial position of the object by the pointing action is performed by extracting the palm area where only the finger is cut by performing contraction / expansion processing, extracting the palm from the hand area and taking the difference Teaching method.   2. The method for teaching the spatial position of an object by pointing action according to claim 1, wherein the index is a marker.   2. The method for teaching the spatial position of an object by pointing action according to claim 1, wherein the index is a characteristic point unique to the finger.   5. The method for teaching the spatial position of an object by a pointing action according to claim 4, wherein the characteristic points unique to the finger are a tip and a joint of the finger.   2. The method for teaching the spatial position of an object by pointing action according to claim 1, wherein the robot teaches the position of the object to the robot.

Images