JP2001282456A - Man-machine interface system - Google Patents

Abstract

(57) [Summary] [Problem] To provide a man-machine interface system using an image of a fingertip on a desk. A description will be given of a process performed by the system to obtain coordinates of a fingertip. The region of the user's hand placed on the desk 10 is extracted from the image of the infrared camera 50. A fingertip point of the user is detected from the extracted hand region. From the position coordinates of the fingertip point on the camera image, conversion is performed to the position coordinates on the desk using the projection conversion parameters obtained in advance by calibration. Interact with objects and information projected on the desk based on the fingertip position information on the desk.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a man-machine interface, and more particularly, to attaching a measuring instrument or the like to a user's body while placing importance on user's comfort without being conscious of existing operations when using a computer. The present invention relates to an interface that does not reduce user flexibility.

[0002]

[Technical Background] Computers have penetrated deeply into our lives, and their use is diverse. They carry very small computers that they could not have imagined before, or are used in places that are embedded in the surrounding environment and seemingly invisible. In the future, it is expected that the system will be developed in two forms: a form such as a wearable personal computer and a PDA that is attached to a person, and a form that is integrated with the environment around the user such as a desk, wall, and room. Considering the man-machine interface in such a flow, the interface represented by the current GUI etc. binds the user to the interface,
I had to pay attention to the interface apart from what I wanted to do. Therefore, in the next-generation interface, an interface capable of performing a more direct and more natural operation for a task that the user wants to perform is required.

[0003] Therefore, in order to reduce the burden of the user interface in searching and using information using the computer, the computer supports the work and operation of the user in the real world, and the conventional interface using only a keyboard and a mouse. The next-generation interface that can be used more comfortably is expected to be a real-world oriented, situation-aware interface. In the real world oriented user interface, the computer constantly recognizes the situation of the user in the real world, and the computer assists the work by sampling the user's intention. In this way, we aim to realize a transparent interface that does not feel the operation of existing computers. By applying this real-world-oriented interface to work on a desk, a real-world-oriented interface (Enhanced Desk) integrated into a “desk” is being developed as an example of a desk-type real-world-oriented interface. Digital Desk (for example, see Koike "Bit Separate Volume Visual Interface-Towards Post-GUI", Kyoritsu Shuppan, Chapter 2.1, pp.24-44, etc.) as an example focusing on the fusion of work on a desk and work on a computer Is well known. In Digital Desk, a computer screen projected on a desk is operated with a fingertip or the like. The user can cut and copy the illustration projected on the desk, or perform calculations with the calculator projected on the desk.

[0004] In some cases, attention has been paid to paper documents on a desk, and an attempt has been made to integrate and utilize paper documents and electronic information (M. Kobayashi and
H.Koike: Enhanced Desk, Integrating Paper Documen
ts and Digital Documents; Proceedings of 1998 Asia
Pacific Computer Human Interaction, pp.167-174
(1998)). In this example, a barcode previously assigned to the paper document is used to associate the paper document with the electronic information. However, since the barcode needs to be observed in a certain size or more, a large barcode is required. Had to use code. In addition, skin color extraction has been performed to recognize the user's hand region. However, if an object close to the skin color is erroneously recognized on the desk or an image is projected on the user's hand, the hand region extraction does not work well. There was a problem. Regarding the recognition of the user's fingertip position, there are many restrictions such as always assuming a specific hand orientation and one fingertip, and insufficient recognition accuracy has been a serious problem. Since a series of processing is performed by software, real-time processing is impossible.

[0005]

SUMMARY OF THE INVENTION An object of the present invention is to provide an interface that is integrated into the environment around a user.
This is the realization of an information interface system integrated on a “desk”. In an office or the like, work using a PC is often performed on a desk, and at the same time, work using paper documents is often performed. By making a computer observe these desk work, an information interface system that can be handled in an integrated manner is provided.

[0006]

To achieve the above object, the present invention relates to a man-machine interface system using an image of a fingertip on a desk, comprising: an infrared camera for obtaining an image on a desk; and the infrared camera. And a processing system for inputting the image according to the above, and the processing system extracts the hand region from the image using the body temperature, and extracts the finger region from the extracted hand region with a pattern prepared in advance. A man machine characterized by pattern matching and finding the coordinates of the specified fingertip on the desk.
Interface system. The previously prepared pattern may be a circular pattern. Thereby, the recognition accuracy of the fingertip can be improved, and the recognition process can be performed at high speed.

Further, when the number of fingertips obtained is one, it can be recognized as a pointing operation, and a pointing operation performed by a mouse or the like can be performed by the pointing operation. In addition, a camera capable of controlling the shooting position is provided, and by performing the pointing operation, it is possible to perform control for adjusting the shooting position of the camera to the coordinates of the fingertip on the desk. This makes it possible to capture an image captured by the camera, extract a barcode from the captured image, and perform a barcode recognition process.
In addition, a voice input means is provided, and a voice command can be input by inputting voice from the voice input means and performing voice recognition processing. Furthermore, a projector is provided, and by the projector,
By projecting the computer screen on the desk, interactive processing with the computer screen can also be performed. The present invention also includes a recording medium storing a program for performing the above-described processing.

[0008]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A man-machine interface system according to the present invention has the following features. (1) A user's hand region is stably extracted by measuring radiation light from a user's skin region using an infrared camera. (2) High-speed tracking of a user's fingertip is realized by using template matching based on normalized correlation or the like. (3) In order to recognize a small barcode on the desk, the area around the user's fingertip is enlarged and tracked using a pan-tilt camera with a zoom function. (4) Use image processing hardware and apply distributed processing to speed up processing. Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings, focusing on these features.

FIG. 1 is an external view of an enhanced disk (Enhanced Desk) according to an embodiment of the present invention. As shown in FIG. 1, a projector 3 is placed above a normal desk 10.
0 and mirror the screen of the computer (not shown)
The light is reflected at 4 and the screen 12 is projected on the desk. Two cameras were installed to observe the desk. One is an infrared camera 50, which is installed with the desk 10 facing upward from the left side,
2. Take a photo on the desk. Another pan-tilt camera 20 with a zoom function is installed at the upper right of the desk, and can enlarge a part of the desk to take a picture. Although not shown, a speaker and a microphone are also provided, and have a function of inputting and outputting voices and the like.

FIG. 2 shows a system configuration diagram in the embodiment. In FIG. 2, a projection image generated by the computer system 60 is projected on the desk 10 by the projector 30 via the surface reflecting mirror 44. In addition, the posture and the like of the pan / tilt camera 20 that is capturing an image on the desk 10 are controlled by the computer system 60. Images captured by the pan / tilt camera 20 are also captured and processed in the computer system 60. The image on the desk captured by the infrared camera 50 is also captured by the computer system 60 and processed. The main processing flow in the enhanced disk of the present invention is shown below. The user's hand region is extracted from the image of the infrared camera 50. A fingertip point of the user is detected from the extracted hand region. From the position coordinates of the fingertip point on the camera image, conversion is performed to the position coordinates on the desk using the projection conversion parameters obtained in advance by calibration. Interact with objects and information projected on the desk based on the fingertip position information on the desk. These processes will be described in detail below.

<Extraction of Hand Region> Extraction of a hand region generally uses background subtraction or skin color extraction using a color camera. However, with these methods, it is often difficult to extract a hand region depending on the state of the background. In particular, in the work environment envisioned by this system, paper documents and books may be opened on a desk, and the projection of electronic information by a projector may cause problems such as the color of the user's hand area not being constant. is there. Therefore, in this system,
The infrared camera 50 was used to stably extract the hand region. From the image of the infrared camera observing the desk, the hand region can be cut out based on the human body temperature. Using an infrared camera, near the human body temperature (30 ° C to 34 ° C)
By taking a picture of the hand on the desk, for example, NTS
It can be obtained as an image of 256 gradations of C. This image is binarized with a certain threshold to extract a hand region. FIG.
Next, a process of extracting a hand region based on an image from an infrared camera will be described. FIG. 3A is an image from the infrared camera 50. The resolution at the time of capturing by the computer system 60 is 256 × 220 pixels. Then, an image near the human body temperature (30 ° C. to 34 ° C.) is extracted from this image. This is shown in FIG. 3B, in which the hand region is extracted and binarized. As shown in FIG. 3B, by this method, only the hand region can be stably extracted without being affected by changes in the background or illumination. An infrared camera is, for example, a Nikon Thermal Bichon (LAIR)
D3A) can be used. This camera has 768 x 465 effective pixels and 1/60 field time.
A range from -20 ° C to 50 ° C can be photographed in seconds.

<Recognition of Fingertip Position> The fingertip position is detected at a fixed size because the user's hand is on the desk, the distance between the camera and the user's hand is almost constant, and the apparent size does not change extremely. The fingertip position can be detected using the template. Generally, the fingertip position is recognized based on the fact that the contour of the fingertip shape is close to a circle, and performing pattern matching based on normalized correlation around the hand region using a circular template. Since the user's hand is on the desk and the apparent size does not change extremely, the target of the pattern matching processing is limited to only a certain range of the hand area, and the processing is speeded up. The recognition of the fingertip position will be described with reference to FIG. In the pattern matching process, for example, a hand region is cut out and binarized to 256 ×
For an image of 220 pixels (see FIG. 3B), FIG.
Using a template having a size of 15 × 15 pixels shown in (a), a match is detected using a correlation value. After the pattern matching process, if many points having a high correlation value are found in places that are too close, the point having the highest correlation value is left, and the rest are excluded from fingertip candidate points. As a result, several fingertip candidate points are obtained, and among them, misrecognized points other than the fingertip points are excluded. In this process, as shown in FIG. 4B, pixels corresponding to eight points around the template (four vertices of the rectangle and bisecting points of each side) are examined to determine whether or not the fingertip point. By this process, the fingertip penetrates the template (ERROR in FIG. 4B), and the case where there is no pixel which seems to be a finger around the template is also excluded from the fingertip point candidates. Finally, as shown in FIG. 4C, up to five fingertip candidate points are adopted in descending order of the correlation value, and are used as the user's fingertip points. As an embodiment of the fingertip detection processing in this system, the fingertip detection processing is performed using the Hitachi image processing board IP5010. IP50
Reference numeral 10 denotes an image processing board having a memory for 40 screens of black and white grayscale images, which is a board capable of performing image processing such as binarization and difference between images stored in the memory at high speed by hardware. . OS is WindowsN
T (registered trademark) and Linux, and many image processing functions are prepared as a library. In this embodiment, in consideration of the affinity with the network, the OS
Inux was used.

<Correction of camera image coordinates and desk plane coordinates>
In order to know which position on the desk corresponds to the position detected in the infrared image, the coordinates of the image of the infrared camera and the coordinates of the plane on the desk are corrected. Here, several corresponding points between the infrared camera image coordinates and the desktop plane coordinates are defined, and based on the relationship, the infrared camera image coordinates (x, y) and the desktop plane coordinates (x ', y')
Is expressed using projective transformation as shown in equation (1).

(Equation 1) In order to obtain a corresponding point, a description will be given with reference to FIG. A desk-top coordinate system is set for the desk 10. For this purpose, the calibration plate 70 is created at the position of the target point on the desk, and is placed on the desk to set the desk coordinate system. With an infrared camera that cuts out the hand area, the temperature difference is output as an image,
As shown in FIG. 5, a small light bulb 72 is embedded in the corresponding point of the calibration plate, and the lighted light bulb is photographed by an infrared camera to measure the corresponding point. After measuring the set of corresponding points,
Determining projective transformation parameters (c ₁ ~c ₈₎ on the basis of that relationship. By performing conversion using these parameters, it is possible to correct image distortion, positional deviation, and the like. In order to obtain the projective transformation parameters, at least four sets of corresponding points are required. However, in this system, in order to stably find a solution, as shown in FIG.
I prepared a pair. With respect to these nine sets, the above simultaneous equations were solved using the singular value decomposition method, thereby obtaining conversion parameters. In this system, the coordinate system on the desk is the same as the coordinate system of the image projected on the desk by the projector for the purpose of reducing the processing, and the value of the plane coordinate on the desk is equal to the position coordinate of the pixel of the image projected on the projector. As a result, in order to project the image of the projector at the fingertip position on the desk plane, it is sufficient to simply project the image in which the object is drawn on the coordinate values on the desk plane coordinates, and the processing can be reduced.

<Barcode recognition by gazing around the fingertip>
In the present system, when the user's fingertip recognized by performing the image processing is one, it can be determined that the user is performing the pointing operation. Thus, for example, by pointing only with the index finger, it is recognized as a pointing operation, and the coordinates of the fingertip are obtained, so that it can be used instead of pointing such as a mouse. The area around the fingertip performing the pointing operation can be regarded as a region of interest. The process of tracking the attention area of one fingertip by the pan / tilt camera 20 as this application will be described below.

(Flow of Fingertip Tracking Process) The flow of the fingertip tracking process in the pan / tilt camera 20 will be described below. Corresponding points are measured between two planes, a desktop plane and a pan-tilt drive plane, and a projection transformation parameter is calculated in advance. The position of the fingertip point on the infrared camera image is measured by image processing. The position of the fingertip point on the infrared camera image is transformed into coordinates on the desk plane by projective transformation. The position of the fingertip point on the desk is converted to the pan-tilt drive plane using the projective conversion. The coordinates on the pan / tilt drive plane are sent to the camera in VISCA code, and the camera is directed to the user's fingertip. Process the image of the camera facing the fingertip point.

(Calculation of Projection Transformation Parameters) In this system, it is only necessary to know the coordinate position of the fingertip position on the desk on the pan / tilt drive plane of the camera 20. This process is performed using the pan / tilt camera 20 because it is possible to know which coordinate value is currently being taken on the driving plane. It has been described above that the camera image coordinates and the desk coordinates have been corrected, but the fingertip tracking also uses the camera 20 to perform the coordinate correction using the same method. For this purpose, first, since the desktop coordinates of the fingertip are known in the processing so far, the position on the driving plane of the pan / tilt camera 20 corresponding to the desktop position is calculated. Measure four or more pairs of corresponding points. This measurement adjusts the position so that the pan-tilt camera captures the corresponding point on the desk at the center,
The coordinate value on the pan / tilt drive plane at that time is measured as a corresponding point. Based on the corresponding points, the parameters for converting the coordinates on the desk to the coordinates on the pan-tilt drive plane are obtained by using the above-mentioned projective transformation. Camera 20 for fingertip tracking
Has a function of pointing a camera in that direction by designating coordinates. Using this function, the user's fingertip is tracked. As an example, the tracking camera 20 for fingertip tracking is, for example, in the vertical direction -7794 to 779.
4. Using a Sony EVI-G20 pan / tilt camera having a pan / tilt drive plane with a size of -15570 to 15570 in the horizontal direction, and transmitting VISCA commands to the camera from the PC serial port to control the pan / tilt of the camera .

(Fingertip Tracking Process) Now, the fingertip tracking process will be described. The position of the fingertip point on the desk 10 is measured by the above-described image processing of the image from the infrared camera 50. The position of the fingertip point on the infrared camera image is converted into coordinates on the desktop plane by projective transformation, and it is recognized that there is only one fingertip. The position of the obtained fingertip point on the desk is converted to the pan-tilt drive plane of the camera 20 using the projective conversion according to the above-mentioned parameters. The coordinates on the pan / tilt drive plane are sent to the camera using the VISCA code, and the camera 20 is directed to the fingertip point of the user. Thus, the image of the camera 20 facing the fingertip can be captured.

(Barcode Recognition) An example in which a fingertip is enlarged and tracked by using the image of the camera 20 to recognize a small object such as a barcode will be described below with reference to FIG. . FIG. 6 shows an enlarged tracking of the fingertip,
This shows how barcode recognition is performed. Recognition can be performed by attaching a barcode to a real object (paper document, book, or the like), and a link to electronic information can be created. A two-dimensional matrix code was used for the barcode. This two-dimensional barcode can obtain not only the type of code but also the position and orientation of the code from the code.If the position where the code of the object is attached is stored, the code Posture can be calculated. FIG. 6A shows an example of a two-dimensional barcode. Such a barcode is recognized if it exists in an image captured by the pan / tilt camera 20. For recognition, a certain size is required in the camera image, but since the pan-tilt camera 20 is taking an enlarged image, even a barcode pasted on a document placed on a desk is about 1.5 × 1.5 It is possible to recognize even the size of cm. If a bar code is pointed by a user's pointing operation on a desk, the bar code can be recognized and an interaction corresponding to the bar code can be caused. FIG.
(B) shows a state in which the fingertip is expanded and tracked with respect to this barcode. FIG. 6B shows an image obtained by binarizing the image of the camera 20. First, infrared camera 5
The position of the fingertip point on the image from 0 is measured by image processing. In this way, it is possible to obtain that the finger is pointing with one finger and the position coordinates of the fingertip point. Next, the position of the fingertip point on the infrared camera image is transformed into coordinates on the desk plane by projective transformation. The position of the obtained fingertip point on the desk is converted to a pan-tilt drive plane using projective transformation,
The obtained coordinates on the pan / tilt driving plane are transferred to the camera 20 by VI.
SCA code is sent and the camera 20 is directed to the user's fingertip. Then, the image of the camera 20 facing the fingertip point (FIG. 6B) is processed, the barcode image is recognized, and the barcode reading process is performed. As an example of the barcode recognition processing, for example, a video image is captured in a size of 320 × 240 pixels from a video input terminal of O2 which is a system of SGI, and a process such as binarization is performed. Can be used for recognition. The image input uses an SGI video library. The barcode recognition processing is performed by software, and can be executed at a speed of about 10 to 15 f / s.

<Speech Recognition> The present system uses speech recognition as an auxiliary role for interaction with a user. Here, it is used only for recognizing a keyword for performing an operation mode change such as a user's pointing operation or moving or rotating an object. Speech recognition is performed using IBM ViaVoice's speech recognition engine. In this system, in order to cause some interaction when the user utters a word registered in advance, the speech recognition process always sends the type of the recognized word as a message to the message server. The message sent to the server is received by the information presentation process, and the interaction corresponding to the word can be caused on the desk.

<Distributed Processing> This system roughly performs four processes: fingertip detection, fingertip tracking by a pan-tilt camera, two-dimensional barcode recognition, voice recognition, and information presentation. In this system, each process is distributed by a plurality of machines for speeding up. FIG. 7 shows a schematic diagram of the distribution processing. The fingertip detection and the fingertip tracking processing by the pan / tilt camera 20 are performed, for example, by Pentium (registered trademark) II 450
Linux and image processing board IP5010 on Mhz PC 64
It is performed using a machine that has introduced. This process measures the number and position of the user's fingertips on the desk,
The desktop coordinate position of the fingertip is sent to the message server 68. Further, the camera 20 is controlled so that the line of sight of the pan / tilt camera 20 is located at the fingertip position on the user's desk. In the two-dimensional barcode recognition process, the image of the pan / tilt camera 20 is input to the video input of the SGI02 system 66, and the two-dimensional barcode is recognized by software processing. The recognition result is always sent to the message server 68. Voice recognition, for example,
WindowsNT and ViaVoic on PC 62 of PentiumII 330Mhz
e is introduced and processing is performed. When the user recognizes all voices spoken into the microphone and recognizes a word registered in advance, a specific message is sent to the message server 68. Finally,
There is an information display process for constantly retrieving information stored in the message server 68 and creating corresponding interactions and images. This is an application process, which is performed by the SGI O2 system 66. A message server 68 for performing communication between the processes includes a tuple space communication system TS Sy
stem / s, commonly known as Linda. The system 68 realizes communication by sharing a space called a tuple space on a network and exchanging a message called a tuple in which an arbitrary character string is set through the space. Because of asynchronous communication, each process can operate completely independently, and there is an advantage that a new process can be easily added or changed. This system was able to detect the fingertip stably at a practical speed of 20 frames per second or more.

<Example of Application> (Simplified Character Recognition) Characters written by the user with his / her finger can be recognized by utilizing the tracking of the user's pointing at the desk. For example, when a number is written with a finger on the square projected on the desk from the projector 30, the number is recognized, and the written number is placed on the desk by the projector 30.
Can be displayed. It can be used to give the user's fingertip a meaning other than simple pointing. (Other Applications) As other applications, not only positional information of the user's fingertip but also gesture recognition using the trajectory of five fingertips is possible. Since it becomes possible to recognize small characters on a paper because the camera can perform enlarged photographing, it is possible to add a function for integrated use of a paper document and electronic information using character recognition. Also,
This system is applied to remote coordination work.
It is also possible to build a communication environment via a desk surface by sharing one desk. The present invention may be applied not only to the above-mentioned distributed computer system but also to a client-server system or a stand-alone system including a plurality of systems.
By reading the program from the storage medium storing the program according to the present invention by the system and executing the program,
The configuration of the present invention can be realized. This recording medium includes a floppy (registered trademark) disk, a CD-RO
M, magnetic tape, ROM cassette and the like.

[0022]

As described above, when the present invention is used, an interface necessary for realizing a desk "Enhanced Desk" capable of integrally handling electronic information on a desk and a real object can be realized. Further, since the fingertip is enlarged and tracked by the camera, for example, it is possible to recognize a small barcode, and it is not necessary to unnaturally attach a large barcode to an object on a desk. By using the interface of the present invention, a user can move a 3D object projected on a desk while rotating it with a fingertip, or display a related homepage by pointing to a barcode attached to a book. Can be more intuitive and more closely linked to reality than with a mouse or keyboard.

[Brief description of the drawings]

FIG. 1 is an external view of an enhanced disk.

FIG. 2 is a system configuration diagram of the present invention.

FIG. 3 is a diagram showing extraction of a hand region.

FIG. 4 is a diagram illustrating fingertip recognition.

FIG. 5 is a diagram illustrating measurement of corresponding points.

FIG. 6 is a diagram for explaining the enlarged tracking of a fingertip.

FIG. 7 is a diagram illustrating a distributed process.

[Explanation of symbols]

 Reference Signs List 10 desks 12 screens 20 Pan-tilt camera with zoom function 30 Projector 42 Surface reflector 44 Surface reflector 50 Infrared camera 60 Computer system 62 Personal computer 64 Personal computer 66 SGIO2 system 68 Message server 70 Calibration plate 72 Small light bulb

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (reference) G06T 1/00 340 G06T 1/00 340Z 5L096 7/00 300 7/00 300D F-term (reference) 5B057 AA20 BA08 BA19 CA08 CA12 CA16 CB06 CB12 CB16 CC01 CE09 CE12 DA08 DB02 DB08 DC09 DC34 5B068 AA05 AA36 BB18 BC05 BD09 CC11 CC19 DD11 5B072 CC24 DD02 5B087 AA10 AB02 AD02 AE06 BC06 BC32 CC09 CC33 5E501 AA03 CB05 CC13 CB14 EA21 EB06 5L096 AA07 BA08 BA18 CA18 DA02 EA35 EA43 GA17 HA08 JA09 JA28 LA01

Claims (8)

[Claims] 1. A man-machine interface system using an image of a fingertip on a desk, comprising: an infrared camera for obtaining an image on a desk; and a processing system for inputting and processing an image from the infrared camera. The processing system extracts a hand region from the image using body temperature, specifies a fingertip from the extracted hand region by pattern matching with a prepared pattern, and obtains a coordinate of the specified fingertip on a desk. A man-machine interface system. 2. The man-machine interface system according to claim 1, wherein the pattern prepared in advance is a circular pattern. 3. The man machine according to claim 1 or 2,
In the interface system, the processing system further includes a process of recognizing a finger pointing operation when the number of obtained fingertips is one, further. 4. The man-machine interface system according to claim 3, further comprising a camera capable of controlling a photographing position, and wherein said processing system recognizes the pointing operation when said finger pointing operation is performed. And controlling the camera to adjust the photographing position to the coordinates on the desk. 5. The man-machine interface system according to claim 4, wherein the processing system captures an image captured by the camera, extracts a barcode from the captured image, and performs a barcode recognition process. A man-machine interface system characterized by performing: 6. The man-machine interface system according to claim 1, further comprising voice input means, wherein said processing system inputs voice from said voice input means, A man-machine interface system that performs recognition processing. 7. The man-machine interface according to claim 1, further comprising a projector, wherein the projector projects a computer screen onto a desk. ·system. 8. A recording medium storing a program for causing a computer system to execute processing performed by the man-machine interface system according to claim 1.