JP2010108500A - User interface device for wearable computing environmental base, and method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a user interface device for a wearable computing environmental base, and a method therefor. <P>SOLUTION: The user interface device for the wearable computing environmental base includes a signal measuring means worn on a finger or the vicinity of a wrist of a user, and including a plurality of image measuring parts provided with an image sensor for receiving a light signal generated from a position indication device generating the light signal, and for measuring respectively images of a user front face, and a signal processing means for calculating three-dimensional coordinates by analyzing the respective images measured by the signal measuring means, for recognizing a movement pattern of a hand of the user on the three-dimensional coordinates, based on the light signal received by the signal measuring means, and for outputting a corresponding command. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a wearable computing environment-based user interface apparatus and method, and more particularly, to a wearable system or a method for controlling movements of both hands of a user in a three-dimensional space in front of a user while being suitable for the wearable computing environment. The present invention relates to a wearable computing environment-based user interface device and method that can be used by inputting peripheral devices.

  Many attempts have been studied to detect user movements, especially hand movements, and use them as interactions between humans and computers in a limited space with existing systems. It was. Existing systems have the disadvantage that the user can wear gloves-like devices or perform input only in limited places that are properly equipped and defined.

  In addition, a device such as a three-dimensional space mouse or a pen currently on the market measures a user's hand movement using a gyro sensor and uses it as a user input. This device must also be held and used by the user, and there is an inconvenience that the user has to carry around when necessary.

  Multi-touch, such as Apple (registered trademark) Ipod Touch (registered trademark), Microsoft (registered trademark) Surface (registered trademark), and Jeff Han's Multi-Touch device, applies touch to the display of the device. Although the advantages of touch are maximized, there is an inconvenience that the device must be held by a hand or used in a restricted device.

  In particular, in the case of a user interface for a wearable system in which the device or computer is attached to or worn on the body, such as the mobility with which the device must be carried and the wearability that the user can easily carry around The design must take into account factors.

Japanese Patent Laid-Open No. 10-031551

  In order to solve the above-described problems, the present invention is suitable for a wearable computing environment, and is used as an input of a wearable system or a peripheral device by movement like a user's hand in a three-dimensional space in front of the user. It is an object of the present invention to provide a wearable computing environment-based user interface device and method thereof.

  To achieve the above object, a wearable computing environment-based user interface device according to the present invention is generated from a position indicator device that is worn near a wrist of a user and generates an optical signal, the position indicator device. A signal measuring unit including a plurality of video measuring units having an image sensor for receiving an optical signal and measuring a video in front of a user, and analyzing each video measured by the signal measuring unit to obtain a three-dimensional Signal processing means for calculating coordinates, recognizing a movement pattern of the user's hand on the three-dimensional coordinates from the optical signal received by the signal measuring means, and outputting a corresponding command is provided.

  The signal measuring means includes a plurality of video measurement units including an image sensor and a body shake measurement unit that measures the degree of body shake based on a user's movement.

  The body shake measuring unit includes an inertial sensor.

  The image measuring unit may include a filter for distinguishing the image from the position signal generated from the position pointing device.

  The signal processing means implements a virtual display screen in a region where the angle of view of the image sensors provided in the plurality of video measurement units overlap each other, and calculates three-dimensional coordinates from the video of the virtual display screen. Features.

  The signal processing means includes a two-dimensional coordinate calculation unit that extracts a two-dimensional coordinate from each video measured by the plurality of video measurement units, and the two-dimensional coordinate extracted by the two-dimensional coordinate calculation unit and the position signal 3D coordinates are calculated based on the position information of the hand sensed by.

  The signal processing unit may further include a body shake correction unit that corrects the image shake measured from the image measurement unit based on the degree of body shake measured by the body shake measurement unit.

  The position pointing device includes position signal generating means for generating a position signal for every movement of the user's hand, and input signal measuring means for receiving an input of a control command from the user, and the user input to the input signal measuring means The operation of the position signal generating means is controlled based on a control command.

  The position signal generating means generates the position signal in the form of an optical signal.

  Meanwhile, in order to achieve the above-described object, a user interface method based on a wearable computing environment according to the present invention uses a position signal generated from a position indicator device that is worn near a user's wrist and generates a position signal. Receiving and measuring the image in front of the user, calculating the three-dimensional coordinates from the image measured in the measuring step, grasping the position of the user's hand from the position signal received in the measuring step, Recognizing a motion pattern of a user's hand on the calculated three-dimensional coordinates, and outputting a command corresponding to the motion pattern recognized in the recognizing step.

  The step of measuring includes the step of measuring the degree of body shake from the movement of the user.

  The method further comprises a step of correcting the shaking of the image measured in the measuring step based on the measured degree of shaking of the body.

  The calculating step includes implementing a virtual display screen in a region where the angles of view of a plurality of image sensors that measure video in the measuring step overlap each other, and calculating three-dimensional coordinates from the video on the virtual display screen. It is characterized by.

  The calculating step includes a step of extracting a two-dimensional coordinate from each image measured by the plurality of image measuring means, and is detected by the extracted two-dimensional coordinate and the position signal of the measuring step. Three-dimensional coordinates are calculated based on hand position information.

  The position signal is generated by the position pointing device every time the hand of the user moves.

  The position signal may be generated based on the input signal when a signal indicating the start and end of hand movement is input from the user to the position pointing device.

  The position signal is generated in the form of an optical signal.

  According to the present invention, when a gesture is taken with both hands in the three-dimensional space in front of the user, the movement is tracked and recognized and processed as a predetermined three-dimensional pattern, so that the computer moves while the user moves. Supporting multi-point input function in user space in a wearable computing environment that must be used, and selecting and manipulating objects on the user display is as user friendly as manipulating things in space There is an advantage that the input interface can be supported.

FIG. 5 is a diagram referred to for explaining the operation of the wearable computing environment-based user interface device according to the present invention; FIG. 5 is a diagram referred to for explaining the operation of the wearable computing environment-based user interface device according to the present invention; It is a block diagram which shows the structure with respect to the motion processor of the user interface apparatus which concerns on this invention. It is a block diagram which shows the structure with respect to the position indicator machine of the user interface apparatus which concerns on this invention. It is an illustration referred in order to demonstrate the image | video measuring method which concerns on this invention. It is an example figure referred in order to explain the position coordinate measuring method concerning the present invention. 5 is a flowchart illustrating a flow of operations for a user interface method according to the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
1 and 2 are diagrams for explaining a wearable computing environment-based user interface device according to the present invention. 1 and 2, the wearable computing environment-based user interface apparatus according to the present invention is mounted near the wrist and detects a movement of the hand. A motion processor 10 for recognizing hand movements from the received signals and processing the corresponding movements.

  First, as shown in FIG. 1, the user controls the display screen 1 on the virtual display screen 2 instead of the actual display screen 1 such as a wall-mounted display device for wearable computing or an HMD (Head Mounted Display). Move your hands to do.

  Here, the movement of the hand corresponds to everything that can be expressed by the user, such as a character, a symbol, and a gesture, and includes not only one hand but also a complex gesture with both hands.

  Accordingly, by inputting with both hands on the virtual display screen 2, the user can control the thing actually output in front of the user's eyes in a three-dimensional space to be similar to multi-touch. . In particular, as shown in FIG. 2, input can be performed in a fixed three-dimensional space in front of the user via a plurality of video measurement units 11a and 11b attached to both ends of the glasses, It is determined by the angle of view from the plurality of video measurement units 11a and 11b.

  At this time, a plurality of position indicator devices 20 may be provided. The position indicator device 20 is embodied in a bracelet shape and can be worn on one or both wrists of the user. In addition, the position pointing device 20 may be embodied in a ring shape and worn on the user's finger. Thus, in the embodiment of the present invention, the case where the position pointing device 20 is embodied in a bracelet shape and worn on both wrists of the user is illustrated, but the present invention is not limited thereto. Refer to FIG. 4 for a specific description of the position pointing device 20.

  On the other hand, the motion processor 10 is embodied in a form that can be worn on the user's body in the same manner as the position indicator 20, and as an example, the motion processor 10 is worn on any part of the body such as glasses, a hat, and clothes. You can also. The embodiment of the present invention exemplifies that the motion processor 10 is implemented in the form of glasses in order to produce the same effect as that seen by the user.

  The motion processor 10 includes a plurality of video measurement units 11a and 11b. At this time, the plurality of video measurement units 11a and 11b are arranged at different positions, and measure signals generated from the position indicator 20 at the corresponding positions. Refer to FIG. 3 for a detailed description of the motion processor 10.

  FIG. 3 is a block diagram showing the configuration of the motion processor 10 according to the present invention. As shown in FIG. 3, the motion processor 10 according to the present invention includes a signal measuring unit 11, a command input unit 13, a signal processing unit 15, and a communication unit 17.

  The signal measuring means 11 includes a first video measurement unit 11a, a second video measurement unit 11b, and a body shake measurement unit 11c.

  The first video measurement unit 11a and the second video measurement unit 11b are arranged at different positions, and can be arranged at both ends of the glasses as shown in FIG. Of course, FIG. 3 illustrates that the first video measurement unit 11a and the second video measurement unit 11b are provided. However, a third video measurement unit and the like can be provided.

  At this time, the first video measurement unit 11a and the second video measurement unit 11b are exemplified as image sensors that can measure signals generated from the position indicator device 20. Of course, it may be in the form of infrared rays, visible rays, lasers, etc. generated by the position pointing device 20.

  In addition, the first video measurement unit 11a and the second video measurement unit 11b receive a signal generated from the position pointing device 20, and sense the position of the user's hand from the received signal. At this time, the first video measurement unit 11a and the second video measurement unit 11b are provided with a physical filter for distinguishing the video signal measured by the image sensor and the signal received from the position pointing device 20.

  At this time, an infrared passband filter or the like can be used as the physical filter. The infrared passband filter removes interference caused by visible light so that the first video measurement unit 11a and the second video measurement unit 11b can more clearly measure the infrared signal.

  On the other hand, the body shaking measuring unit 11c measures the degree of shaking of the user's body. When measuring the hand movement, the body shake measuring unit 11c can additionally include an inertial sensor (IMU: Inertial Measurement Unit) in order to compensate for an error caused by a user body shake. At this time, the body shake measuring unit 11 c transmits the measured signal to the signal processing means 15.

  The command input means 13 is a means for receiving a control command input from a user, and includes a communication module for receiving a control command from the motion device.

  If a control command from the user is input to the position pointing device 20, the position pointing device 20 transmits the corresponding command to the motion processing device 10. Therefore, the command input means 13 receives the control command from the position pointing device 20 and transmits it to the signal processing means 15.

  The signal processing means 15 includes a two-dimensional coordinate calculation unit 15a, a three-dimensional coordinate calculation unit 15b, a body shake correction unit 15c, a pattern recognition unit 15d, and a command processing unit 15e.

  First, the two-dimensional coordinate calculation unit 15a performs two-dimensional coordinates with respect to the region where the position pointing device 20 is arranged from the images measured by the first image measurement unit 11a and the second image measurement unit 11b, that is, the region where the hand is located. Calculate At this time, the two-dimensional coordinate calculation unit 15a extracts the two-dimensional coordinates of the infrared image indicated by the point form in each measurement video.

  Thereafter, the three-dimensional coordinate calculator 15b calculates the three-dimensional coordinates at the corresponding position using the two-dimensional coordinates extracted by the two-dimensional coordinate calculator 15a. See FIG. 6 for a model for calculating the three-dimensional coordinates.

  At this time, the first image measurement unit 11a and the second image measurement unit 11b, the two-dimensional coordinate calculation unit 15a, and the three-dimensional coordinate calculation unit 15b may be combined and processed as one processor.

  The body shake correction unit 15c grasps the degree of body shake from the signal measured by the body shake measurement unit 11c. At this time, the body shake correction unit 15c corrects the body shake with respect to the image measured by the first image measurement unit 11a and the second image measurement unit 11b based on the grasped information.

  The pattern recognition unit 15d recognizes the motion pattern at the three-dimensional coordinates calculated by the three-dimensional coordinate calculation unit 15b from the video corrected by the body shake correction unit 15c.

  Thereafter, the command processing unit 15e extracts a command corresponding to the motion pattern recognized by the pattern recognition unit 15d, and transmits the command to the wearable computer via the communication unit.

  The command processing unit 15e can be connected to another device via a wired / wireless communication interface and transmit the command to the corresponding device. At this time, the three-dimensional coordinate calculation unit 15b, the pattern recognition unit 15d, and the command processing unit 15e can add a communication interface to the two-dimensional coordinate calculation unit 15a and perform processing by other external devices.

  On the other hand, FIG. 4 is a block diagram showing the configuration of the position pointing device 20 according to the present invention. Referring to FIG. 4, the position pointing device 20 according to the present invention includes a position signal generating unit 21, an input signal measuring unit 23, a signal processing unit 25, and a communication unit 27.

  The position signal generating means 21 is a means for generating a position signal for informing the current position of the corresponding position indicator device 20. The position signal generating means 21 outputs in the form of a signal that can be measured by the first image measuring unit 11a and the second image measuring unit 11b of the motion processor 10 such as infrared rays, visible rays, and lasers.

  The input signal measuring means 23 is means for receiving a control command input from the user. That is, the user instructs the effectiveness of the hand movement currently performed by the user by an operation such as a mouse click which is a computer input device. At this time, the input signal measuring means 23 recognizes a control command from the user by measuring a ring-shaped button operation, finger or wrist contact sound, electromyogram, and the like.

  For example, if the user performs an action of moving a finger in the space, the input signal measuring unit 23 interprets the current operation as an effective operation and recognizes it as a signal instructing the start of the movement. Further, the input signal measuring means 23 can measure a command for instructing the start and end of movement from the action of striking fingers and the action of spreading when there is no movement of the user or an electromyogram is used. .

  The input signal measuring unit 23 transmits the measured user control command to the signal processing unit 25.

  The signal processing means 25 includes an input signal recognition unit 25a and a command processing unit 25b.

  The input signal recognition unit 25 a recognizes the user's control command measured by the input signal measurement unit 23. At this time, the command processing unit 25 b transmits the user control command recognized by the input signal recognition unit 25 a to the command input unit 13 of the motion processor 10 through the communication unit 27.

  On the other hand, the command processing unit 25b outputs a predetermined control signal to the position signal generating unit 21 when the command notifying the start of the user movement is recognized by the input signal recognition unit 25a. Therefore, the position signal generating means 21 generates a position signal by the control signal from the command processing unit 25b.

  FIG. 5 is an exemplary diagram that is referred to for explaining the operation of the motion measuring device according to the present invention for the video measurement unit. As shown in FIG. 5, the first video measurement unit 11 a and the second video measurement unit 11 b acquire a front video. At this time, an area where the virtual display screen 2 is implemented is determined based on the angle of view of the first video measurement unit 11a and the second video measurement unit 11b.

  In other words, the first video measurement unit 11a and the second video measurement unit 11b have a certain angle of view θ, and the virtual display screen is in an area where the field angles of the first video measurement unit 11a and the second video measurement unit 11b overlap. 2 is embodied.

  Therefore, on the virtual display screen 2 in the three-dimensional space embodied as described above, the user selects an object in the virtual space by taking a gesture in the form of grasping a fist or striking a finger and holding the hand in that state. The computer can be controlled to move and move the object.

  FIG. 6 is an exemplary diagram that is referred to in order to explain the operation of the coordinate calculation method according to the present invention. Referring to FIG. 6, when a position signal is generated from the position signal generation unit 21 of the position indicator device 20, the first image measurement unit 11 a and the second image measurement unit 11 b receive position signals.

  At this time, the three-dimensional coordinate calculation unit 15b calculates the three-dimensional coordinates based on the positions of the position signal generation unit 21, the first video measurement unit 11a, and the second video measurement unit 11b. A model for calculating the three-dimensional coordinates can be obtained using Equation 1 below.

  Here, dr is the distance from the point where the position signal generated from the position signal generating means 21 reaches the first video measurement unit 11a to the center. dl is the distance from the point where the position signal generated from the position signal generating means 21 arrives at the second image measuring unit 11b to the center. L is the distance from the center of the first video measurement unit 11a to the center of the second video measurement unit 11b. Further, f is a position signal generated from the position signal generating means 21 in the direction perpendicular to the first image measuring unit 11a and the second image measuring unit 11b from the center of the first image measuring unit 11a and the second image measuring unit 11b. The distance to the point of encounter. That is, f is the focal length of the first video measurement unit 11a and the second video measurement unit 11b. Further, x is a vertical distance from the first video measuring unit 11a and the second video measuring unit 11b to the position signal generating unit 21.

  Therefore, the three-dimensional coordinate calculation unit 15b calculates the three-dimensional coordinates using the two-dimensional coordinates extracted from the two-dimensional coordinate calculation unit 15a and the x value calculated above.

  The operation of the present invention configured as described above is as follows. FIG. 7 is a flowchart illustrating an operation flow of the wearable computing environment-based user interface method according to the present invention, and illustrates an operation of the motion processor 10.

  As shown in FIG. 7, the motion processor 10 measures the user's body movement based on the signal from the position indicator 20 (S100). Further, the motion image measured using the image sensors of the first image measuring unit 11a and the second image measuring unit 11b of the motion processor 10 is processed (S110).

  Thereafter, the two-dimensional coordinate calculation unit 15a calculates the two-dimensional coordinates from the video in the “S110” process (S120), and the three-dimensional coordinate calculation unit 15b performs the two-dimensional coordinates in the “S120” process and the “S100” process. Based on the received signal, a three-dimensional coordinate is calculated (S130).

  On the other hand, the body shake measuring unit 11c measures the degree of body shake from the signal received in the “S100” process (S140), and the body shake correcting unit 15c uses the information measured in the “S140” process to determine the corresponding image. The body shake and the error due to this are corrected (S150).

  The pattern recognition unit 15d recognizes the user's motion pattern from the three-dimensional coordinates calculated in the “S130” process and the video corrected in the “S150” process (S160), and converts the pattern to the pattern recognized in the “S160” process. Corresponding command data is extracted and transmitted to the wearable computer via the communication module (S170).

  Although the wearable computing environment-based user interface apparatus and method according to the present invention have been described as being applied to wearable computing as an embodiment, not only wearable computing but also general computer interfaces Needless to say, it can also be used as a device.

  As described above, the wearable computing environment-based user interface device and the method thereof according to the present invention are not applied with the configuration and method of the above-described embodiment being limited, and the above-described embodiment is various. All or some of the embodiments may be selectively combined so that various modifications are made.

Claims (14)

A wearable computing environment based user interface device comprising:
A position indicator that is worn near the user's wrist and generates an optical signal;
A signal measuring unit including a plurality of video measuring units each including an image sensor for receiving an optical signal generated from the position pointing device and measuring a video on the front side of the user;
Analyzing each image measured by the signal measuring means to calculate a three-dimensional coordinate, recognizing the movement pattern of the user's hand on the three-dimensional coordinate from the optical signal received by the signal measuring means, A wearable computing environment-based user interface device, comprising: a signal processing means for outputting a command to perform. The signal processing means includes
The virtual display screen is implemented in a region where the angles of view of image sensors provided in the plurality of video measurement units overlap with each other, and three-dimensional coordinates are calculated from the video on the virtual display screen. 2. A wearable computing environment-based user interface device according to 1. The signal processing means includes
A two-dimensional coordinate calculation unit that extracts a two-dimensional coordinate from each image measured by the plurality of image measurement units;
The wearable computing according to claim 1, wherein three-dimensional coordinates are calculated based on the two-dimensional coordinates extracted by the two-dimensional coordinate calculation unit and hand position information detected by the optical signal. Environment-based user interface device. The video measurement unit
The wearable computing environment based user interface device of claim 1, further comprising a filter for distinguishing the position signal generated from the position pointing device and the image. The signal measuring means is
The wearable computing environment-based user interface device according to claim 1, further comprising a body shake measuring unit that measures a degree of body shake from a user's movement. The body shake measuring unit is
The wearable computing environment-based user interface device according to claim 5, further comprising an inertial sensor. The signal processing means includes
6. The body shake correction unit according to claim 5, further comprising a body shake correction unit that corrects the shake of the image measured from the image measurement unit based on the degree of body shake measured by the body shake measurement unit. Wearable computing environment-based user interface device. The position indicator machine is:
Position signal generating means for generating an optical signal indicating the position of the user's hand;
An input signal measuring means for receiving a control command input from a user,
The wearable computing environment-based user interface device according to claim 1, wherein the operation of the position signal generating unit is controlled based on a user control command input to the input signal measuring unit. A wearable computing environment based user interface method comprising:
Generating a light signal indicating the position of the user's hand from a position indicator device worn near the user's wrist;
Receiving an optical signal generated from the position pointing device and measuring a plurality of images on the front surface of the user;
Analyzing each image measured in the measuring step and calculating three-dimensional coordinates;
Recognizing the position of the user's hand from the optical signal received in the measuring step and recognizing the movement pattern of the user's hand on the calculated three-dimensional coordinates;
And a step of outputting a command corresponding to the motion pattern recognized in the step of recognizing the wearable computing environment based user interface method. The calculating step includes:
A virtual display screen is implemented in a region where the angles of view of a plurality of image sensors that measure video in the measuring step overlap each other, and three-dimensional coordinates are calculated from the video on the virtual display screen. Item 10. A wearable computing environment-based user interface method according to Item 9. The calculating step includes:
A step of extracting a two-dimensional coordinate from each image measured by the plurality of image measuring means,
The wearable computing according to claim 9, wherein three-dimensional coordinates are calculated based on the extracted two-dimensional coordinates and hand position information sensed by the optical signal of the measuring step. Environment-based user interface method. The measuring step includes
The wearable computing environment-based user interface method according to claim 9, further comprising a step of measuring a degree of body shaking from the user's movement.   The wearable computing environment-based user according to claim 12, further comprising a step of correcting the shaking of the image measured in the measuring step based on the measured degree of shaking of the body. Interface method. The optical signal is
The wearable computing according to claim 9, wherein if the user inputs a signal indicating the start and end of hand movement to the position pointing device, the wearable computing is generated based on the input signal. Environment-based user interface method.

Images