JP2012079138A - Gesture recognition device - Google Patents

Abstract

An object of the present invention is to provide a gesture recognition device capable of suppressing an erroneous operation of a wearable device while adopting an intuitive and natural gesture that is easy for a user to be performed as a gesture for controlling the operation of the wearable device. .
An imaging unit 2 that is directly attached to a user or attached to an information device 4 that is attached to the user, and an image of a user's finger that is imaged by the imaging unit 2 is used. The gesture recognition device 1 includes a gesture recognition unit 3 that recognizes a gesture of a user's finger that includes a gesture for creating a ring and outputs a control command corresponding to the recognized gesture to the information device 4. Constitute.
[Selection] Figure 1

Description

  The present invention relates to a gesture recognition apparatus that recognizes a gesture of a user's finger based on an image and controls an operation of a wearable device or the like according to the recognition result.

  It is difficult to control the operation of the wearable device by operating a computer keyboard, a mouse, a complex remote controller having a large number of keys, or the like. This means that the user always holds a keyboard, mouse, or remote control in his / her hand under various daily activities (walking, sitting, watching other things) This is because it is difficult. Therefore, it is important to have a means for transmitting the control content desired by the user to the wearable device even if the user does not bother to take out the keyboard, mouse, or remote controller or always have it. In other words, an interface that can control the operation of the wearable device in a hands-free state as much as possible becomes important. For example, as a wearable device, an information presentation device such as a head mount display (hereinafter referred to as HMD) or a display of a personal computer can be considered. In such a case, it is important to transmit control contents (page feed, page return, item selection, confirmation, rejection, etc.) related to information presentation to the wearable device. In addition, when the wearable device is an information input device such as a camera or a microphone, it is important to transmit the most basic control contents such as information input ON / OFF (start and end of imaging and recording) to the wearable device. become.

  As a method of transmitting the control content desired by the user to the wearable device, for example, there is a method of recognizing the user's voice and controlling the operation of the wearable device based on the voice recognition result (see, for example, Patent Document 1). .

  However, the method using voice recognition has a problem that the voice itself generated by the user causes noise of people around the user or information is disclosed to people around the user. In addition, when the surroundings of the user are noisy due to the voice of another person other than the user or noise in the environment, there is a possibility that the user's voice is erroneously recognized by reacting sensitively to the sound around the user. Therefore, there is a high possibility that the method using voice recognition cannot be effectively used under daily activities.

  Therefore, as a method for transmitting the control content desired by the user to the wearable device, for example, there is a method of recognizing the gesture of the user's finger and controlling the operation of the wearable device based on the gesture recognition result.

  As a method of recognizing a gesture, for example, there is a method of attaching various sensors (acceleration sensor, gyro sensor, etc.) to a user's finger and recognizing the gesture according to the output of these sensors.

  However, in such a gesture recognition method, an active type (with a battery) sensor (for example, an acceleration sensor or a gyro sensor) is usually used as a sensor that is worn on the user's finger. There is a risk of getting in the way.

  Therefore, a method for recognizing the gesture of the user's finger without attaching a special sensor to the user's finger is desired. For example, gesture recognition by so-called image processing, in which a gesture of a user's finger is photographed with a camera and the gesture is recognized using the photographed image, can be considered (for example, see Patent Document 1).

  However, in such a gesture recognition method, if the gesture corresponding to the control content of the wearable device is a simple gesture that is often performed under daily actions, the wearable device may cause the wearer's careless finger movement to There is a risk of malfunction.

  Further, as another gesture recognition method, for example, the content displayed on the HMD is controlled by detecting the fluctuation (movement) of the user's head by the change of the image taken by the camera mounted on the HMD. There is. In addition, there is one that controls the contents displayed on the HMD by detecting a change in movement of the marker with a camera mounted on the HMD after mounting the marker on the user's arm. For example, when the movement of the marker is small, there is one that displays a menu screen on the HMD, assuming that the user's hand is occupied (see, for example, Patent Document 2).

  However, in such a gesture recognition method based on image processing, there is no problem when a sudden change does not always occur in an image captured by the camera, such as when the user is concentrating on work, but when walking or standing. When the image can change, the HMD malfunctions easily.

  As another gesture recognition method, for example, there is a method for controlling the operation of home appliances such as a TV and an air conditioner based on a gesture of a user's finger recognized based on an image taken by a camera attached to a display. Yes (see, for example, Patent Document 3).

  However, in such a gesture recognition method based on image processing, as described above, if the gesture of the user's finger is likely to be performed under daily action, the wearable device malfunctions due to the user's careless finger movement. There is a risk of it.

  As another gesture recognition, for example, a color marker is attached to a user's fingertip, and an image projected from a projector applied to the user's neck is controlled by recognition of the user's finger gesture (for example, Non-Patent Document 1 or Non-Patent Document 2).

  However, such gesture recognition may cause an image of the projector to be projected onto a finger, resulting in a reduction in gesture recognition performance.

Japanese Patent No. 4013100 JP 2009-251154 A JP 2006-268209 A

Pranav Mistry and Pattie Maes, SixthSense: A Wearable Gestural Interface, MIT Media Laboratory, SIGGRAPH Asia 2009, Yokohama, Japan, December 16-19, 2009 Pranav Mistry and Pattie Maes, SixthSense: A Wearable Gestural Interface, MIT Media Laboratory, Emerging Technologies, Yokohama, Japan, 2009, page 85

  Since wearable devices are often used under daily behavior, it is possible to eliminate a gesture that was performed unintentionally under daily behavior from being mistakenly recognized as a gesture intended by the user. It is important as a requirement for recognition. In other words, it is important to construct a mechanism that allows the user to recognize only gestures that behave with will.

  However, even if the gesture that the user behaves with intention is a gesture that is easily accepted by the gesture recognition device (it is difficult to be erroneously recognized), the gesture must not be a behavior that the user does not behave easily. For example, a gesture that crosses the index finger and the middle finger is not necessarily a gesture that can be easily performed by all users. In other words, it is important that the gesture is a gesture that is easy to behave by the user and that is not easily misrecognized by the gesture recognition device.

  Therefore, in the present invention, there is provided a gesture recognition device capable of suppressing malfunction of a device to be controlled while adopting an intuitive and natural gesture that is easy for the user to behave as a gesture for controlling the operation of the device to be controlled. The purpose is to provide.

The gesture recognition device of the present invention includes an imaging unit and a gesture recognition unit.
The imaging means is directly attached to a user or attached to a device attached to the user.

  The gesture recognition device includes a gesture of the user's finger defined in advance including a gesture in which the user forms a ring with a thumb and an index finger using an image of the user's finger captured by the imaging unit. And a control command corresponding to the recognized gesture is output to the device to be controlled.

  The gesture recognition unit includes a finger region extraction unit and a ring region extraction unit. When the ring region is extracted by the ring region extraction unit, the gesture of the user's finger that is currently performed is You may comprise so that a user may recognize that it is a gesture which makes a ring with a thumb and an index finger.

The finger region extraction unit extracts a finger region where the user's finger exists in the video imaged by the imaging unit.
The ring area extraction unit extracts a ring area indicating a ring formed by the thumb and index finger of the user in the finger area extracted by the finger area extraction unit.

  The gesture recognizing means is configured such that when the size of the ring area extracted by the ring area extracting means is greater than or equal to a threshold value, the user's finger gesture currently being performed is determined by the user using the thumb and index finger. You may comprise so that it may be recognized as the gesture which makes a ring.

  The finger region extraction unit may include a marker detection unit and a peripheral region extraction unit, and may be configured to extract the finger region based on the peripheral region extracted by the peripheral region extraction unit. .

The marker detection means detects a marker attached to the user's finger.
The surrounding area extracting means extracts a surrounding area of the marker detected by the marker detecting means in the video imaged by the imaging means.

  In addition, the finger region extraction unit may be configured so that an image having the same color as the peripheral region extracted by the peripheral region extraction unit in the image captured by the imaging unit is used as the finger region.

  The finger region extraction unit includes a finger region prediction unit that predicts the finger region of the video of the next frame based on the finger region extracted from the image of the previous frame imaged by the imaging unit. You may comprise as follows.

The device to be controlled may be a control device that controls the content presented by the information presenting device that presents information to the user.
The device to be controlled may be a device attached to the user.

  The gesture recognition means is a first control command for switching between a first operation mode of the device to be controlled and a second operation mode of the device to be controlled, and the first operation mode. A second control command for controlling the operation of the device to be controlled, and a third control command for controlling the operation of the device to be controlled in the second operation mode, The user's finger gesture types corresponding to the second and third control commands may be the same.

  The device to be controlled may be configured to display a gesture recognition state or a gesture recognition result in the gesture recognition unit.

  According to the present invention, in a gesture recognition device for recognizing a gesture of a user's finger, an intuitive and natural gesture that is easy for a user to behave is adopted as a gesture for controlling the operation of a device to be controlled, and Malfunctions of equipment can be suppressed.

It is a figure which shows the gesture recognition apparatus of 1st Embodiment of this invention. It is a figure which shows an example of a gesture recognition part. It is a flowchart for demonstrating operation | movement of a marker detection part. It is a figure which shows an example of a square area | region. It is a figure which shows an example of a peripheral region. It is a figure which shows an example of a ring area | region. It is a figure which shows the other example of a gesture recognition part. It is a figure which shows an example of the screen displayed on the display of a wearable apparatus. It is a figure which shows an example of the screen displayed on the display of a wearable apparatus. It is a figure which shows the gesture recognition apparatus of 2nd Embodiment of this invention. It is a figure which shows the state transition of the gesture recognition in a gesture recognition apparatus. It is a figure which shows an example of the screen displayed on an information presentation apparatus. It is a figure which shows an example of the screen displayed on an information presentation apparatus. It is a figure which shows an example of the screen displayed on an information presentation apparatus.

<First Embodiment>
FIG. 1 is a diagram illustrating a gesture recognition device according to a first embodiment of the present invention.
A gesture recognition device 1 shown in FIG. 1 includes an imaging unit 2 and a gesture recognition unit 3.

  The imaging unit 2 is, for example, a small camera attached directly to the user or a small camera attached to the information device 4 attached to the user. When the imaging unit 2 is a small pendant camera that is directly worn by the user, the imaging unit 2 may be configured to have a camera shake prevention function or a camera shake reduction function. Further, the imaging unit 2 may be a small camera that is mounted on either the left or right shoulder of the user, or may be a small camera that is attached to a head ornament such as a headband. What is basically required of the imaging unit 2 is that the user can capture an intuitive finger gesture performed in front of the chest, and has an angle of view that can capture an image of the movement range of the user's finger. This is a necessary condition.

  The gesture recognition unit 3 recognizes a predefined gesture of the user's finger based on the video imaged by the imaging unit 2 and outputs a control command corresponding to the gesture to the information device 4.

  The information device 4 is, for example, an HMD with a camera or a display device with an imaging function that can present an image from the front of one eye of the user toward the eyeball. The information device 4 also provides information (for example, GUI (Graphical User Interface)) and information input operation (for example, still image capturing operation) to be presented to the user based on the control command output from the gesture recognition unit 3. To control.

FIG. 2 is a diagram illustrating the gesture recognition unit 3.
The gesture recognition unit 3 illustrated in FIG. 2 includes a marker detection unit 5, a peripheral region extraction unit 6, a finger region extraction unit 7, a ring region extraction unit 8, and a control command output unit 9.

  The marker detection unit 5 detects a marker attached to the user's finger in the video imaged by the imaging unit 2. The color and shape of the marker are not particularly limited, but the marker of the present embodiment is a green circular marker, and two markers are attached to the base of the thumb that is easily photographed by the imaging unit 2 at a predetermined interval. In addition, the two markers are arranged so that the fingertip of the thumb comes on an extension of a straight line passing through the two markers.

  The peripheral area extraction unit 6 extracts the peripheral area of the marker from the video based on the position of the marker detected by the marker detection unit 5 (the position on the coordinates of the video imaged by the imaging unit 2). For example, the peripheral area extraction unit 6 calculates a rough scale of fingers in the video based on the actual interval between the two markers and the interval between the two markers in the video, and the two in the video. Based on the position of the marker, a peripheral region where a finger is estimated to be present is calculated.

  The finger region extraction unit 7 detects pixels of a color (for example, skin color) that can be estimated as a finger in the peripheral region extracted by the peripheral region extraction unit 6, and extracts all pixels having the same color as the detected pixels from the video. The pixel region composed of all the extracted pixels is defined as a finger region.

The ring region extraction unit 8 extracts a ring region formed by the user's thumb and index finger from the finger region extracted by the finger region extraction unit 7.
The control command output unit 9 recognizes the gesture of the user's finger based on the marker detected by the marker detection unit 5 and the ring region extracted by the ring region extraction unit 8. When the control command output unit 9 determines that the recognized gesture is a predetermined gesture, the control command output unit 9 outputs a control command corresponding to the predetermined gesture to the information device 4.

  In this way, by using the marker, it is possible to easily recognize the user's finger, so that it is possible to suppress erroneous recognition of gestures due to confusion of left and right fingers, shielding of fingers, confusion of other people's fingers, and the like. .

Note that a finger region or a ring region may be extracted without using a marker.
Next, a marker detection method will be described.

  As a marker detection method, for example, there is a method in which after extracting all the green regions in the video imaged by the imaging unit 2, a green region having a high degree of circularity is extracted from the extracted green regions.

FIG. 3 is a flowchart for explaining the operation of the marker detection unit 5.
First, the marker detection unit 5 inputs an image (S1).
Next, the marker detection unit 5 calculates the correlation value C (value of 0 to 1) with the marker color using the RGB values in all pixels of the video (S2). The color of each pixel in the video is R, G, B, the marker color is R _M , G _M , B _M , the color vector of each pixel is v = (R, G, B) ^T , and the color vector of the marker is v _{When M} = (R _M , G _M , B _M ) ^T , the correlation value C is expressed by Equation 1.

The reason why the color vector of each pixel and the color vector of the marker are set as homogeneous coordinates is to prevent zero division. When the marker color is green, (R _M , G _M , B _M ) = (0, 255, 0).

Next, the marker detection unit 5 binarizes the video based on the correlation value C and a threshold value (for example, 180/256) (S3).
Next, the marker detection unit 5 performs a labeling process for dividing the binarized video into blobs (blocks) (S4). For example, a labeling process is performed using a sample code for a line labeling process using OpenCV (registered trademark).

  Next, the marker detection unit 5 calculates the circularity P for each blob (S5). Assuming that the area of the blob is A and the perimeter of the blob is S, the circularity P of the blob is expressed by Equation 2.

  And the marker detection part 5 recognizes that circularity P is more than predetermined value (for example, 0.3) as a marker, and calculates the position of the marker (S6). When two markers are recognized, the respective positions of these markers are calculated.

Next, a method for extracting a peripheral area will be described.
As a peripheral region extraction method, for example, there is a method in which a color around a detected marker is set as a “skin color template”, and pixels having a color similar to the color of the “skin color template” are extracted from the entire video. Note that the pixel color is treated as RGB converted to HSV.

  First, the peripheral area extraction unit 6 cuts out an image around the marker into a square, and sets the color of the pixel in the square area as a “skin color template”. For example, as shown in FIG. 4, first, a distance in the x direction (horizontal direction of the video) between two markers is calculated, and the distance is set to L1 (pixel).

  Next, the peripheral area extraction unit 6 determines a 2L1 (pixel) square area centering on the left marker of the two markers. Pixels satisfying the condition of 6 <H <68, V> 30 are extracted from the square area. Note that H and V represent H (hue) and V (brightness) when RGB is converted to HSV, respectively.

  Next, the surrounding area extraction unit 6 determines that a finger is present in the square area if the pixels satisfying this condition are 50% or more in the square area. In addition, the peripheral area extraction unit 6 sets the average values of hue H, saturation S, and brightness V of pixels satisfying the above conditions in the square area as H ^, S ^, V ^, and these are referred to as "skin color template". To do.

  Then, the surrounding area extraction unit 6 extracts a surrounding area that is estimated that the thumb and the index finger are inserted, using the positions of the two markers as clues. For example, as illustrated in FIG. 5, first, the peripheral region extraction unit 6 sets the distance between two markers to L2 (pixel). Next, the peripheral area extraction unit 6 calculates an angle θ formed by a line connecting the two markers and the x-axis of the coordinates in the video. Then, the peripheral area extraction unit 6 moves the position of 1.5 × L2 × cos θ (pixel) to the right and 1.5 × L2 × sin θ (pixel) to the right from the right marker of the two marks. A pixel is defined as a pixel in the lower right corner of the peripheral area, and an area having a height of 6 × L2 and a width of 8 × L2 from the pixel is defined as the peripheral area.

Next, a finger region extraction method will be described.
First, the finger region extraction unit 7 determines that the HSV values of the pixels in the peripheral region extracted by the peripheral region extraction unit 6 are H ^ −20 <H <H ^ + 15, S ^ −20 <S <S ^ + 20, When the condition of V ^ -40 <V is satisfied, the skin color is used, and in other cases, the non-skin color is used for binarization.

  Then, the finger region extraction unit 7 performs an expansion / contraction process on the skin color portion region of the binarized peripheral region, and then performs noise removal, thereby setting the skin color portion region as the finger region. As a process for smoothing the finger area, the contour line of the finger area may be smoothed (approximated) using the cvFindContours () function of OpenCV (registered trademark).

Next, a ring region extraction method will be described.
First, the ring region extraction unit 8 thins the finger region extracted by the finger region extraction unit 7.

Next, as shown in FIG. 6, the ring area extraction unit 8 detects an isolated area (area closed by a thin line) in the thinned finger area.
If the detected isolated area is equal to or larger than the predetermined area, the ring area extraction unit 8 determines that the ring area is formed by the thumb and the index finger. For example, when the area C of the isolated region satisfies the condition shown in Equation 3, it is determined that the isolated region is a ring region formed by the thumb and the index finger.

  As shown in FIG. 7, the gesture recognition unit 3 further includes a finger region prediction unit 10, and in the finger region prediction unit 10, an image of the next frame is extracted based on the finger region extracted from the image one frame before. You may comprise so that the finger area | region in may be estimated.

  In the gesture recognition device 1 according to the first embodiment, the predetermined gesture type is, for example, “knob movement operation: after a ring is formed with the thumb and index finger, the finger is moved up, down, left, and right in that state. "Gesture to move to", "Click action: Gesture that repeats the action of creating a ring with the thumb and forefinger twice from a state where the thumb and index finger are separated", "Double click action: Gesture to repeat" click action "twice" , “Triple click operation: gesture that repeats“ click operation ”three times”. In addition, if the gesture of making a ring with the thumb and forefinger only once from the state in which the thumb and forefinger are separated is used as the “click action” gesture, the “click action” is erroneously recognized due to noise. There is a possibility of being. Therefore, in consideration of the stability of the recognition of the “clicking operation”, a gesture is adopted in which an operation of forming a ring with the thumb and the index finger is repeated twice from a state where the thumb and the index finger are separated.

  As described above, in the gesture recognition device 1 according to the first embodiment, the predetermined gesture includes a gesture that forms a ring with at least the thumb and the index finger. Therefore, a gesture that is difficult to perform under daily action unless both are intended. It is. Therefore, it is unlikely that the user unconsciously performs these gestures in front of the imaging unit 2, and malfunction of the information device 4 can be suppressed.

In addition, the gesture of creating a ring with the thumb and index finger is an intuitive and natural gesture that is easy for the user to behave, and can improve operability.
The reason for making the user perform these gestures is also to make the user aware of the actual action for causing the information recognition device 4 to output a control command from the gesture recognition device 1.

  The types of control commands in the gesture recognition device 1 of the present embodiment include, for example, “page forward feed command”, “page reverse feed command”, “still image capturing command”, and “mode switching command”. Note that the types of modes executed by the information device 4 include an album playback mode and a still image capturing mode.

  For example, when the gesture recognition device 1 recognizes a “click operation” at the time of activation, the gesture recognition device 1 outputs a “mode switching command” to the information device 4. At this time, in the information device 4, the album playback mode or the still image capturing mode is executed as a mode executed at the time of activation. In the information device 4, the album playback mode and the still image capturing mode are alternately switched every time the “mode switching command” is input.

  Also, for example, when the information device 4 is executing the album playback mode, the gesture recognition device 1 confirms that the marker has moved a predetermined pixel (for example, 25 pixels) or more in a plurality of frames while detecting the ring area. To detect. Then, the gesture recognition device 1 recognizes that the “knob moving operation” is being performed by the user, and outputs a “page forward command” or “page reverse command” to the information device 4. When the “page advance command” is input, the information device 4 takes out the image immediately after the image being displayed from the memory and displays it on the display. Further, when the “page reverse feed command” is input, the information device 4 takes out the image immediately before the image being displayed from the memory and displays it on the display.

  For example, in the gesture recognition device 1, when the information device 4 is executing the still image capturing mode, the marker has moved a predetermined pixel (for example, 25 pixels) or more in a plurality of frames while detecting the ring region. Is detected. Then, the gesture recognition device 1 recognizes that the “knob moving operation” is being performed by the user, and outputs a “still image capturing command” to the information device 4. When the “still image capturing command” is input, the information device 4 acquires a still image and stores it in a memory or the like.

  As described above, in the gesture recognition device 1, at least two types of gestures are adopted as gestures for controlling the operation of the information device 4, and the first gesture (for example, “clicking operation”) is performed by the user under consciousness. Then, it is configured so that recognition of the second gesture becomes effective. Therefore, even if a gesture (for example, “knob moving operation”) that the user may unconsciously perform under daily action is adopted as the second gesture, it is possible to suppress erroneous recognition of the second gesture. it can.

In addition, since the mode switching function is installed, a plurality of operation controls can be executed with one control command, so that the operability is improved.
FIG. 8 is a diagram showing an example of a screen displayed on the display of the information device 4 when the album playback mode is being executed.

  A circular icon 11 displayed at the lower left of the display indicates a mode being executed in the information device 4. For example, when the still image capturing mode is switched to the album playback mode, the color of the icon 10 is switched from yellow to red.

Also, one image of a plurality of images stored in the memory of the information device 4 is displayed at the center of the display.
An icon 13 composed of a display window 12 and an arrow displayed below the center of the display is for guiding the user what gesture should be performed next. The video imaged by the imaging unit 2 is displayed in real time. Thus, since the video imaged by the imaging unit 2 is displayed on the display window 12, the user can check how the current gesture is captured by the imaging unit 2.

FIG. 9 is a diagram illustrating an example of a screen displayed on the display of the information device 4 when the “page advance command” is output from the gesture recognition device 1 to the information device 4.
An icon 14 including a display window 12 and an arrow displayed at the lower right of the display is for guiding the user that the “page forward command” is currently being executed. Thereafter, the information device 4 displays the next image on the display.

In addition, you may display the simple description by a character beside the icons 13 and 14 displayed on the lower part of the display of the information equipment 4. FIG.
In this way, icons and characters indicating the current mode, what gesture should be performed next, how the gesture is imaged, and the like are displayed on the display of the information device 4 in real time. It is displayed. Thereby, usability can be improved. That is, effectively displaying the imaging state of the gesture and the recognition result on the display of the information device 4 helps the user to make the gesture intuitively.

  Note that, in the above description, every time a “click operation” is recognized by the gesture recognition unit 3, the album playback mode and the still image capturing mode are alternately switched as modes to be executed by the information device 4. However, when the gesture recognition unit 3 recognizes the “double-click operation”, the gesture recognition unit 3 outputs an “album playback mode switching command” and the information device 4 executes the album playback mode. Also good. When the gesture recognition unit 3 recognizes a “triple click operation” gesture, the gesture recognition unit 3 outputs a “still image capturing mode switching command” so that the information device 4 executes the still image capturing mode. You may comprise.

<Second Embodiment>
FIG. 10 is a diagram showing a gesture recognition apparatus according to the second embodiment of the present invention. In addition, the same code | symbol is attached | subjected to the same structure as the gesture recognition apparatus 1 of 1st Embodiment shown in FIG.

  Similar to the gesture recognition device 1 of the first embodiment, the gesture recognition device 15 of the second embodiment includes an imaging unit 2 and a gesture recognition unit 3. In addition, the detection method of the finger area | region and ring area in the gesture recognition part 3 of 2nd Embodiment is the same as the detection method of the finger area | region and ring area | region in the gesture recognition part 3 of 1st Embodiment, The description is abbreviate | omitted.

  The gesture recognition device 15 according to the second embodiment is different from the gesture recognition device 1 according to the first embodiment in that the operation of the control device 16 is controlled by a control command output from the gesture recognition unit 3, and the information presentation device 17. The presentation content (for example, GUI (Graphical User Interface) etc.) is controlled. For example, the control device 16 is a personal computer, and the information presentation device 17 is a display device connected to the personal computer.

  The types of gestures that are predefined in the gesture recognition device 15 of the second embodiment are, for example, “knob movement operation” and “click operation”. The “knob moving operation” and the “clicking operation” are the same as the “knob moving operation” and the “clicking operation” in the first embodiment.

  The types of control commands in the gesture recognition device 15 of the second embodiment are, for example, “finger mouse function on command”, “finger mouse function off command”, “cursor movement command”, “click command”, and the like. Note that cursor movement and click have almost the same meaning as cursor movement and click of an existing mouse function used in a personal computer or the like, and similar words are used for convenience. That is, in a state where no application is activated in the control device 16, the same operation as the cursor movement or click of the existing mouse function is performed. However, when an application is activated, an operation different from cursor movement or click of an existing mouse function may be performed depending on the specification of the application.

  The “finger mouse function on command” is a control command for switching the finger mouse function mounted on the control device 16 from off to on. When the finger mouse function is turned on, a cursor is displayed on the information presentation device 17. The For example, when the gesture recognition device 15 recognizes a marker continuously for a plurality of frames (for example, 20 frames (about 2 seconds when the imaging unit 2 can capture an image of 10 frames / second)) or more, The “finger mouse function on command” is output from the gesture recognition device 15. Basically, the user does not need to be particularly aware of this control command, and when the user causes the imaging unit 2 to image a finger with a marker, the finger mouse function is automatically turned on. In addition, in this way, the recognition of the marker continuously for a plurality of frames or more in the gesture recognition device 15 is intended to prevent the marker from being erroneously recognized to some extent when a marker-like thing appears irregularly in the frame due to noise. It is said.

  The “cursor moving command” is a control command for moving the cursor displayed on the information presenting device 17 when the finger mouse function is on. When the gesture recognition device 15 recognizes the “knob movement operation”, the “cursor movement command” corresponding to the amount of movement of the marker is output from the gesture recognition device 15. For example, when the marker moves to the right by a predetermined pixel (for example, 25 pixels) or more in a plurality of frames while detecting the ring region, the cursor displayed on the display of the information presentation device 15 moves to the right by the distance corresponding to the amount of movement of the marker. Move to.

  The “click command” is a control command for selecting a button or the like displayed on the display of the information presentation device 15 when the finger mouse function is on. When the “click operation” is recognized by the gesture recognition device 15, a “click command” is output from the gesture recognition device 15.

  The “finger mouse function off command” is a control command for switching the finger mouse function from on to off. When the finger mouse function is turned off, the cursor displayed on the information presenting device 17 disappears. For example, if the marker is not continuously recognized by the gesture recognition device 15 for more than a plurality of frames (for example, 100 frames (about 10 seconds when the imaging unit 2 can capture a moving image at 10 frames / second)) or more, A mouse function off command ”is output from the gesture recognition device 15. Similar to the “finger mouse function on command”, basically, the user does not need to be particularly aware of this control command, and if the user does not cause the imaging unit 2 to photograph a finger with a marker, the finger mouse function is automatically provided. Turns off. Note that a button for turning off the finger mouse function is displayed on the information presenting device 17, and a “finger mouse function off command” is output from the gesture recognition device 15 when the button is selected. Good.

Next, gesture recognition state transition in the gesture recognition device 15 will be described.
FIG. 11 is a diagram for explaining the state transition of gesture recognition in the gesture recognition device 15.

  As shown in FIG. 11, the gesture recognition state in the gesture recognition device 15 includes four states (“finger mouse function ON state: FU_STATE_FREE”, “cursor movement state: FU_STATE_GRIPMOVE”, respectively) corresponding to the above four control commands. All six states are defined: "Click state: FU_STATE_CLICK", "Finger mouse function off state: FU_STATE_OFF") and two preliminary states ("Temporary ring detection state: FU_STATE_GRIP", "Pre-click detection state: FU_STATE_PRECLICK") Is done. Although not shown, a preliminary flag is provided to assist the determination in the two preliminary states, and the two states of true and false are maintained.

  The “finger mouse function off state” is a state where marker non-detection continues for a predetermined frame or more, and the cursor is not displayed on the information presenting device 17. When the gesture recognition device 15 or the control device 16 is activated, the state transition of the gesture recognition starts from the “finger mouse function off state”. If the marker detection continues for a predetermined frame in the “finger mouse function off state”, the state transits to the “finger mouse function on state”. At this time, the gesture recognition device 15 outputs a “finger mouse function on command”.

  The “finger mouse function ON state” is a state in which marker detection continues for a predetermined frame or more, and a cursor is displayed on the information presentation device 17. If the non-detection of the marker continues for a predetermined frame in the “finger mouse function on state”, the state transits to the “finger mouse function off state”. At this time, the gesture recognition device 15 outputs a “finger mouse function off command”. In addition, when the gesture recognition device 15 detects the ring area in the “finger mouse function on state”, the gesture recognition device 15 transits to the “ring temporary detection state”.

  The “ring temporary detection state” is a state immediately after the ring region is detected, and is a state in which the cursor is displayed on the information presenting device 17. When the gesture recognition device 15 transitions from the “finger mouse function on state” or the “pre-click state” to the “ring temporary detection state”, the gesture recognition device 15 sets the reserve flag to true. Next, if the “ring temporary detection state” continues for a predetermined frame without recognizing the “knob movement operation”, the gesture recognition device 15 sets the preliminary flag to FALSE. Then, when the ring flag is not detected when the reserve flag is false, the gesture recognition device 15 transitions from the “ring temporary detection state” to the “finger mouse function on state”. On the other hand, when the preliminary flag is true, the gesture recognition device 15 transitions from the “ring temporary detection state” to the “pre-click state” when the ring region is not detected. In addition, when the gesture recognition device 15 recognizes the “knob moving operation” in the “ring temporary detection state”, the gesture recognition device 15 transitions from the “ring temporary detection state” to the “cursor moving state” and sets the preliminary flag to false.

  The “cursor moving state” is a state in which the “knob moving operation” is continuously recognized, and the cursor displayed on the information presenting device 17 is moving. When the “knob moving operation” is no longer recognized, the gesture recognition device 15 transitions from the “cursor moving state” to the “finger mouse on state”.

  The “pre-click detection state” is a state in which the ring region is not detected in the “ring temporary detection state”, and a cursor is displayed on the information presenting device 17. When the gesture recognition device 15 detects the ring area again within a predetermined frame (for example, 5 frames) in the “previous click detection state”, the gesture recognition device 15 transits to the “click detection state”. Further, when the gesture recognition device 15 detects the ring region again after exceeding a predetermined frame (for example, 5 frames), the gesture recognition device 15 transits to the “ring temporary detection state”. If the ring area is not detected again even after exceeding a predetermined frame (for example, 5 frames), the gesture recognition device 15 transitions to the “finger mouse function on state”.

  The “click detection state” is a state immediately after the “click operation” is recognized, and immediately transitions to the “finger mouse function on state”. In general, the user's thumb and index finger after performing a “click operation” tend to be separated from each other, and therefore, the “finger mouse-on state” is set immediately after the “click detection state”. . Even if the user's thumb and index finger are in contact with each other after performing a “click operation,” the “finger mouse function on state” transitions to the “ring temporary detection state”. There is no problem even if a transition to the “finger mouse function on state” is made immediately.

  For example, when the gesture recognition device 15 detects a marker attached to the user's finger at a predetermined frame or more at the time of activation, the gesture recognition device 15 transitions from the “finger mouse function off state” to the “finger mouse function on state” to generate a “finger mouse function command”. Is output. Then, for example, as shown in FIG. 12, a cursor 18 is displayed on the information presenting device 17. In the screen example displayed on the information presentation device 17 shown in FIG. 12, in addition to the cursor 18, the image display window 19 for displaying the image taken out from the memory and the next image are taken out from the memory. A button 20 for displaying on the image display window 19, a button 21 for extracting the previous image from the memory and displaying it on the image display window 19, and a video for displaying the current video imaged by the imaging unit 2 A display window 22, a button 23 for executing a “slide show”, and a state transition display window 24 for displaying the current gesture recognition state are displayed.

  In addition, for example, when the gesture recognition device 15 detects the ring area and recognizes the “knob moving operation” in the “finger mouse function on state”, the gesture recognition device 15 transitions to the “ring temporary detection state” and then “cursor movement” Transition to "state". Then, the gesture recognition device 15 outputs a “cursor movement command” corresponding to the amount of movement of the marker. For example, the cursor 18 shown in FIG. 12 moves onto the button 23 as shown in FIG.

  Further, for example, when the gesture recognition device 15 recognizes the “click operation” in the “finger mouse function on state”, the gesture recognition device 15 transits to “temporary ring detection state”, “previous click detection state”, and “click detection state”. To output a “click command”. For example, as illustrated in FIG. 13, when the “click command” is output from the gesture recognition device 15 when the cursor 18 is present on the button 23, the information presenting device 17 may display the click command as illustrated in FIG. 14. The image display window 19 is enlarged and displayed, and the images displayed in the image display window 19 are switched one after another.

  Thus, even the predetermined gestures employed in the gesture recognition device 15 of the second embodiment are gestures that are difficult to perform under daily action unless they are intended. Therefore, it is unlikely that the user will unintentionally perform these gestures in front of the imaging unit 2, and malfunction of the control device 16 can be suppressed.

  In addition, the operation of putting and releasing the thumb and index finger is an intuitive and natural gesture that is easy for the user to behave, and can improve operability.

DESCRIPTION OF SYMBOLS 1 Gesture recognition apparatus 2 Imaging part 3 Gesture recognition part 4 Information equipment 5 Marker detection part 6 Peripheral area extraction part 7 Finger area extraction part 8 Ring area extraction part 9 Control command output part 10 Finger area prediction part

Claims (10)

Imaging means attached directly to a user or attached to a device worn by the user;
Recognizing and recognizing a predefined gesture of the user's finger, including at least a gesture in which the user forms a ring with a thumb and an index finger, using an image of the user's finger imaged by the imaging means Gesture recognition means for outputting a control command corresponding to the gesture to the controlled device;
A gesture recognition device comprising: The gesture recognition device according to claim 1,
The gesture recognition means includes:
A finger area extracting means for extracting a finger area where the user's finger is present in the video imaged by the imaging means;
A ring area extracting means for extracting a ring area indicating a ring formed by the thumb and index finger of the user in the finger area extracted by the finger area extracting means;
With
When the ring area is extracted by the ring area extraction unit, the gesture of the user's finger that is currently performed is recognized as a gesture in which the user forms a ring with a thumb and an index finger. Recognition device. The gesture recognition device according to claim 2,
When the size of the ring region extracted by the ring region extraction unit is greater than or equal to a threshold value, the gesture recognition unit determines that the user's finger gesture is being performed with a thumb and an index finger. A gesture recognition device that recognizes a gesture to be created. The gesture recognition device according to claim 2,
The finger region extraction means includes
Marker detection means for detecting a marker attached to the user's finger;
A peripheral area extracting means for extracting a peripheral area of the marker detected by the marker detecting means in the video imaged by the imaging means;
With
The gesture recognition device characterized in that the finger region is extracted based on the peripheral region extracted by the peripheral region extraction means. The gesture recognition device according to claim 4,
The gesture recognition apparatus according to claim 1, wherein the finger region extraction unit uses an image of the same color as the peripheral region extracted by the peripheral region extraction unit in the video imaged by the imaging unit. The gesture recognition device according to claim 4,
The finger region extraction unit includes a finger region prediction unit that predicts the finger region of the video of the next frame based on the finger region extracted in the image of the previous frame captured by the imaging unit. A feature gesture recognition device. The gesture recognition device according to claim 1,
The gesture recognition apparatus, wherein the device to be controlled is a control device that controls the presentation content of an information presentation device that presents information to the user. The gesture recognition device according to claim 1,
The gesture recognition apparatus, wherein the device to be controlled is a device worn by the user. The gesture recognition device according to claim 1,
The gesture recognition means is a first control command for switching between a first operation mode of the device to be controlled and a second operation mode of the device to be controlled. Outputting a second control command for controlling the operation of the device to be controlled and a third control command for controlling the operation of the device to be controlled in the second operation mode;
The gesture recognition device, wherein the types of gestures of the user's fingers respectively corresponding to the second and third control commands are the same. The gesture recognition device according to claim 1,
The device to be controlled displays a gesture recognition state or a gesture recognition result in the gesture recognition unit.