JP2020013348A - Gesture detection device, gesture detection method, and gesture detection control program - Google Patents

Abstract

To provide a gesture detection device, a gesture detection method, and a gesture detection control program that can recognize a gesture intended by an operator without increasing time and effort of a gesture detection operation.SOLUTION: A gesture detection device, which detects a gesture when an input operation is performed on an operation target based on a gesture of an operator, includes: a gesture section extraction unit 120 that extracts, based on feature amount data extracted from imaging data of the operator imaged by the imaging device 10, among gestures, a former of a set of gestures similar to each other, as a gesture immediately before a start of the input operation, and a latter of the set of similar gestures, as a gesture immediately after an end of the input operation; and a gesture identification unit 130 that identifies, as an input gesture, a gesture performed between the gesture immediately before the start and the gesture immediately after the end.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a gesture detection device that detects a gesture used for an input operation, a gesture detection method, and a gesture detection control program.

  As a conventional gesture detection device, for example, a device described in Patent Document 1 is known. The gesture detection device (hand pattern switch device) disclosed in Patent Literature 1 reliably identifies whether an operator's gesture has been performed intentionally or carelessly, and suppresses the occurrence of erroneous recognition. are doing. Specifically, when the position of the finger is stopped for a predetermined time in the shape of the finger set by the operator in the imaging region of the camera, the detection device performs the operation by the gesture with the intention of the operator. Is recognized as being started. After that, the detection device executes a gesture operation detection process based on the shape and movement of the finger.

JP 2004-145723 A

  However, in Patent Literature 1, in order to recognize the start of the gesture operation with the intention of the operator, the detection device needs to be stopped for a predetermined time with a predetermined finger shape, which makes the gesture operation complicated. (It takes time).

  In view of the above problems, it is an object of the present invention to provide a gesture detection device, a gesture detection method, and a gesture detection method capable of recognizing a gesture intended by an operator without increasing the time and effort for gesture detection. It is to provide a detection control program.

  The present invention employs the following technical means to achieve the above object.

According to a first aspect, in a gesture detection device that detects a gesture when performing an input operation on an operation target based on a gesture of an operator,
Based on the feature amount data extracted from the imaging data of the operator captured by the imaging device (10), of the gestures, the first of a set of gestures similar to each other is the gesture just before the start, and the last is the end. A gesture section extraction unit (120) for extracting a section of a gesture as a gesture immediately after;
And a gesture identification unit (130) for identifying a gesture performed between the gesture immediately before the start and the gesture immediately after the end as a gesture for input.

Further, in the second invention, in a gesture detection method for detecting a gesture when performing an input operation on an operation target based on a gesture of an operator,
Based on the feature amount data extracted from the image data of the operator imaged by the imaging device (10), of the gestures, the first of a set of gestures similar to each other is the gesture just before the start, and the latter is the end. Immediately after extracting a gesture section as a gesture,
A gesture performed between the gesture immediately before the start and the gesture immediately after the end is identified as an input gesture.

In the third invention, in a gesture detection control program for detecting a gesture when performing an input operation on an operation target based on a gesture of an operator,
Computer
Based on the feature amount data extracted from the image data of the operator imaged by the imaging device (10), of the gestures, the first of a set of gestures similar to each other is the gesture just before the start, and the latter is the end. A gesture section extraction unit (120) for extracting a section of the gesture as a gesture immediately after;
A gesture performed between the gesture immediately before the start and the gesture immediately after the end is made to function as a gesture identification unit (130) for identifying the gesture as an input gesture.

  Generally, the operator takes a similar posture at the time of starting and at the time of ending when performing a gesture. That is, when starting the predetermined gesture, the operator first takes an action of holding the hand with a predetermined finger shape (basic shape of the gesture), and after executing the predetermined gesture, the same predetermined finger shape as the first time. To end the gesture.

  Therefore, according to the present invention, by capturing a set of gestures that are similar to each other, it is possible to extract a gesture immediately before the start and a gesture immediately after the end, and extract a section of the gesture. Then, by identifying the gesture performed between the gesture immediately before the start and the gesture immediately after the end as the input gesture, for example, instead of the carelessly performed gesture, the gesture having the intention of the operator can be reliably performed. Can be recognized. Therefore, it is possible to recognize an input gesture without increasing the time and effort for gesture detection as in the related art, and it is possible to suppress occurrence of erroneous recognition.

  Note that the reference numerals in parentheses of the above means indicate the correspondence with specific means described in the embodiment described later.

It is a block diagram showing a gesture detection device. FIG. 9 is an explanatory diagram showing a procedure for extracting a candidate for starting and ending a gesture. FIG. 9 is an explanatory diagram showing a point of calculating a similarity between gesture start / end candidates. It is an explanatory view showing a point at the time of determining start and end of a gesture.

  Hereinafter, a plurality of embodiments for carrying out the present invention will be described with reference to the drawings. In each embodiment, portions corresponding to the items described in the preceding embodiment are denoted by the same reference numerals, and redundant description may be omitted. When only a part of the configuration is described in each embodiment, the other embodiments described above can be applied to other parts of the configuration. Not only the combination of the parts that clearly indicate that a combination is possible in each embodiment, but also the embodiments can be partially combined without being specified, unless there is any particular problem with the combination. It is also possible.

(1st Embodiment)
The gesture detection device 100 according to the first embodiment will be described with reference to FIGS. The gesture detection device 100 according to the present embodiment is mounted on a vehicle, and detects a gesture when performing an input operation on various vehicle devices based on a movement (gesture) of a specific part of a body of a driver (operator). Device.

  Examples of various vehicle devices include an air conditioner that air-conditions a vehicle interior, a car navigation device (hereinafter, a car navigation device) that displays a current position of a vehicle or a guide display to a destination, a television broadcast, a radio broadcast, and a CD. / DVD playback audio device. These various vehicle devices correspond to the operation target of the present invention. The various types of vehicle equipment are not limited to those described above, and include a head-up display device, a room lamp device, a rear seat sunshade device, an electric seat device, a glow box opening / closing device, and the like.

  As shown in FIG. 1, the gesture detection device 100 detects a gesture for an input operation based on image data of a specific part of a driver's body captured by the imaging device 10. Various vehicle devices are operated based on the detected gesture, and the result of the operation (operation state or the like) is displayed on the information device 20.

  The specific part of the driver's body may be, for example, the driver's finger, palm, arm, or the like. In the present embodiment, as a gesture for an input operation, a finger (a pointing gesture, a circle gesture, a wave gesture, a flick gesture, or the like) is mainly used.

  For example, in the case of an air conditioner, a change in set temperature, a change in the amount of conditioned air, and the like are performed by various input and output gestures. Location setting and the like are performed, and in the audio device, change of a television station or a radio station, selection of a music piece, change of a volume, and the like are performed.

  The imaging device 10 is configured to continuously (with time) capture the movement of a specific part of the driver's body and output captured data to the gesture detection device 100 (hand detection unit 110).

  As the imaging device 10, a camera that forms a luminance image of an object, a distance image sensor that forms a distance image, or a combination thereof can be used. Examples of the camera include a near-infrared camera that captures near-infrared light and a visible light camera that captures visible light.

  Further, as a distance image sensor, for example, a stereo camera that simultaneously captures images with a plurality of cameras and measures information in the depth direction from parallax, or the depth from the time when light from a light source is reflected by an object and returned. There is a ToF (Time of Flight) camera to measure. In the present embodiment, a camera is used as the imaging device 10.

  The information device 20 is a display unit that displays an operation state of various vehicle devices and the like, and includes an air-conditioning display unit, a car navigation display unit, an audio display unit, and the like. The information device 20 is formed by, for example, a liquid crystal display or an organic EL display.

  The gesture detection device 100 is, for example, a computer including a CPU, a ROM, a RAM, and the like, and includes a hand detection unit 110, a gesture section extraction unit 120, a gesture identification unit 130, and the like.

The hand detection unit 110 specifies an area where the driver's hand is present from the image data (image) captured by the imaging apparatus 10 and outputs data of the specified area to the gesture section extraction unit 120. ing. Note that a technique for detecting a hand is disclosed in, for example, the following literature.
・ M. Kolsch and M. Turk .: Robust Hand Detection. FGR, 2004
・ Wei Fan, Li Chen, Wei Liu, Yuan He, Jun Sun, Shutao Li: Monocular Vision Based Relative Depth Estimation for Hand Gesture Recognition. MVA 2013.

  The gesture section extraction section 120 is a section for extracting a section of a gesture intentionally performed by the driver for an input operation, and includes a start / end candidate extraction section 121, a similarity calculation section 122, a start / end determination section 123, and the like. (Details will be described later).

  The gesture identification unit 130 is a unit that identifies a gesture in a section extracted by the gesture section extraction unit 120 as a gesture intentionally performed by the driver for an input operation. The gesture identification unit 130 is a part that outputs the identified gesture to various vehicle devices and the information device 20.

  The gesture detection device 100 according to the present embodiment has the above-described configuration. Hereinafter, the operation and the effect will be described with reference to FIGS.

  The image data captured by the image capturing device 10 is output to the hand detection unit 110. The hand detection unit 110 specifies an area where a hand is present in the imaging data, and outputs data of the specified area to the start / end candidate extraction unit 121 of the gesture section extraction unit 120.

  As illustrated in FIG. 2, the start / end candidate extracting unit 121 extracts a frame start and end frame candidate from the frame of the image data output from the hand detection unit 110. Here, the frame at the start of the gesture is a frame indicating a state at the time of the start when the driver performs the gesture. For example, assuming that the gesture to be executed is a flick gesture, the start of the gesture corresponds to a state in which a hand with a finger is held first. The end of the gesture corresponds to a state in which the posture of the hand with the finger raised is maintained when the flick gesture is completed, as in the start.

  The start / end candidate extraction unit 121 uses three or more consecutive imaging frames (imaging data) to generate a static feature amount (feature amount data) based on one frame and a dynamic feature amount based on a plurality of frames. Amount (feature amount data) is extracted and combined to create one feature amount. Preferably, HOG (Histogram of Oriented Gradient) can be used as the static feature and HoOF (Histogram of Optical Flow) can be used as the dynamic feature.

  The database of the start / end candidate extraction unit 121 stores in advance reference data that has learned the classification criteria for the feature values at the start and end of the gesture and the feature values at other times. Then, the start / end candidate extraction unit 121 classifies the feature amounts extracted from the actual imaging frames based on the reference data, and extracts candidates that satisfy the criteria as the feature amounts at the start and end times as candidates. For learning and classification, a machine learning method can be used, and Random Forest can be preferably used. In FIG. 2, a portion indicated by reference numerals 1, 2, 3, and 4 is a portion (frame) extracted as a plurality of candidates at the start and end of the gesture among the continuous imaging frames.

  The similarity calculation unit 122 forms all combinations of two candidates among the plurality of candidates extracted by the start / end candidate extraction unit 121, and generates a static combination from the imaging frame of each set of candidates. A feature amount (feature amount data) is extracted, and a similarity is calculated for each set using the extracted feature amount. In FIG. 3, all combinations forming one pair are denoted by reference numerals 1 and 2, reference numerals 1 and 3, reference signs 1 and 4, reference signs 2 and 3, reference signs 2 and 4, and reference signs 3 and 4. ing. The similarity calculating unit 122 calculates the similarity of the two candidates for all of these combinations.

  The start / end determination unit 123 extracts a combination (one set) having the highest similarity among the similarities calculated by the similarity calculation unit 122, and selects the earlier (earlier) combination immediately before the start. The gesture frame is determined, and the later (later) gesture frame is determined as the gesture frame immediately after the end. Then, the start / end determination unit 123 determines a frame between the frame immediately before the start and the frame immediately after the end as a section of the gesture performed by the driver for the input operation. FIG. 4 shows that among the combinations in FIG. 3, the combination of each of the codes 1 and 2 has the highest similarity, and the area between the codes 1 and 2 is determined as the gesture section.

  Then, the gesture identification unit 130 extracts a static feature amount and a dynamic feature amount for all the imaging frames in the gesture section determined by the start / end determination unit 123. Gesture reference data indicating a criterion for classifying the feature amount of each gesture operation is stored in the database of the gesture identification unit 130 in advance. Then, the gesture identifying unit 130 identifies (classifies) which gesture operation the feature amount of the imaging frame in the gesture section is based on the gesture reference data.

  The gesture identification unit 130 outputs the identified gesture to various vehicle devices and each information device 20, so that the various vehicle devices perform an input operation according to the gesture, and each information device 20 is operated. The operation state and the like are displayed.

  In general, a driver takes a similar posture at the start and end of a gesture. That is, when starting a predetermined gesture, the driver first takes an action of holding a hand with a predetermined finger shape (basic shape of the gesture), and after performing the predetermined gesture, the same predetermined finger shape as the first time. To end the gesture.

  Thus, according to the present embodiment, the gesture section extraction unit 120 can extract a gesture section immediately before the start and a gesture immediately after the end by capturing a set of gestures similar to each other to extract a section of the gesture. Become. Then, the gesture identification unit 130 identifies the gesture performed between the gesture immediately before the start and the gesture immediately after the end as the input gesture, so that, for example, not the gesture performed carelessly but the intention of the driver. Can be reliably recognized as a gesture having a symbol. Therefore, it is possible to recognize an input gesture without increasing the time and effort for gesture detection as in the related art, and it is possible to suppress occurrence of erroneous recognition.

  Further, in the present embodiment, before the similarity calculation unit 122 performs the processing, a start / end candidate extraction unit 121 that extracts a plurality of sets of gesture candidates that are similar to each other is provided. Accordingly, by extracting a plurality of sets of gesture candidates in advance, it becomes possible for the similarity calculating unit 122 to sequentially calculate the similarity for each set of candidates, thereby reducing the calculation load. .

(Other embodiments)
In the above-described first embodiment, the hand detection unit 110 has been described as being provided. However, the hand detection unit 110 may be omitted, and image data captured by the imaging device 10 may be output to the gesture section extraction unit 120. .

  In the first embodiment described above, the start / end candidate extraction unit 121 is provided. However, the start / end candidate extraction unit 121 may be eliminated if the load of calculation is not emphasized.

  The target operator is not limited to the driver, and may be a passenger. In this case, the passenger also performs the gesture, the gesture detection is performed by the gesture detection device 100, and the various vehicle devices can be operated.

  Note that the content executed by the gesture detection device 100 described in the first embodiment corresponds to the “gesture detection method” of the present invention. In the above-described first embodiment, the gesture detection device 100 has been described as being used for gesture detection. However, for example, a control program stored in a storage medium such as a center server or a cloud may cause a computer to operate each unit. Gesture detection is possible by functioning as 110, 120, and 130.

Reference Signs List 10 imaging device 100 gesture detection device 120 gesture section extraction unit 121 start / end candidate extraction unit 122 similarity calculation unit 123 start / end determination unit 130 gesture identification unit

Claims (5)

When performing an input operation on an operation target based on a gesture of an operator, in a gesture detection device that detects the gesture,
Based on feature amount data extracted from the imaging data of the operator captured by the imaging device (10), a gesture of a set of gestures similar to each other is gestured immediately before the start of the gesture and a gesture after the gesture is started. A gesture section extraction unit (120) for extracting a section of the gesture as a gesture immediately after the end of
A gesture detection device comprising: a gesture identification unit (130) that identifies the gesture performed between the gesture immediately before the start and the gesture immediately after the end as an input gesture. A similarity calculating unit (122) that calculates a similarity for each of a plurality of sets of gestures that are similar to each other;
From the similarities calculated by the similarity calculating unit, a set of gestures having the highest similarity is extracted as the set of gestures, and a start / end determination for determining the gesture immediately before the start and the gesture immediately after the end is performed. The gesture detection device according to claim 1, further comprising: a unit (123).   The gesture detection device according to claim 2, wherein the gesture section extraction unit includes a candidate extraction unit (121) that extracts in advance the plurality of sets of gesture candidates that are similar to each other. When performing an input operation on an operation target based on a gesture of an operator, in the gesture detection method for detecting the gesture,
Based on feature amount data extracted from the imaging data of the operator captured by the imaging device (10), a gesture of a set of gestures similar to each other is gestured immediately before the start of the gesture and a gesture after the gesture is started. Is extracted immediately after the end of the gesture as a gesture,
A gesture detection method for identifying the gesture performed between the gesture immediately before the start and the gesture immediately after the end as an input gesture. When performing an input operation on an operation target based on a gesture of an operator, in a gesture detection control program for detecting the gesture,
Computer
Based on feature amount data extracted from the imaging data of the operator captured by the imaging device (10), a gesture of a set of gestures similar to each other is gestured immediately before the start of the gesture and a gesture after the gesture is started. A gesture section extraction unit (120) for extracting a section of the gesture as a gesture immediately after the end of
A gesture detection control program for causing the gesture performed between the gesture immediately before the start and the gesture immediately after the end to function as a gesture identification unit (130) that identifies the gesture as an input gesture.

Images