JP2018010538A - Image recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image recognition device capable of accurately identifying rotational motion of gesture manipulation using a simple system.SOLUTION: An image recognition device includes; a camera 10 functioning as imaging means for imaging a user's hand 80 designated as an imaging target from a predetermined direction; and a controller configured to compute an inclination of a motion trajectory of a reference point of the imaging target based on image information in a plane perpendicular to the predetermined direction used by the camera 10 for imaging, and to detect a predetermined type of motion of the hand 80 based on the number of polarity inversions of the motion trajectory inclination.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image recognition device, and more particularly to an image recognition device that determines a gesture motion (rotation motion) by image recognition.

  As a conventional technique, there has been proposed an image recognition device that provides an intuitive and easy-to-understand interface for a user by using gesture recognition in combination and provides a device capable of complicated input operations (for example, (See Patent Document 1). This image recognition device performs an input operation of a device by using a combination of gesture recognition based on video captured by an imaging unit and contact detection unit such as a touch panel.

  This image recognition apparatus performs processing for recognizing a hand shape from an image taken by an imaging unit. Extract gesture feature from recognized hand shape and save. The gesture feature value is information for specifying a gesture, and is stored in the gesture feature value database in association with the control command. For example, a hand-shaped area, the number of fingers being operated, and fingertip position information. The position of the fingertip of the thumb is registered as an initial value from the video acquired by the imaging means by gesture recognition processing. Next, when the operator rotates the thumb to change continuous parameters, the amount of rotation is recognized by gesture recognition.

JP 2009-42796 A

  However, in a situation where the image recognition device of Patent Document 1 is installed in, for example, a passenger compartment, the image recognition device vibrates, and as a result, the operation locus tends to become unstable, resulting in an extreme movement locus or a direction different from the movement direction. Therefore, there is a possibility that the operation is erroneously determined. Further, in the vehicle interior, the space in which the camera can be mounted is limited, and the shooting position and direction are restricted. Therefore, there is a possibility that the movement locus is distorted due to the relationship between the camera installation position and the operation position. For this reason, the conventional image recognition apparatus has a problem that it is difficult to accurately determine the rotational motion.

  Accordingly, an object of the present invention is to provide an image recognition apparatus that can accurately determine a rotation operation by a gesture operation with a simple system.

[1] In order to achieve the above object, the imaging object is based on imaging means for imaging an imaging object from a predetermined direction and image information in a plane orthogonal to the predetermined direction imaged by the imaging means. A control unit that calculates the inclination of the motion trajectory of the reference point of the object and detects a predetermined motion type of the imaging object based on the number of positive and negative inversions of the slope of the motion trajectory. An image recognition apparatus is provided.

[2] The image recognition apparatus according to [1], wherein the imaging unit is a monocular camera, and the predetermined direction is an optical axis direction of the monocular camera.

[3] The control unit may determine that the predetermined operation type of the imaging target object is a rotation operation when the number of positive / negative inversions is 4 or more. 2] may be used.

  According to the image recognition apparatus of the present invention, it is possible to provide an image recognition apparatus that can accurately determine a rotation operation by a gesture operation with a simple system.

FIG. 1 is a schematic perspective view showing an example of a camera arrangement of the image recognition apparatus according to the embodiment of the present invention. FIG. 2 shows an example of a block diagram of the image recognition apparatus according to the embodiment of the present invention. 3A is a frame image S (n) captured on the XY coordinates viewed in the Z-axis direction in FIG. 1, and FIG. 3B is a frame image S (0) of the background image 110. FIG. 3C shows an image obtained by removing the background image by the difference between the frame image S (n) and the frame image S (0) and binarizing the image. FIG. 4 is a flowchart showing the operation of the image recognition apparatus according to the embodiment of the present invention. FIG. 5 is a graph showing an example of F1 (n). FIG. 6 shows an example of a determination point of F1 (n) × F1 (n−1) <0 (an example of a rotation operation) when the trajectory inclination formula for each fixed frame is F1 (n) = (Yn / Xn). FIG. FIGS. 7A to 7F are exemplary diagrams showing the trajectory of the rotational motion to be determined. FIGS. 8A to 8D are illustrations showing the locus of the payment operation to be determined.

(Embodiment of the present invention)
An image recognition apparatus 1 according to an embodiment of the present invention includes a camera 10 as an imaging unit that images an operator's hand 80 that is an object to be imaged from a predetermined direction, and a direction orthogonal to a predetermined direction captured by the camera 10. The inclination of the motion trajectory of the reference point of the imaging object is calculated based on the in-plane image information, and the predetermined motion type of the hand 80 is detected based on the number of positive and negative inversions of the slope of the motion trajectory. And a control unit 30.

  The predetermined direction is set in the optical axis direction (Z-axis direction, upward direction) of the camera 10.

  FIG. 1 is a schematic perspective view showing an example of a camera arrangement of the image recognition apparatus according to the embodiment of the present invention. FIG. 2 shows an example of a block diagram of the image recognition apparatus according to the embodiment of the present invention. 3A is a frame image S (n) imaged on the XY coordinates viewed in the Z-axis direction in FIG. 1, and FIG. 3B is a frame image S (0) of the background image 110. FIG. 3C shows an image obtained by removing the background image based on the difference between the frame image S (n) and the frame image S (0) and binarizing the image.

  As shown in FIG. 1, in the image recognition apparatus 1, a camera 10 as an imaging unit is arranged in the upward direction, and images the A direction shown in the figure, that is, the optical axis direction (Y-axis direction) of the camera 10. In the present embodiment, the camera 10 is a single-eye system. The camera 10 includes a semiconductor image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The infrared illumination 20 illuminates a range that is substantially the same as the imaging range 60 of the camera 10. In addition, although the light of the light to image can use the light of arbitrary frequencies, it illuminates and images using infrared light and near-infrared light. Thereby, an operator does not see illumination light at the time of imaging, and the uncomfortable feeling at the time of imaging can be prevented or reduced.

(Camera 10)
As illustrated in FIG. 1, the camera 10 captures an image of the operator's hand 80 in the A direction, and image data that is image information in the XY plane orthogonal to the optical axis direction (Z-axis direction) of the camera 10. The frame image 100 is output to the control unit 30 at a predetermined frame period. This frame image 100 is a frame image including a background 110 as shown in FIG. As shown in FIG. 2, the frame image 100 sequentially outputs the frame image S (n) of the nth frame at a certain time to the control unit 30 at a predetermined timing (for example, 1/60 seconds).

  Note that only the background image 110 shown in FIG. 3B is captured in advance, and this background image 110 is used as a frame image S (0) for background processing described later.

(Lighting 20)
As shown in FIGS. 1 and 2, a lighting unit 20 is connected to the control unit 30. Note that a plurality of lights may be connected to the control unit 30. As shown in FIG. 1, the illumination 20 illuminates the imaging target by irradiating infrared light onto a range that covers the imaging range (angle of view) of the camera 10.

(Configuration of control unit 30)
As shown in FIG. 2, one camera 10 is connected to the control unit 30. In addition, the illumination unit 20 is connected to the control unit 30.

  The control unit 30 includes, for example, a CPU (Central Processing Unit) that performs operations and processing on acquired data according to a stored program, a RAM (Random Access Memory) that is a semiconductor memory, a ROM (Read Only Memory), and the like. Microcomputer. For example, a program for operating the control unit 30 is stored in the ROM. For example, the RAM is used as a storage area for storing a plurality of pieces of image information, temporary calculation results, and the like. In addition, the control unit 30 has a means for generating a clock signal therein, and executes processing based on the clock signal.

  The control unit 30 includes an image processing unit 31, a centroid detection unit 32, a reference point detection unit 33, an inter-frame calculation unit 34, and an operation identification unit 35.

(Image processing unit 31)
The image processing unit 31 executes image processing such as background removal and binarization in advance in order to perform post-process gravity center detection, reference point detection, and the like for each frame image S (n) input from the camera 10. It is a pre-process for.

  3A is a frame image S (n) captured on the XY coordinates viewed in the Z-axis direction in FIG. 1, and FIG. 3B is a frame image S (0) of the background image 110. FIG. 3C shows an image obtained by removing the background image by the difference between the frame image S (n) and the frame image S (0) and binarizing the image. The frame image S (0), which is the background image, is a reference image for the difference image calculation process, and is preferably an image that does not change with time.

  The image processing unit 31 extracts a difference between the frame image S (n) and the frame image S (0) of the background image 110, removes the difference from the frame image S (n), and the frame image S of only the hand 80. Create (n). Also, this is binarized to create a binarized frame image S (n) of the hand 80 shown in FIG. In the following description, it is assumed that each process is executed by the binarized frame image S (n).

(Centroid detection unit 32)
The centroid detection unit 32 detects the position of the centroid G by calculating the position of the centroid G based on the binarized frame image S (n). The X coordinate of the position of the center of gravity G is an average value of the numerical values in FIG. The average value can be calculated by dividing the sum of the X coordinate values of the pixels where the hand 80 is present by the number of pixels where the hand 80 is present. Similarly, the Y coordinate of the position of the center of gravity G is the average value of the numerical values in FIG. For the average value, the center of gravity G (X, Y) is calculated by dividing the sum of the Y coordinate values of the pixels where the hand 80 is present by the number of pixels where the hand 80 is present.

(Reference point detector 33)
The reference point detection unit 33 detects the farthest point P from the calculated center of gravity G shown in FIG. The reference point is not limited to the point farthest from the center of gravity G as long as it is a specific position of the hand 80.

  The reference point detection unit 33 extracts the fingertip portion from the wrist by, for example, template matching from the image obtained as shown in FIG. In this extracted hand frame image, the X coordinate and Z coordinate farthest from the center of gravity G are obtained. Thereby, the point P farthest from the center of gravity G can be detected, and this is set as the reference point (x, y) of the motion trajectory. Note that the point P farthest from the center of gravity G in the hand is, for example, a fingertip, as shown in FIGS.

(Inter-frame operation unit 34, operation identification unit 35)
The inter-frame calculation unit 34 executes various calculations from the frame images S (n) sequentially input from the camera 10 at a predetermined timing. The inter-frame computing unit 34 detects the x coordinate and the y coordinate (x, y) of the reference point based on the frame image S (n).

  Here, the term “several frames” refers to between five or less frames, “certain frames” refers to frames of about one digit, and “window frames” refers to a certain number of frames. Between window frames is larger than between certain frames. The number of frames varies depending on conditions of recognition accuracy and determination speed. In the present embodiment, it is assumed that several frames are 9 frames, certain frames are 3 frames, and window frames are 500 frames.

The inter-frame operation unit 34 and the operation identification unit 35 perform the following calculations and determinations.
(1) The inter-frame calculation unit 34 stores the coordinates (x _t , y _t ) of the reference point extracted by the reference point detection unit 33 at a certain time t.
(2) A unit movement distance D = √ ((x _t −x _t−3 ) ² + (y _t −y _t−3 ) ² )) is calculated.
Note that √ () is equivalent to () ^1/2 .
(3) The value T = a / D between unit frames is calculated. a is a constant.
(4) A time change value GX (t) = x _t −x _t−T of the X value between certain frames is calculated. Similarly, a time change value GY (t) = y _t −y _t−T of the Y value is calculated.
(5) GX (t) × GX (t−1) or GY (t) × GY (t−1) is calculated.
(6) Starting from several frames after the frame falls within the threshold, Yn = y _t −y _t−3 and Xn = x _t −x _t−3 are calculated for every fixed frame (T = 3). Then, the trajectory inclination formula for every fixed frame, F1 (n) = (Yn / Xn) is calculated. When Xβ = 0, it is assumed that F1 (β) = F1max.
(7) The operation identification unit 35 determines whether there are four or more points where the sign of F1 (n) is reversed between the window frames. If it is less than 4 points, it is determined as a paying operation.
(8) The operation identification unit 35 determines whether GX (t) × GX (t−1) and GY (t) × GY (t−1) are equal to or less than zero between window frames. If at least one of them is always greater than zero, it is determined that the payment operation is performed.

(Operation of the image recognition apparatus 1)
FIG. 4 is a flowchart showing the operation of the image recognition apparatus according to the embodiment of the present invention. The operation of the image recognition apparatus 1 will be described below according to this flowchart.

The inter-frame computing unit 34 first calculates a unit movement distance D = √ ((x _t −x _t−3 ) ² + (y _t −y _t−3 ) ² ) (Step 01).

  The inter-frame calculator 34 determines whether the unit movement distance D is zero (Step 02). If the unit movement distance D is zero, it is determined that the operator's hand 80 is not moving, and the process returns to Step 01 (Step 02: Yes). If the unit movement distance D is not zero, the process proceeds to Step 03 (Step 02: No). ).

  The inter-frame computing unit 34 calculates a value T = a / D between unit frames (Step 03). a is a constant.

The inter-frame operation unit 34 calculates a time change value GX (t) = x _t −x _t−T of the X value between certain frames (Step 04).

Similar to Step 04, the inter-frame computing unit 34 calculates a time change value GY (t) = y _t −y _t−T of the Y value (Step 05).

  The inter-frame operation unit 34 determines whether GX (t) × GX (t−1) or GY (t) × GY (t−1) is equal to or less than zero (Step 06). If GX (t) × GX (t−1) or GY (t) × GY (t−1) is less than or equal to zero, the process proceeds to Step 07 (Step 06: Yes), and if not less than zero, returns to Step 01. Then, the movement distance measurement is continued (Step 06: No).

  The inter-frame operation unit 34 sets n = n + 1 and proceeds to the next frame (Step 07).

The inter-frame operation unit 34 acquires x _n , y _n , X ′ _n , and Y ′ _n (Step 08).

  The inter-frame operation unit 34 determines whether n <3 (Step 09). If n <3, the fixed frame number (n = 3) has not been reached, so the process returns to Step 07 (Step 09: Yes). If n <3, the fixed frame number (n = 3) has been reached. It progresses to Step10 (Step09: No).

Interframe arithmetic unit 34 _{calculates Xn = x t -x t-3} , Yn = y t -y t-3 (Step10).

  The inter-frame computing unit 34 determines whether Xn is zero (Step 11). If Xn is not zero, the process proceeds to Step 12, which is the next division process (Step 11: No), and if Xn is zero, the process proceeds to Step 13 (Step 11: Yes).

  The inter-frame computing unit 34 proceeds to Step 14 with F1 (n) = F1max (Step 13).

  The inter-frame calculation unit 34 calculates a trajectory inclination formula for every fixed frame (n = 3), F1 (n) = (Yn / Xn) (Step 12). FIG. 5 is a graph showing an example of F1 (n) having an inversion point.

  The inter-frame operation unit 34 determines whether n <4 (Step 14). If n <4, the process returns to Step 07 (Step 14: Yes), and if not n <4, the process proceeds to Step 15 (Step 14: No).

  The inter-frame computing unit 34 determines that F1 (n) × F1 (n−1)> 0 (Step 15). FIG. 6 shows an example of a determination point of F1 (n) × F1 (n−1) <0 (an example of a rotation operation) when the trajectory inclination formula for each fixed frame is F1 (n) = (Yn / Xn). FIG. If F1 (n) × F1 (n−1)> 0 is not satisfied, the process proceeds to Step 16 (Step 15: No), and if F1 (n) × F1 (n−1)> 0, the process proceeds to Step 17 (Step 15: Yes). ).

  The inter-frame computing unit 34 sets the inversion number A to A = A + 1 (Step 16).

  The inter-frame computing unit 34 determines whether n <500 (Step 17). If n <500, it is determined that the number of window frames has not been reached, and the process returns to Step 07 (Step 17: Yes). If n <500, the process proceeds to Step 18 (Step 17: No).

  The operation identification unit 35 determines whether A <4 (Step 18). When the inversion number A is 4 <0, the process proceeds to Step 20 (Step 18: Yes), and when the inversion number A is not 4 <0, the process proceeds to Step 19 (Step 18: No).

  The operation | movement identification part 35 determines with payment operation | movement (Step20).

  The motion identification unit 35 determines whether GX (t) × GX (t−1) or GY (t) × GY (t−1) is greater than zero (Step 19). When smaller than zero, it progresses to Step21 (Step19: No), and when larger than zero, it progresses to Step22 (Step19: Yes).

  The motion identification unit 35 determines that the motion is a rotational motion (circle motion) (Step 21).

  The operation identification unit 35 determines that the payment operation is performed (Step 22).

  FIGS. 7A to 7F are exemplary diagrams illustrating the locus of the rotational motion determined by the motion identification unit 35. The trajectory of this rotational motion includes a rotational motion that is difficult to determine with the prior art. FIGS. 8A to 8D are exemplary diagrams illustrating the locus of the payment operation determined by the operation identification unit 35.

(Effect of embodiment)
The embodiment of the present invention has the following effects.
(1) An image recognition apparatus 1 according to an embodiment of the present invention includes a camera 10 as an imaging unit that images an operator's hand 80 that is an imaging target from a predetermined direction, and a predetermined image captured by the camera 10. Based on the image information in the plane orthogonal to the direction, the inclination of the motion locus of the reference point of the imaging object is calculated, and the predetermined motion type of the hand 80 is determined based on the number of positive and negative inversions of the inclination of the motion locus. And a control unit 30 for detecting. When the predetermined direction is the optical axis direction of the camera, it is difficult to identify the operation type (rotation operation, payment operation), but according to the present embodiment, the inclination of the operation locus of the reference point of the hand 80 is positive or negative. Since the determination is made based on whether or not there are a plurality of inversion points, it is possible to easily determine the rotation operation.
(2) With this, it is possible to realize an image recognition device that can accurately determine a rotation operation by a gesture operation with a simple system.
(3) Since it is a monocular camera system, it is possible to construct an image recognition apparatus at a lower cost than a stereo camera or a TOF camera, and there is an effect that a processing load is light and a processing time is not required.
(4) Even when the image recognition apparatus 1 according to the present embodiment is installed in the vehicle interior, the motion locus becomes unstable, becomes an extreme movement locus, or includes a waveform that amplitudes in a direction different from the moving direction. It is possible to accurately determine and discriminate the rotation operation and the payment operation without the operation.

  As mentioned above, although some embodiment of this invention was described, these embodiment is only an example and does not limit the invention which concerns on a claim. Moreover, these novel embodiments can be implemented in various other forms, and various omissions, replacements, changes, and the like can be made without departing from the scope of the present invention. In addition, not all the combinations of features described in these embodiments are essential to the means for solving the problems of the invention. Furthermore, these embodiments and modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 ... Image recognition apparatus 10 ... Camera 20 ... Illumination 30 ... Control part, 31 ... Image processing part, 32 ... Gravity center detection part, 33 ... Reference | standard point detection part, 34 ... Inter-frame calculation part, 35 ... Motion identification part 60 ... Imaging Range 80 ... hand 100 ... frame image, 110 ... background

Claims (3)

Imaging means for imaging an imaging object from a predetermined direction;
Based on image information in a plane orthogonal to the predetermined direction imaged by the imaging means, the inclination of the motion trajectory of the reference point of the imaging object is calculated, and the number of positive / negative reversals of the inclination of the motion trajectory is calculated. Based on a control unit for detecting a predetermined operation type of the imaging object;
An image recognition apparatus comprising:   The image recognition apparatus according to claim 1, wherein the imaging unit is a monocular camera, and the predetermined direction is an optical axis direction of the monocular camera.   The image recognition apparatus according to claim 1, wherein the control unit determines that the predetermined operation type of the imaging object is a rotation operation when the number of positive / negative inversions is 4 or more. .