CN102200830A - Non-contact control system and control method based on static gesture recognition - Google Patents

Abstract

The invention provides a non-contact control system and method based on static gesture recognition. The non-contact control system based on static gesture recognition has: a camera unit, which shoots a video of the current user; a processing unit, which recognizes the gesture of the user from the video of the user captured by the camera unit, and converts the recognized gesture into a corresponding The operation instruction of the processing unit; the target unit executes the operation according to the operation instruction of the processing unit. Thus, a simple and highly reliable real-time non-contact control system is realized.

Description

Non-contact control system and control method based on static gesture recognition

technical field

The invention relates to a non-contact control system and control method based on static gesture recognition.

Background technique

Gesture is a natural and intuitive mode of interpersonal communication. Control based on gesture recognition can be used in various devices such as TVs, computers, electronic whiteboards, and electronic advertising screens.

In Patent Document 1, a device control system based on gesture recognition is proposed to replace an existing remote control of a television. Figure 12 is a schematic diagram of the control system. When using the control system, users can use gestures to remotely operate the display at a distance. For example, after the user's "trigger" gesture is recognized by the system, the icons on the display will follow the movement of the user's hand, so it can be used to perform various operations such as volume adjustment, channel switching, color adjustment, brightness adjustment, etc. control.

The device control system consists of two parts: a specific TV camera and a display. A specific TV camera is used to shoot video of the user, and the TV camera also includes a computing device for processing the captured video. In processing, the TV camera first removes the background in the video, then recognizes the user's gestures, and finally generates corresponding instructions for the display to execute. The monitor then shows the result of the recognition and executes the instructions from the TV camera.

Fig. 13 is a flowchart of the device control system. The device control system is implemented in the following way: First, a raw video is taken, and the video is decoded into an image. Afterwards, background segmentation is used to remove the background from the image. To recognize the "trigger" gesture, an associative technique is used. Once a "trigger" gesture is recognized, its movement is tracked to control the display.

However, the above prior art shown in Patent Document 1 has a problem of low reliability. This is because in the above technologies, the background segmentation technology is used to remove the background from the image, so the system sometimes produces the following problems.

(1) Before the system starts working, the user cannot stay in front of the camera. In fact, no object can stay in place except the background. Otherwise, large noisy blocks will appear.

(2) Lighting should remain constant. Once the lighting changes, loud noise will follow. Therefore, in general, this system is not suitable for outdoor scenarios.

(3) The camera must be fixed throughout the process. Furthermore, all camera parameters such as focal length, exposure, and white balance must be fixed.

Patent Document 1: US Patent Application Publication US05594469A

Contents of the invention

The present invention is made in view of the above-mentioned problem of low reliability in the prior art, and its purpose is to provide a simple, direct, and highly reliable non-contact control system and non-contact control method based on static gesture recognition.

The present invention is a non-contact control system based on static gesture recognition, which is characterized in that it has: a camera unit, which shoots a video of the current user; a processing unit, which recognizes the user's gesture from the video of the user captured by the camera unit, and The recognized gesture is converted into a corresponding operation instruction; the target unit executes the operation according to the operation instruction of the processing unit.

In addition, the non-contact control system based on static gesture recognition of the present invention is characterized in that: the above-mentioned processing unit includes a gesture recognition module and a signal conversion module, and the above-mentioned gesture recognition module recognizes the user's gesture from the video of the user captured by the above-mentioned camera unit The above-mentioned signal conversion module converts the recognized gesture into a corresponding operation command for execution by the target unit according to the corresponding relationship between the gesture and the operation command.

In addition, the non-contact control system based on static gesture recognition of the present invention is characterized in that the above-mentioned processing unit further includes a hand detection module and a hand tracking module, and the above-mentioned hand detection module detects the user's movement from the user's video captured by the above-mentioned camera unit. hand, transmitting information including the position and area of the user's hand to the above-mentioned hand tracking module, and the above-mentioned hand tracking module analyzes the video input from the camera unit according to the information including the position and area of the user's hand, Get information about gestures.

In addition, the non-contact control system based on static gesture recognition of the present invention is characterized in that the above-mentioned hand tracking module establishes a tracker for each detected hand, and the above-mentioned tracker is based on information including the position of the hand and the area where it is located. , analyzing the video input from the camera unit to obtain information about the gesture of the hand.

In addition, the non-contact control system based on static gesture recognition of the present invention is characterized in that the above-mentioned gesture recognition module recognizes the user's gesture according to the information about the gesture from the above-mentioned hand tracking module, and transmits the recognized gesture result to the above-mentioned Signal conversion module.

In addition, the non-contact control system based on static gesture recognition of the present invention is characterized in that: the target unit is any one of a TV, a computer, an electronic whiteboard, and an electronic advertising screen.

In addition, the non-contact control system based on static gesture recognition of the present invention is characterized in that the gesture is a gesture representing a number from 1 to 10.

In addition, the present invention is also a non-contact control method based on static gesture recognition, which is characterized in that it includes the following steps: taking a video of the current user; , and a processing step of converting the recognized gesture into a corresponding operation instruction; and an operation step of performing a corresponding operation according to the operation instruction of the above processing step.

In addition, the non-contact control method based on static gesture recognition of the present invention is characterized in that the above-mentioned processing step includes a gesture recognition step and a signal conversion step, and the gesture recognition step recognizes the user's voice from the video of the user captured in the above-mentioned imaging step. For gestures, the signal conversion step converts the gestures recognized in the gesture recognition step into corresponding operation instructions according to the corresponding relationship between gestures and operation instructions.

In addition, the non-contact control method based on static gesture recognition of the present invention is characterized in that the above-mentioned processing step further includes a hand detection step and a hand tracking step, and the above-mentioned hand detection step detects the user from the video of the user captured in the above-mentioned imaging step. hand, obtain the information including the position and the area of the user's hand, and the above-mentioned hand tracking step is based on the information including the position and the area of the user's hand obtained in the above-mentioned hand detection step. Analyze the user's video to obtain information about gestures.

In addition, the non-contact control method based on static gesture recognition of the present invention is characterized in that: in the above-mentioned hand tracking step, each detected hand is tracked separately, and according to the information including the position and area of the hand, The video captured during the camera step is analyzed to obtain information about the gesture of the hand.

In addition, the non-contact control method based on static gesture recognition of the present invention is characterized in that in the gesture recognition step, the user's gesture is recognized based on the information about the gesture obtained in the hand tracking step.

In addition, the non-contact control method based on static gesture recognition of the present invention is characterized in that the above-mentioned operation instruction is an operation instruction executed by any one of a TV, a computer, an electronic whiteboard, and an electronic advertising screen.

In addition, the non-contact control method based on static gesture recognition of the present invention is characterized in that the gesture is a gesture representing a number from 1 to 10.

Invention effect

The above technical solution of the present invention provides a reliable new solution for non-contact control based on static gesture recognition.

First, the non-contact control system and method based on static gesture recognition of the present invention captures the live video of the user through the camera unit, and automatically detects and tracks the user's hand by analyzing the live video. Once a gesture appears in the video, it is recognized in real time. Afterwards, the recognized images are converted into operating instructions for execution by the target unit. Thus, the user is able to control the target unit using a series of static gestures without any direct contact with the target unit.

Here, the traditional background segmentation technology is replaced by the new method of the present invention. Therefore, in the non-contact control system and method based on static gesture recognition of the present invention, the user can move arbitrarily in front of the camera, and the parameters of the camera can also be based on Any setting is required. Moreover, the non-contact control system and method based on static gesture recognition of the present invention can be used both outdoors and indoors. Therefore, the problem of low reliability existing in the prior art such as Patent Document 1 is reasonably solved.

Second, a series of recognition functions can be designed for various users. Currently, many gestures are widely used in the real world. However, they are often difficult to recognize by current gesture recognition systems. Therefore, in the non-contact control system and method based on static gesture recognition of the present invention, a set of new recognition functions are designed for these common recognition gestures. Furthermore, since each recognized gesture is independent of each other, they can be respectively defined as different operation instructions. Therefore, if a gesture is recognized, its corresponding operation command can be executed immediately, thereby realizing a simple, effective and highly reliable control method.

Third, the system performs real-time processing for live video. According to the non-contact control system and method based on static gesture recognition of the present invention, since the hand detection algorithm and the hand tracking algorithm are first used to locate the user's hand, in the subsequent gesture recognition algorithm, it can only be in a small range instead of Recognize user gestures in a large range. Therefore, a lot of time can be saved and a real-time system can be realized.

Fourth, since the non-contact control system and method based on static gesture recognition of the present invention only rely on the user's hand, the user can directly control the target unit without other additional devices.

Description of drawings

FIG. 1 is a block diagram showing the structure of the non-contact control system for static gesture recognition of the present invention.

FIG. 2 is a schematic diagram showing the appearance of the non-contact control system for static gesture recognition of the present invention.

FIG. 3 is a flow chart showing the non-contact control method for static gesture recognition of the present invention.

4 is a flow chart showing the processing performed by the hand detection module of the non-contact control system for static gesture recognition of the present invention.

5 is a flow chart showing the processing performed by the hand tracking module of the non-contact control system for static gesture recognition of the present invention.

FIG. 6 is a flow chart showing the processing performed by the gesture recognition module of the non-contact control system for static gesture recognition of the present invention.

FIG. 7 is a flow chart showing the processing performed by the signal conversion module of the non-contact control system for static gesture recognition of the present invention.

Fig. 8 is a diagram illustrating gestures representing numbers in one embodiment of the present invention.

FIG. 9 shows an embodiment of the present invention to control a TV set with four gestures representing numbers.

FIG. 10 shows an embodiment of the present invention for controlling a television set with multiple gestures representing numbers.

Figure 11 is an example of another series of gestures.

Fig. 12 is a schematic diagram showing the appearance of a conventional gesture device control system.

FIG. 13 is a flowchart of a gesture control method of a gesture machine control system in the prior art.

Detailed ways

Hereinafter, embodiments of the present invention will be described in detail using the drawings.

FIG. 1 is a structural block diagram of a non-contact control system for static gesture recognition in this embodiment, where arrows indicate the flow direction of information flow. FIG. 2 is an external schematic view showing a non-contact control system for static gesture recognition according to an embodiment of the present invention.

As shown in FIG. 1 , the non-contact control system 1 for static gesture recognition in this embodiment mainly includes three parts: a camera unit 2 , a processing unit 3 and a target unit 4 . The camera unit 2 is a device for taking video of the user, and may be a TV camera, a camera of a computer, or the like. The processing unit 3 is used for processing the captured video, for example, recognizing user gestures and converting them into corresponding instructions and the like. The processing unit 3 may be a computer processor or the like. The target unit 4 is a device that executes various operations according to instructions, such as a TV, a computer, a monitor, an electronic whiteboard, an electronic advertising screen, and other devices.

As shown in Figure 2, the above camera unit 2, processing unit 3 and target unit 4 can be set separately (left figure), wherein the camera unit 2 can be placed at a position higher than the display device, or they can be set as three Integrated (right picture), for example, the camera unit 2 can be placed on the top of the device, and the processing unit 3 can be built in.

As the processing unit 3, it includes four modules: a hand detection module 5, a hand tracking module 6, a gesture recognition module 7 and a signal conversion module 8, and their respective functions will be described in detail later.

FIG. 3 is a flowchart showing basic steps of a static gesture recognition based non-contact control method in the static gesture recognition non-contact control system according to the present embodiment.

First, the camera unit 2 shoots live video of the user. The video signal is transmitted to the processing unit 3 . The user's hand is detected in the video by the hand detection module 5 in the processing unit 3 . After the user's hand is detected, the detection result is sent to the hand tracking module 6 . The hand tracking module 6 tracks the user's hand. Furthermore, the detection result and the tracking result are analyzed in the gesture recognition module 7 to recognize the user's gesture. Once a gesture is recognized, the signal conversion module 8 converts the gesture into a corresponding operation instruction. Finally, the processing unit 3 outputs the operation instruction to the target unit 4, and the target unit 4 performs an operation based on the operation instruction.

In this way, by shooting a live video of the user and automatically detecting and tracking the user's hand in the live video, once a gesture appears in the video, it will be recognized in real time. Furthermore, the recognized gesture is converted into a corresponding operation instruction, and the target unit performs an operation according to the operation instruction. Thus, the user is able to use a series of static gestures to control the target unit from a distance without directly touching the target unit for control.

In the following, the non-contact control method based on static gesture recognition in the non-contact control system based on static gesture recognition in this embodiment will be further described in conjunction with the structure of the four modules included in the processing unit 3 .

FIG. 4 is a schematic flowchart of the processing performed by the hand detection module 5 in the processing unit 3 .

After the current live video shot by the camera unit 2 is input into the hand detection module 5, in the hand detection module 5, the input live video is detected according to the trained hand model. The trained hand model is an image taken in advance of the user's hand, and can be designed as required. For example, models of various gestures can be pre-designed only for internal employees of the company, or for random users according to the needs of security or usage restrictions. In addition, a database is established for the upper hand model and stored in the memory.

During the detection process, the hand detection module 5 uses the Adaboost algorithm or other target detection algorithms to analyze whether the user's hand appears in the video against the hand model. In this algorithm, Harr features (edge features, linear features, center features and diagonal features) are mainly used to search for the user's hand through the integral graph. If the user's hand is not found, the hand detection module 5 continues to analyze new videos. If the user's hand is found, the hand detection module 5 records the position, the area and other related information of the user's hand, such as the data (center, width, length) representing the rectangular area of the hand in the image. middle. And, the recorded information is output to the hand tracking module 6 .

In addition, according to needs, sometimes the user's face, eyes, mouth, nose, etc. can be detected to provide further information.

FIG. 5 is a schematic diagram of the processing flow of the hand tracking module 6 in the processing unit 3 .

After the information of the detected user's hand, such as the position of the hand, the area where the hand is related, is input to the hand tracking module 6, the hand tracking module 6 initializes the tracker, that is, for each A detected hand builds a tracker. The tracker analyzes the live video input from the camera unit 2 frame by frame. For each new frame, the tracker searches for the most similar region to the detected hand based on where the hand is located, the region it is in, and the corresponding region in the live video. Once stable tracking is achieved, all information about the user's hand, such as hand position, area, information about gestures, etc., will be obtained. The information of the user's hand obtained above is output to the gesture recognition module 7 for gesture recognition.

In the above processing carried out by the hand tracking module 6, according to the data (center, width, length) representing the rectangular area of the hand in the image, the color histogram can be used to obtain the representation of the hand in the image through the MeanShift algorithm or other target tracking algorithms. The data (center, width, length, direction) of the elliptical area in , thereby obtaining gesture information about the user's hand.

FIG. 6 is a schematic diagram of the processing flow of the gesture recognition module 7 in the processing unit 3 .

The gesture recognition module 7 utilizes the HOG feature to analyze all the information about the user's hand input from the hand tracking module 6 to identify the user's gesture by SVM (Support Vector Machine) algorithm or other target detection algorithms, and the recognized gesture The result (data or signal representing the type of gesture) is output to the signal conversion module 8 .

To recognize the user's gesture by analyzing it through the SVM (Support Vector Machine) algorithm or other target detection algorithms, it is traditionally necessary to search the user's hand in the entire image range to finally realize gesture recognition. However, in the present invention, since the user's hand has been positioned by the hand detection module 5 and the hand tracking module 6, in the most time-consuming gesture recognition process, there is no need to search for the user's hand, thereby saving a lot of time. The time makes it possible for the user to control in real time.

In the present invention, based on the hand model pre-recorded in the memory, the object detection method is used to recognize the hands and gestures in the video. Therefore, compared with the background segmentation technology of the prior art, the user does not need to remove It can move arbitrarily in front of the camera, and the parameters of the camera can also be set arbitrarily according to needs. Moreover, the non-contact control system and method based on static gesture recognition of the present invention can be used both outdoors and indoors. Therefore, the problem of low reliability existing in the prior art such as Patent Document 1 is reasonably solved.

FIG. 7 is a schematic diagram of the processing flow of the signal conversion module 8 in the processing unit 3 .

Since the result (category of the gesture) of the gesture output from the gesture recognition module 7 is various data or signals corresponding to the gesture, it cannot be directly understood and executed by the target module. Therefore, in the signal conversion module 8, the identified The gestures are converted into corresponding operation instruction signals and output to the target unit 4 .

Here, according to different target units, for each different gesture, it is set in advance, and a one-to-one correspondence relationship between each gesture and an operation instruction is established. The above corresponding relationship can be recorded in the memory through the gesture-instruction comparison table.

For example, as described later, in the example where the target unit is a television set, the gesture representing the number "5" is designated as the operation instruction "power on", when the gesture output from the gesture recognition module 7 is the gesture representing the number "5" , the signal conversion module 8 converts the gesture of the number "5" into a corresponding operation instruction "power on", and transmits the operation instruction to a target unit such as a TV to execute a power-on action.

When the target unit is, for example, a computer, the signal conversion module 8 converts the gesture into an operation command signal that the computer can recognize and execute, such as computer commands such as "double click" and "click", forming a non-trivial keyboard and mouse instead of the traditional one. contact control system.

In this way, through the processing of the hand detection module 5, the hand tracking module 6, the gesture recognition module 7 and the signal conversion module 8 in the processing unit 3 above, based on the correspondence between a series of static gestures and operation instructions set in advance, the user can Touch to control the target unit to perform actions.

The operation process of using static gestures to control the TV is specifically described below by taking the operation of the TV as an example.

As a simple solution to gestures, gestures representing numbers as shown in Fig. 8 are widely used. Moreover, each of these gestures is independent of each other. Therefore, in this embodiment, they are respectively defined as different operation instructions, and the gestures of the left and right hands are distinguished. In this way, a set of 20 different operation instructions is formed.

For example, set the left-hand gesture representing "5" as the "power on" command; the left-hand gesture representing "8" as the "next" command; the left-hand gesture representing "6" as the "previous" command; The left-hand gesture is the "close" command; the left-hand gesture representing "9" is the "call" command.

FIG. 9 shows a schematic diagram of a simple TV controller made using four gestures.

When the user stood in front of the TV set (target unit 4 ), the TV camera as the camera unit 2 took a live shot of it, and the user's live video was input to the hand detection module 5 in the processing unit 3 . When the user starts to move his hand in front of the TV and makes a left-hand gesture representing "5", the user's hand will be detected by the hand detection module 5 in real time. Then, in the hand tracking module 6, the user's hand is tracked, thereby obtaining information about the user's hand position, area, gesture. Afterwards, in the gesture recognition module 7, the obtained hand information is analyzed, and the user's gesture is identified as a left-hand gesture representing "5". After the gesture is recognized, in the signal conversion module 8, the left-hand gesture representing "5" is converted into the corresponding operation instruction "start up", so that the TV is turned on.

Subsequently, when the user makes a left-hand gesture representing "8", according to the same process as above, the left-hand gesture representing "8" is converted into a corresponding instruction "next", and the TV performs the operation of switching to the next channel. Thereafter, when the user makes a left-hand gesture representing "6", the left-hand gesture representing "6" is converted into a corresponding instruction "previous", and the TV performs the operation of switching to the previous channel. Finally, when the user makes a left-hand gesture representing "10", the left-hand gesture representing "10" is converted into a corresponding instruction "turn off", and the TV performs an operation of turning off the TV.

The above 4 gestures can be used to make a simple TV controller. In fact, more complex control can be realized by adding other gestures.

Fig. 10 shows a schematic diagram of a relatively complex TV controller made using five gestures,

In FIG. 10, a left-hand gesture representing "9" is added as an operation instruction of "call" to call all function menus. In general, "direct" mode is set as the default mode. In this mode, users can switch channels directly through right hand gestures. If the user wants more complex operations, the user can select the "call" command to call out all function menus. In this way, all functions can be freely selected.

In Figure 10, complex control functions are implemented using one hand (left hand). In fact, many other functions can be easily performed with two hands.

Gestures representing numbers are enumerated above as a simple solution to gestures. However, it is not limited to the above-mentioned 20 kinds of gestures for representing numbers, and a series of gestures respectively shown by left and right hands as shown in FIG. 11 can also be listed. In fact, as long as each of these gestures is independent and easily distinguishable from each other, they can be defined as different operation instructions to be set according to control needs. However, since the above gestures representing numbers have been widely used, the above gestures representing numbers are preferably used as operation instructions.

The above uses the operation of the TV as an example to illustrate the operation process of using static gestures to control the TV. However, in addition to the TV, as the target unit, it can also be a computer, a monitor, an electronic whiteboard, an electronic advertising screen and other devices. When controlling different target units, the signal conversion module converts gesture signals into corresponding operation instruction signals that the device can recognize and execute.

All of the above-described embodiments are illustrative and not restrictive. The scope of the present invention is shown not by the above-described description but by the claims, and all changes within the meaning and range equivalent to the scope of the claims are included.

Claims (14)

1. Touchless control system based on static gesture identification is characterized in that having:

Image unit is taken the current video of user;

Processing unit, identification user's gesture from the user's that takes by described image unit video, and the gesture of being discerned is converted to corresponding operational order; And

Object element according to the operational order of described processing unit, is carried out operation accordingly.

2. the Touchless control system based on static gesture identification according to claim 1 is characterized in that:

Described processing unit comprises gesture identification module and signal conversion module,

Described gesture identification module identification user's from the user's that takes by described image unit video gesture,

Described signal conversion module is converted to the gesture of being discerned the corresponding operational order of carrying out for object element according to the corresponding relation of gesture and operational order.

3. the Touchless control system based on static gesture identification according to claim 2 is characterized in that:

Described processing unit also comprises hand detection module and hand tracking module,

Described hand detection module detects user's hand from the user's that taken by described image unit video, will comprise that user's the position of hand and the information in zone of living in send described hand tracking module to,

Described hand tracking module to analyzing from the video of described image unit input, obtains the information about gesture according to the position of the hand that comprises the user and the information in zone of living in.

4. the Touchless control system based on static gesture identification according to claim 3 is characterized in that:

Described hand tracking module is set up a tracker to each hand that is detected, and described tracker to analyzing from the video of described image unit input, obtains the information about the gesture of this hand according to the information that comprises the position and the zone of living in of this hand.

5. according to claim 3 or 4 described Touchless control systems, it is characterized in that based on static gesture identification:

Described gesture identification module is discerned user's gesture according to the information about gesture from described hand tracking module, and the gesture result who is discerned is sent to described signal conversion module.

6. the Touchless control system based on static gesture identification according to claim 1 is characterized in that:

Described object element is any in televisor, computing machine, electronic whiteboard, the electric advertisement screen.

7. the Touchless control system based on static gesture identification according to claim 1 is characterized in that:

Described gesture is the gesture of the numeral of expression 1～10.

8. the non-contact control method based on static gesture identification is characterized in that, may further comprise the steps:

Take the shooting step of the current video of user;

Identification user's gesture from the user's that described shooting step, takes video, and the gesture of being discerned is converted to the treatment step of corresponding operational order; And

According to the operational order of described treatment step, carry out the operation steps of corresponding operating.

9. the non-contact control method based on static gesture identification according to claim 8 is characterized in that:

The treating step comprises gesture identification step and conversion of signals step,

In described gesture identification step, identification user's gesture from the user's that described shooting step, takes video,

In described conversion of signals step,, will be converted to corresponding operational order in the gesture of described gesture identification step identification according to the corresponding relation of gesture and operational order.

10. the non-contact control method based on static gesture identification according to claim 9 is characterized in that:

Described treatment step comprises that also hand detects step and hand tracking step,

Detect in the step at described hand, from the user's that described shooting step, takes video, detect user's hand, obtain comprising user's the position of hand and the information in zone of living in,

In described hand tracking step, according to the position of detecting the hand that comprises the user that step obtains at described hand and the information in zone of living in, analyze in the video to the user that in described shooting step, takes, obtain information about gesture.

11. the non-contact control method based on static gesture identification according to claim 10 is characterized in that:

In described hand tracking step, each hand that is detected is followed the tracks of respectively, according to the information in position that comprises hand and zone of living in, the video of taking in the shooting step is analyzed, obtain information about the gesture of this hand.

12. the non-contact control method based on static gesture identification according to claim 10 is characterized in that:

In described gesture identification step, according to the information that in described hand tracking step, obtains about gesture, identification user's gesture.

13. the non-contact control method based on static gesture identification according to claim 8 is characterized in that:

Described operational order is the operational order of any execution in televisor, computing machine, electronic whiteboard, the electric advertisement screen.

14. the non-contact control method based on static gesture identification according to claim 7 is characterized in that:

Described gesture is the gesture of the numeral of expression 1～10.

Images