CN102053702A - Dynamic gesture control system and method - Google Patents

Abstract

The invention discloses a dynamic gesture control system and method. The system includes a computer and a camera, and the camera is directly connected to the computer. The method includes a static gesture recognition stage, a dynamic gesture tracking stage and a dynamic gesture recognition stage. The present invention (1) is non-contact, the user does not need any auxiliary equipment, and the tracking and recognition of the user's dynamic gesture can be realized only through the camera; Accurate identification; (3) Low cost. At present, the purchase and maintenance of various data gloves are expensive, while ordinary cameras are cheap and maintenance-free, which greatly reduces the purchase and maintenance costs of gesture control systems.

Description

Dynamic Gesture Control System and Method

technical field

The dynamic gesture control system and method described in the present invention belong to a video-based data processing system and method, and are suitable for human-computer interaction in virtual reality, large-scale interactive projection systems, and display systems, especially human-computer interaction in volumetric three-dimensional display systems. Interactive link.

Background technique

Interactive technology is a key technology of virtual reality system, and also a key technology of new display systems such as volumetric three-dimensional display. Interaction technology realizes the interaction between man and machine, the interaction between the real world and the virtual world. The premise of realizing interaction through dynamic gesture control is to obtain information such as the position of the user's hand in real time, and to recognize the meaning of the gesture trajectory. The former is gesture tracking and the latter is gesture recognition.

At present, gesture tracking is mainly divided into data glove-based gesture tracking and video-based gesture tracking.

See "Shi Jiaoying, "Basics and Practical Algorithms of Virtual Reality", Science Press, 2002", which systematically expounds the relevant content of the data glove: the data glove converts various gestures of fingers and palms into digital signals to send For the computer, it is used for recognition and execution, and then realizes human-computer interaction. The standard configuration is that there are two sensors on each finger, controlling two fiber optic rings or other measuring elements mounted on the back of the finger, which are used to measure the bending angle of the major joints of the finger. The data glove can also provide sensors for measuring the closing/opening and up/down angle of the thumb as an option.

Data gloves can accurately obtain finger position and posture information, but its shortcomings are also obvious: (1) The structure is complex and inconvenient to wear, which affects the natural and intuitive interaction between humans and machines; (2) It is expensive and has high precision The precision sensor components increase the manufacturing cost.

Video-based gesture tracking is mainly achieved through cameras tracking special color markers worn on fingers or tracking handheld auxiliary devices such as SONY's PlayStation Move. See "T. Grossman, D. Wigdor, R. Balakrishnan, Multi-Finger Gestural Interaction with 3D Volumetric Displays. Proceedings of the 17th Annual ACM Symposium on User interface Software and Technology. UIST '04 Ober US to Santa Fe, N M -27, 2004: 61-70", By wearing special colored markers on the fingers, the true 3D display system Perspecta allows users to realize video-based gesture control on a hemispherical display housing. Although the above video-based gesture tracking methods improve the tracking accuracy through auxiliary equipment, they also affect the natural and intuitive interaction between humans and machines.

Contents of the invention

The purpose of the present invention is to provide a computer vision-based, non-contact, low-cost, portable and practical dynamic gesture control system and method to realize the dynamic gesture control function. The technical problems to be solved include: fast start of gesture tracking, fast tracking of dynamic gestures, and accurate recognition of dynamic gestures.

In order to achieve the above object, the present invention adopts the following technical solutions:

The dynamic gesture control system of the present invention includes a computer and a camera, and the camera is directly connected with the computer.

A control method for a dynamic gesture control system, including a static gesture recognition stage, a dynamic gesture tracking stage and a dynamic gesture recognition stage;

(1) Static gesture detection stage

Proceed as follows to detect a fully open static gesture:

1) For the binary image of the frame difference between the current frame and the previous frame, extract the external contour of the static hand;

2) Calculate the convex hull of the external contour according to the Sklansky algorithm, and the vertex of the convex hull is the fingertip point;

3) Calculate the convex defect and its depth of the outer contour, that is, the distance from each defect point to the convex hull; set the maximum depth to , taking the depth beyond The defect point is the gap point between fingers;

4) If the number of fingertips is 5 and the number of gaps between fingers is 4, the static gesture detection is successful; the set of raised defect points; the determined rectangular area is the initial tracking area, determined by all fingertip points and raised defect points The rectangular area is the hand boundary area, establish the hand skin color distribution model and exit the gesture detection; otherwise read the new input frame and enter step 1);

(2) Dynamic gesture tracking stage

First update the position of each feature point in the tracking area according to the optical flow, and the position of each feature point can neither be far from the center nor overlap each other;

(3) Dynamic gesture recognition stage

a) Quantify the trajectory contained in the input dynamic gesture I according to the eight-direction Freeman chain code, and obtain the observation value sequence O={ b ₁ , b ₂ , … , b _m };

是初始概率向量，

为状态数；

是状态概率转移矩阵；

是观测值概率矩阵，

为观测符号数；l是手势HMM库中的模型数，手势编号从1到l；b) Calculate each model in the dynamic gesture HMM library ( ) produces the probability of the sequence of observations O: ; Using the discrete HMM model, namely DHMM, denoted as ,in:

is the initial probability vector,

is the state number;

is the state probability transition matrix;

is the observation probability matrix,

is the number of observation symbols; l is the number of models in the gesture HMM library, and the gesture numbers are from 1 to l ;

，如果

，

为一确定阈值, j即为识别出的手势编号。c) take medium maximum

,if

,

is a certain threshold, and j is the recognized gesture number.

Among them, step (2) in the dynamic gesture tracking stage first adjusts the feature points that are beyond a certain range from the center position to be close to the center position and above the skin color pixel point according to the skin color distribution model, and then any two feature points that are too close to each other One of them adjusts the distance beyond the threshold so that all feature points do not overlap with each other, and then updates the centers of all feature points.

、

，计算其连线

与水平线的夹角

，根据夹角

按八方向Freeman链码对进行编码，编码值为

。Among them, the step (3) dynamic gesture tracking stage only performs feature extraction on the trajectory curve of the center position of the hand: for the two adjacent sampling points on the gesture trajectory line, the center position

,

, calculate its connection

angle with horizontal

, according to the angle

According to the eight directions Freeman chain code pair To encode, the encoded value is

.

_Wherein , only when the distance between the current sampling point and the previous valid sampling point is greater than the threshold DT , the current sampling point is set as a valid sampling point.

The beneficial effects of the present invention are:

Compared with auxiliary equipment such as data gloves, special color markings, and handheld game controllers, it has the following obvious advantages: (1) Non-contact, users do not need any auxiliary equipment, and the user's dynamic gestures can be tracked and recognized only through the camera; (2) There is no need for calibration, which simplifies the use of the system, and the gesture control starts quickly, tracks quickly, and recognizes accurately; (3) The cost is low. At present, the purchase and maintenance of various data gloves are expensive, while ordinary cameras are cheap and maintenance-free, which greatly reduces the purchase and maintenance costs of gesture control systems.

Description of drawings

Figure 1 is a block diagram of the dynamic gesture control system.

Figure 2 is a schematic diagram of static gesture detection.

1- fingertip

2- Between the fingers

3- Wrist pit

Fig. 3 is a schematic diagram of feature point position adjustment in the dynamic gesture tracking phase.

1- center point

2- Feature points.

Detailed ways

First, the system configuration block diagram shown in Figure 1 is used for system layout, and then the user gesture motion information is collected through the camera, and the static gesture recognition, dynamic gesture tracking and dynamic gesture recognition are completed through the computer.

(1) Static gesture detection stage

Proceed as follows to detect a fully open static gesture:

1) For the binary image of the frame difference between the current frame and the previous frame, extract its outer contour.

2) Calculate the convex hull of the external contour according to the Sklansky algorithm, and the apex of the convex hull is the fingertip point (remove the convex hull apex located at the wrist, as shown in Figure 2, the third type of point), as shown in Figure 2, the first type of point.

，取深度超过

的缺陷点为指间缝隙点，如图2中第2类点。3) Calculate the convex defects of the outer contour and their depths, that is, the distance from each defect point to the convex hull. Note that the maximum depth value is

, taking the depth beyond

The defect point is the inter-finger gap point, such as the second type of point in Figure 2.

4) If the number of fingertips is 5 and the number of gaps between fingers is 4, the static gesture detection is successful. The rectangular area determined by the raised defect point set (as shown in Figure 2, including the first and third types of points) is the initial tracking area, and the rectangular area determined by all fingertip points and raised defect points is the hand boundary area , establish a hand skin color distribution model and exit gesture detection; otherwise, read a new input frame and enter step 1).

The above static gesture detection algorithm makes full use of the convex hull vertices and convex defect points in the hand area, has direction invariance, and the Sklansky algorithm has low complexity, thus ensuring the accuracy and speed of gesture control startup.

(2) Dynamic gesture tracking stage

In Figure 3, the position of each feature point (the second type of point in Figure 3) in the tracking area is firstly updated according to the optical flow. In addition, according to the theory of "group" (Flock), the position of each feature point can neither be far from the center position (point 1 in Figure 3), nor can it overlap with each other. Therefore, first adjust the feature points that are beyond a certain range from the center position to be close to the center position and above the skin color pixel point according to the skin color distribution model, and then adjust one of any two feature points that are too close to the threshold. so that all feature points do not overlap with each other, and then update the centers of all feature points. Such processing makes the change of the central position of the feature point more stable and smooth, and improves the reliability of the tracking process.

The above tracking algorithm combines the advantages of the optical flow method and the feature point group algorithm, making the non-contact gesture tracking process accurate and fast.

(2) Dynamic gesture recognition stage

、

，计算其连线

与水平线的夹角，根据

，可按八方向Freeman链码对

进行编码，码值为

。此外，为避免采样点过于密集、编码过长，只有在当前采样点与前一有效采样点的间距大于阈值D _T时，当前采样点才为被设置为有效采样点。In order to improve the real-time performance of gesture recognition, feature extraction is only performed on the trajectory curve of the center position of the hand. For two adjacent sampling points (center position) on the gesture trajectory line

,

, calculate its connection

angle with horizontal ,according to

, according to the eight-direction Freeman chain code pair

To encode, the code value is

. In addition, in order to avoid too dense sampling points and too long encoding, only when _{the distance between the current sampling point and the previous valid sampling point is greater than the threshold DT} , the current sampling point is set as a valid sampling point.

The specific process of dynamic gesture recognition is:

1) Quantify the trajectory contained in the input dynamic gesture I according to the eight-direction Freeman chain code, and obtain its observation value sequence O={ b ₁ , b ₂ , … , b _m }.

。（各个动态手势HMM模型按Baum-Welch算法训练得到，训练样本同样按1）方式编码）2) Calculate each model in the dynamic gesture HMM library ( ) produces the probability of O:

. (Each dynamic gesture HMM model is trained according to the Baum-Welch algorithm, and the training samples are also coded according to 1) method)

中最大值，如果

（

为一确定阈值）, j即为识别出的手势编号。3) take

medium maximum ,if

(

is a certain threshold), j is the recognized gesture number.

For the above dynamic gesture recognition algorithm, the eight-direction Freeman chain code is selected instead of the sixteen-direction chain code to encode the dynamic gesture trajectory, which is a compromise between the real-time performance of the system and the encoding accuracy.

Claims (5)

1. A dynamic gesture control system is characterized in that it comprises a computer and a camera, and the camera is directly connected with the computer. 2. A control method of a dynamic gesture control system as claimed in claim 1, characterized in that comprising a static gesture recognition stage, a dynamic gesture tracking stage and a dynamic gesture recognition stage; The static gesture detection phase detects fully open static gestures as follows: 1) For the binary image of the frame difference between the current frame and the previous frame, extract the external contour of the static hand; 2) Calculate the convex hull of the external contour according to the Sklansky algorithm, and the vertex of the convex hull is the fingertip point; 3) Calculate the convex defect and its depth of the outer contour, that is, the distance from each defect point to the convex hull; set the maximum depth to

, taking the depth beyond

The defect point is the gap point between fingers; 4) If the number of fingertips is 5 and the number of gaps between fingers is 4, the static gesture detection is successful; the set of raised defect points; the determined rectangular area is the initial tracking area, determined by all fingertip points and raised defect points The rectangular area is the hand boundary area, establish the hand skin color distribution model and exit the gesture detection; otherwise read the new input frame and enter step 1); (2) Dynamic gesture tracking stage First update the position of each feature point in the tracking area according to the optical flow, and the position of each feature point can neither be far from the center nor overlap each other; (3) Dynamic gesture recognition stage a) Quantify the trajectory contained in the input dynamic gesture I according to the eight-direction Freeman chain code, and obtain the observation value sequence O={ b ₁ , b ₂ , … , b _m }; b) Calculate each model in the dynamic gesture HMM library (

) produces the probability of the sequence of observations O:

; Using the discrete HMM model, namely DHMM, denoted as

,in:

is the initial probability vector,

is the state number;

is the state probability transition matrix;

is the observation probability matrix,

is the number of observation symbols; l is the number of models in the gesture HMM library, and the gesture numbers are from 1 to l ; c) take

medium maximum

,if

,

is a certain threshold, and j is the recognized gesture number. 3. The control method of the dynamic gesture control system according to claim 2, characterized in that in the step (2) of the dynamic gesture tracking stage, the feature points that exceed a certain range from the center position are adjusted to be close to the center position according to the skin color distribution model. Located on the skin color pixel, then adjust one of the two feature points that are too close to the distance beyond the threshold, so that all feature points will not overlap each other, and then update the centers of all feature points. 4. The control method of the dynamic gesture control system according to claim 2, characterized in that step (3) in the dynamic gesture tracking stage only performs feature extraction on the trajectory curve of the center position of the hand: for two adjacent gesture trajectory lines The sampling point is the center position

, , calculate its connection

angle with horizontal , according to the angle

According to the eight directions Freeman chain code pair

To encode, the encoded value is . 5. The control method of the dynamic _{gesture control system according to claim 3, wherein only when the distance between the current sampling point and the previous effective sampling point is greater than the threshold value DT} , the current sampling point is set as an effective sampling point .

Images