CN107813310A - One kind is based on the more gesture robot control methods of binocular vision - Google Patents

Abstract

The present invention relates to the field of robot control methods, in particular to a multi-gesture robot control method based on binocular vision. The teaching method has the disadvantages of large amount of calculation and high precision requirements for the accuracy of the robot model and the determination of the coordinate system. A multi-gesture robot control method based on binocular vision is proposed, including: setting binocular cameras; Rectangular box; the classifier is trained using the training sample set. The classifier detects the target; then tracks the target, and fuses the tracking result with the detection result; calculates the offset distance of the target center point moving from the initial point to the target point, and outputs a speed control command to make the robot perform translational motion; Extract the feature points in the target frame, and solve the rotation matrix corresponding to the feature points. The invention is suitable for the control method of the painting robot.

Description

A multi-gesture robot control method based on binocular vision

technical field

The invention relates to a robot control method, in particular to a binocular vision-based multi-gesture robot control method.

Background technique

The application of industrial robots mainly uses the operator to use the teaching pendant to manually control the joint movement of the robot, so that the robot moves to a predetermined position, and at the same time record the position and transmit it to the robot controller. After that, the robot can follow the instructions Automatically repeat the task, but the current teaching method has the problems of cumbersome process and low efficiency

At present, there are mainly two methods in the teaching method of industrial robots-manual teaching method and offline teaching method. Manual teaching refers to manually guiding the end effector of the robot, or guiding the mechanical simulation device by manual operation, or using the teaching box to make the robot complete the expected action, because the programming of this type of robot is realized through real-time online teaching program. Realization, and the robot itself operates by memory, so it can be reproduced continuously. The off-line teaching method refers to collecting the model first, simulating and programming on the computer, and performing trajectory planning to automatically generate motion trajectories.

Nowadays, teaching boxes are mostly used in the field of industrial robots. This control method is inefficient and not intuitive. There is a vision-based robot control method that requires the operator to wear gloves of a specific color. There is only one control mode. When fine-tuning is required, the position and attitude instructions will interfere with each other, and the control space and the robot workspace are not easy to unify, resulting in inconsistent operation. Convenience ^[1] . The recognition of bare hands is mainly based on color space segmentation, which is greatly affected by illumination and background color. However, the offline teaching method has a large amount of calculation, complex algorithms, and inconvenient calculation of irregular edges, and has high precision requirements for the accuracy of the robot model and the determination of the robot tool coordinate system.

Contents of the invention

The present invention solves the inconvenient operation of the existing vision-based robot control method, the recognition of the opponent is greatly affected by the illumination and the background color, and the off-line teaching method has a large amount of calculation, which has a high impact on the accuracy of the robot model and the determination of the robot coordinate system. Due to the shortcomings of precision requirements, a control method for multi-gesture robots based on binocular vision is proposed, including:

Step 1. Set up the binocular camera, and perform calibration and correction.

Step 2: The operator performs a gesture demonstration within the field of view of the binocular camera, and manually selects a rectangular frame containing the gesture from the video captured by the left camera of the binocular camera, and adds it to the training sample set.

Step 3: Use the training sample set to train the nearest neighbor classifier and the Bayesian classifier.

Step 4, the operator appears in the field of view of the binocular camera according to the gesture in step 2; the processor uses the cascade variance classifier based on the image of the left camera, the Bayesian classifier based on random forest, and the nearest neighbor classifier to detect target; then track the target, fuse the tracking result with the detection result, and update the samples in the gesture template at the same time, after successful tracking in the left camera image, detect and track on the epipolar line of the right camera image, if the If the view is simultaneously tracked successfully, the target rectangle will be output.

Step 5. Track the center point of the target rectangular frame; calculate the offset distance of the center point from the initial point to the target point, and output a speed control command to make the robot perform translational motion.

Step 6: Extract the feature points used to describe the outline of the gesture in the target rectangular frame, and solve the rotation matrix corresponding to the feature points.

Preferably, step one specifically includes:

Step 11. Set the distance between the left camera and the right camera in the binocular camera to be 20cm, and place them horizontally.

Step 12: Calibrate the binocular camera using the Zhang Zhengyou calibration method, and eliminate distortion and line alignment for the views of the left and right cameras, so that the imaging origin coordinates of the left and right camera views are consistent, the optical axes are parallel, and the imaging planes are coplanar. Pole line alignment.

Preferably, step two specifically includes:

Step 21, the operator performs a gesture demonstration within the field of view of the binocular camera.

Step 22: Manually select a rectangular frame containing gestures from the video captured by the left camera of the binocular camera.

Step 23: Scale, rotate, and affine the image blocks in the selected rectangular frame, and normalize the scaled, rotated, and affine images into image blocks of the same size to form a positive sample set; and select a predetermined number The image blocks whose distance from the selected image block in the original image is greater than a predetermined threshold constitute a negative sample set; the positive sample set and the negative sample set together constitute a training sample set.

Preferably, step three specifically includes: calculating the foreground class posterior probability of the Bayesian classifier by the following formula:

Among them, y ₁ represents the foreground, when y ₁ =0, it means that there is no target in the image, and when y ₁ =1, it means that the image contains the target; x _i represents the i-th feature of the image; each feature of the image is arbitrarily selected The relationship between the gray value of two points, the relationship between the gray value is represented by 0 or 1.

Preferably, step four specifically includes: in step three,

The number of Bayesian classifiers is 10; the number of foreground samples corresponding to feature x _i is #p, the number of background samples is #n, and the total number of samples is #m, then:

Calculate p(y ₁ | _xi ) for each Bayesian classifier, and average the results. If the average value is greater than the preset threshold, it is considered that there is a target in the image.

Preferably, in step 3, the nearest neighbor classifier is used to calculate the similarity between two image blocks, and the calculation formula is:

In the formula, μ ₁ , μ ₂ , σ ₁ , σ ₂ represent the average value and standard deviation of images P1 and P2; the more similar the two images are, the closer the result is to 1; the distance between two images is defined as: Then when the distance between the two images is smaller than a predetermined threshold, the image slice is considered to contain the target.

Preferably, step four is specifically:

Step 41. The operator appears in the field of view of the binocular camera according to the gesture in step 2.

Step 42: The operator manually selects the initial rectangular frame.

Step 42: The processor generates a sliding rectangular frame, uses the cascade variance classifier to filter out the rectangular frames that do not meet the variance threshold condition, and then filters through the Bayesian classifier to obtain image blocks that may contain foreground, and then passes through the nearest neighbor classifier Computes the similarity between the sliding rectangle and the manually selected initial rectangle.

Step 43: Select the rectangular frame with the highest overlapping degree as the sample rectangular frame, and calculate the Shi-Tomasi corner points in the sample rectangular frame as feature points.

Step 44: Calculate the forward prediction error, backward prediction error and similarity in the sample rectangle, and filter out the feature points that are smaller than the average value of the forward prediction error and backward prediction error and greater than the preset similarity threshold.

Step 4 and 5: Calculate the average displacement between the selected feature points in the current frame and the corresponding feature points in the previous frame to obtain the position of the target frame in the current frame, and according to the ratio of the Euclidean distance of the feature points in the previous frame to the current frame , get the size of the target box in the current frame.

Step 46: Normalize the target frame obtained in step 45, and calculate the similarity between the normalized target frame and all images in the positive sample set. If there is a similarity greater than the specified threshold, track is valid, and the obtained target frame is added to the sample set, otherwise the tracking is considered invalid and discarded.

Preferably, step five specifically includes:

Step 51. Use the center point of the target frame obtained in step 46 as the center point of the gesture, and use the principle parallax distance measurement method of stereo vision to calculate the spatial coordinate value of the gesture center, specifically:

In the formula, X, Y, Z are the positions of the gesture center point in space, u ₁ is the x coordinate of the marker ball in the left camera image coordinate system, u ₀ is the x origin of the left camera image coordinate system, and u ₂ is the marker ball The x coordinate in the right camera image coordinate system, d is the translation distance between the two cameras, v ₁ is the y coordinate of the marker ball in the left camera image coordinate system, v ₀ is the y origin of the left camera image coordinate system, f is the focal length of the camera.

Step 52. The operator sets the origin at any position through the calibration button; when the processor detects that the center point of the gesture leaves the sphere with the preset control threshold as the radius, it outputs a speed control command. The calculation formula is:

V=kd

Among them, V is the output speed control command, k is the control coefficient, and d is the distance from the initial position of the gesture center; the speed control command is used to control the translational movement of the end of the robot.

Preferably, step six specifically includes:

Step 61: Obtain the outline of the gesture in the target frame by combining the method based on skin color detection and background difference method in step 46, and then obtain the index finger, middle finger and ring finger and the depression of the index finger and middle finger through the convex hull detection and convex hull defect detection algorithm There are altogether 5 feature points at the depression of the middle finger and ring finger; and the spatial coordinates of these 5 feature points are obtained through the formula in step 51.

Step 62. Define the coordinate system on the palm, take the root of the middle finger as the origin, define the direction pointing to the fingertip of the middle finger as the positive direction of the y-axis, define the line parallel to the two depressions as the x-axis, and define the direction pointing to the little finger as the x-axis Positive direction.

Step 63: Solve the rotation matrix by using 5 feature points according to Carley's theorem.

Step 64: Convert the rotation matrix into pitch-yaw-roll Euler angles, obtain the relative rotation angle during the conversion of the gesture from the current posture to the original state, and output the Euler angular velocity command to control the posture change of the robot according to the relative rotation angle.

Preferably, step 63 specifically includes:

Step 631, establishing any rotation matrix R that does not contain eigenvalue-1 and an antisymmetric matrix S _b has the following relationship:

R＝(IS _b ) ^-1 (I+S _b )

S _b ＝(R+I) ^-1 (RI)

In the formula, I is the identity matrix, b=(b ₁ , b ₂ , b ₃ ) ^T is the Carley vector; where b ₁ , b ₂ , b ₃ are the first, second, and third components in the Carley vector; and

Step 632: Let p _i be the spatial coordinate value of the feature point i, and q _i be the coordinate value of the feature point i in the palm coordinate system. Then solve the rotation matrix equation as:

Do identity transformation on the above formula to get:

In the formula:

v _i =p _i -q _i

u _i =p _i +q _i

S _ui is the antisymmetric matrix corresponding to u _i ;

Available equations:

Ab=c

In the formula:

Step 633: Solve the equation Ab=c to obtain the Carley vector, and then calculate the rotation matrix R.

The beneficial effects of the present invention are as follows: 1. The position and attitude instructions will not interfere with each other, and the control space and the working space of the robot are easily unified, which makes the operation very simple; 2. The identification of the opponent uses feature points and rotation matrices, which are greatly affected by the light. Small; 3. The calculation amount of the offline teaching part is small, and the algorithm is not complicated; 4. The requirements for the accuracy of the robot model and the determination of the robot coordinate system are not high.

Description of drawings

Fig. 1 is a schematic diagram of a gesture robot control device of the present invention;

Fig. 2 is a schematic diagram of control gestures, wherein Fig. 2(a) is a schematic diagram of gestures in attitude control mode; Fig. 2(b) is a schematic diagram of gestures in position control mode;

FIG. 3 is a flow chart of the control method for a multi-gesture robot based on binocular vision of the present invention.

Detailed ways

The multi-gesture robot control method based on binocular vision of the present invention is realized based on the device shown in Figure 1, wherein 101 is a host computer, which includes a processor for computing and controlling the robot; 102 is a painting robot; 103 is The left camera of the binocular camera may also be called the left camera, and 104 is the right camera of the binocular camera, which may also be called the right camera. 105 is the operator's hand. A group of binocular cameras are placed in parallel as the gesture detection part, and the control signals are obtained through computer processing and sent to the robot. The operator only needs to keep the hand in the field of view of both cameras.

In order to prevent the coupling of position control and attitude control, two gestures can be recorded and learned in advance, one is position control and the other is attitude control. The present invention defines gesture control when the palm is open, as shown in Figure 2(a); position control when the thumb, index finger and middle finger are pinched together, as shown in Figure 2(b).

Figure 3 is a multi-gesture robot control method based on binocular vision, specifically including:

Step 1. Set up the binocular camera, and perform calibration and correction.

Step 2: The operator performs a gesture demonstration within the field of view of the binocular camera, and manually selects a rectangular frame containing the gesture from the video captured by the left camera of the binocular camera, and adds it to the training sample set.

Step 3: Use the training sample set to train the nearest neighbor classifier and the Bayesian classifier.

Step 4, the operator appears in the field of view of the binocular camera according to the gesture in step 2; the processor uses the cascade variance classifier based on the image of the left camera, the Bayesian classifier based on random forest, and the nearest neighbor classifier to detect target; then track the target, fuse the tracking result with the detection result, and update the samples in the gesture template at the same time, after successful tracking in the left camera image, detect and track on the epipolar line of the right camera image, if the If the view is simultaneously tracked successfully, the target rectangle will be output.

Step 5. Track the center point of the target rectangular frame; calculate the offset distance of the center point from the initial point to the target point, and output a speed control command to make the robot perform translational motion.

Step 6: Extract the feature points used to describe the outline of the gesture in the target rectangular frame, and solve the rotation matrix corresponding to the feature points.

Step 1 can specifically be: set the distance between the left camera and the right camera in the binocular camera to be 20cm, and place them as parallel as possible. Use the Zhang Zhengyou calibration method to obtain the internal and external parameters of the camera, and then perform stereo correction to eliminate distortion and line alignment for the left and right views, so that the imaging origin coordinates of the left and right views are consistent, the optical axes of the two cameras are parallel, and the left and right imaging planes are coplanar. Align the epipolar lines.

Step 2 and Step 3 are to establish the sample set and the training process, which can be specifically: the training process trains the Bayesian classifier and the nearest neighbor classifier, zooms, rotates, and affines the image block selected by the mouse in the initial frame, Finally, the image blocks normalized to the same size are used as positive sample sets, and several image blocks far away from the above image blocks are selected as negative sample sets, and the nearest neighbor classifier is trained with this sample set. According to this sample set, the positive and negative sample sets of 2bitBP features are extracted to train the Bayesian classifier, and the Bayesian formula for calculating the posterior probability is obtained. In the next step of the tracking mode, the sample set is updated online, and the above two classifiers are iteratively trained.

The device uses cascaded classifiers to detect gestures, including variance classifiers, Bayesian classifiers based on random forests and nearest neighbor classifiers. The variance classifier refers to finding the variance of the image of the sliding rectangular frame to be detected, because the variance of the tracking target area is generally larger than the variance of the background area, so most of the scanning rectangular frames can be filtered out through the variance filter. The random forest contains 10 Bayesian classifiers. The feature selected by the Bayesian classifier is the 2bitBP feature. The 2bitBP feature is the relationship between the gray value of any two points, and the values are only 0 and 1. The class to which the image belongs is represented by y _i (i=1,2). The detection problem in the subject can be regarded as a classification problem, and there are only two classes, namely the foreground class and the background class. Let y ₁ =0 represent the There is no target, and y ₁ =1 indicates that the image contains a target. Use x _i (i=1,2,3,...,2 ¹³ ) to represent the feature set of the image, that is, the above-mentioned 2bitBP features. Then the Bayesian classifier obtains the posterior probability of the foreground class as:

In the sample set, let the number of foreground samples corresponding to x _i be #p, the number of corresponding background samples be #n, and the total number of samples be #m. but

There are 10 posterior probabilities in the entire random forest, and the 10 posterior probabilities are averaged. If it is greater than the threshold, the image is considered to contain a foreground target.

The nearest neighbor classifier has two functions, one is to use the NCC algorithm to sequentially match the similarity between each image block and the online model, and the other is to update the positive sample space of the online model. In order to compare the image blocks P ₁ and P ₂ , the NCC algorithm is used to characterize the similarity of the image blocks.

In the formula, μ ₁ , μ ₂ , σ ₁ , σ ₂ represent the average value and standard deviation of images P1 and P2. The closer the two images are, the closer the result is to 1. Define the distance between two images as:

If the distance is less than the threshold, the image slice is considered to contain the target.

Use the mouse to select the gesture in the left camera video, and the device will train the threshold of the variance classifier and the parameters of the Bayesian classifier and save the template of the nearest neighbor classifier. The operator must keep the gesture unchanged at this time. It can properly move and rotate the hand at multiple angles to simulate possible rotation angles during the control process. After the learning is completed, click Save, and the device can save the multi-scale and multi-transformation template.

Step 4 is the tracking step, the purpose of which is to first identify the rectangular frame where the hand is located in the picture captured by the camera, and then according to the movement trajectory of the hand, specifically: the operator appears in the camera's field of view according to the gesture in Figure 2 Among them, the device automatically detects, using the trained cascade variance classifier, Bayesian classifier based on random forest, and the nearest neighbor classifier to detect the target (that is, the process of identifying the rectangular frame where the hand is located). When the target is detected, it enters the cycle of tracking and detection (that is, the process of tracking the trajectory of the hand):

Use the pyramid LK optical flow method to track the Shi-Tomasi corner points in the target frame. In order to optimize the tracking effect, use the forward and backward error and NCC algorithm to remove some of the poorly tracked points. The tracking process is as follows:

1. According to the initially selected rectangular frame, use the NCC algorithm to perform template matching in another camera to obtain the initial rectangular frame in another camera.

2. Enter the tracking loop. In the generated sliding rectangles, find the rectangle with the highest overlapping degree of the target tracking frame as the best tracking sample, and then calculate the Shi-Tomasi corner points in this rectangle as the feature points.

3. Forward and backward errors and matching similarity, filter out the points that meet the conditions (these points meet the feature points that are less than the given average forward and backward error threshold and greater than the specified matching similarity threshold), and will be filtered after the end Drop about half of the feature points.

4. Use the remaining feature points to predict the size and position of the target frame in the current frame. According to the average translation of the tracked feature points and the feature points corresponding to the previous frame, the position of the target frame in the current frame is obtained. At the same time, according to these features The ratio of the corresponding Euclidean distance between the points in the two frames of images before and after to get the size of the target frame in the current frame. Has no effect if the position exceeds the image.

5. Calculate the similarity between the normalized image slice and the online model. If the similarity is greater than the specified threshold, the tracking is finally considered valid and stored in the positive sample set. Otherwise, it is considered invalid and still discarded.

6. Return to step 2.

The left and right cameras perform tracking calculations in parallel according to the position of the target in the initial frame. The detection cycle using the classifier and the above tracking cycle are calculated in parallel. The entire image is searched and detected in the image obtained by the left camera, and the tracking results are Fusion with the detection results to get the final result, and update the samples in the gesture template at the same time. After the tracking and detection in the left camera image is successful, it will be detected on the corresponding epipolar line of the right camera image and fused with the tracking result of the right camera. The final result can be obtained, which can simplify the calculation amount of the detection algorithm. If the left and right views are tracked successfully at the same time, the target rectangle will be output.

Step five is to determine the space coordinates of the end position and the initial position of the gesture recognized in step four, and then determine how the robot should move, specifically:

The operator appears in the field of view of the camera according to the gesture on the right side of Figure 2(b), points to the device and performs detection and tracking according to the method of step 4. The center of the rectangular box that is being tracked can be regarded as the control point.

The spatial coordinate value of the gesture center is calculated by using the parallax ranging method based on the principle of stereo vision. The formula is as follows:

In the formula, X, Y, Z are the positions of the gesture center in space, u ₁ is the x coordinate of the marker ball in the left camera image coordinate system, u ₀ is the x origin of the left camera image coordinate system, u ₂ is the marker ball in The x coordinate in the right camera image coordinate system, d is the translation distance between the two cameras, v ₁ is the y coordinate of the marker ball in the left camera image coordinate system, v ₀ is the y origin of the left camera image coordinate system, f is Camera focal length.

After the mouse clicks the calibration button, this point is used as the origin. When the control point leaves the sphere with the control threshold as the radius, the speed command is output, and the size is proportional to the offset distance. The calculation formula is as follows

V=kd

Among them, V is the output control command, k is the control coefficient, and d is the distance from the initial position of the gesture center. Control the end of the robot for translational motion.

Step six is to determine the initial posture and end posture of the gesture recognized in step four, and then determine how the robot should adjust the posture. The pose is represented by a rotation matrix or Euler angles. Step six is specifically:

The operator appears in the field of view of the camera according to the gesture on the left side of Figure 2, and the device performs detection and tracking according to the method in step 3. In the target rectangular frame, the contour of the gesture is obtained by using the background modeling method, and the convex hull detection and convex hull defect detection algorithms are used to obtain the depressions where the index finger, middle finger and ring finger are connected to the corresponding fingers, and a total of five feature points are obtained. The spatial coordinates of these five feature points are obtained according to the positioning method in step 4.

Use the method based on the combination of skin color detection and background difference method for human hand segmentation, in which skin color detection converts the color space from RGB space to HSV space to get better segmentation results. The background subtraction method can obtain a more complete segmentation effect. After obtaining the binary image of the human hand, use the open operation and contour area detection in mathematical morphology to filter the noise. Then use the Graham scanning method to obtain the convex points of the human hand contour, and select the top three to get the positions of the index finger, middle finger and ring finger. Then calculate the point farthest from the two convex points between the adjacent convex points, which is the depression where the fingers are connected, and a total of five feature points are obtained. The spatial coordinates of these five feature points are obtained according to the positioning method in step 4.

According to the coordinate system defined on the palm of the left hand gesture in 2, take the root of the middle finger as the origin, define the direction pointing to the fingertip of the middle finger as the positive direction of the y-axis, define the line parallel to the two depressions on the x-axis, and point to the little finger as the x-axis Positive direction.

According to the Carley theorem to solve the rotation matrix, the Carley vector representation of the rotation matrix is: any rotation matrix without eigenvalue-1 and an anti-symmetric matrix have the following relationship:

R＝(IS _b ) ^-1 (I+S _b )

S _b ＝(R+I) ^-1 (RI)

In the formula, I is the unit matrix, b=(b ₁ , b ₂ , b ₃ ) ^T is the Carley vector.

Let p _i be the space coordinate value of feature point i, and q _i be the coordinate value of feature point i in the palm coordinate system. Then solve the rotation matrix equation as:

The above formula can be transformed into:

In the formula:

v _i =p _i -q _i

u _i =p _i +q _i

S _ui is the antisymmetric matrix corresponding to u _i .

Available equations:

Ab=c

In the formula:

The Carley vector can be obtained by solving the equation, and then the rotation matrix R can be calculated. Then convert the rotation matrix into pitch-yaw-roll Euler angles. After clicking the calibration button with the mouse, the current posture is taken as the original posture, and the Euler angular velocity command is output according to the relative rotation angle to control the posture change of the robot.

Finally, if the operator moves his hand out of view, or makes a gesture that is not in the template, control ends. Repeat steps five and six to continue the control.

<Example>

The concrete process of an embodiment of the present invention is:

(1) Place the binocular cameras as parallel as possible, and use Zhang Zhengyou's checkerboard calibration method for stereo calibration and correction.

(2) Enter the learning template mode, use the mouse to select the gesture in the left camera video, and the device will start to learn and save the template. The operator must keep the gesture unchanged at this time, and can properly move and rotate the hand at multiple angles part, simulate the rotation angle that may occur during the control process, click save after learning, this device can save the multi-scale and multi-transformation template. If multiple different gesture templates are required, the above steps can be repeated.

(3) Binocular vision tracking detection, the operator appears in the field of view of the camera according to the gesture in step 2, and the device automatically detects, after the target is detected by the cascade variance classifier, set classifier, and nearest neighbor classifier Use the pyramid LK optical flow method to track the Shi-Tomasi corner points in the target frame, and fuse the tracking results with the detection results, and update the samples in the gesture template. After successful tracking in the left camera image, the right camera The epipolar line of the image is detected and tracked, and if the left and right views are tracked successfully at the same time, the target rectangle is output.

(4) In the position control mode, the operator appears in the field of view of the camera according to the gesture in step 2, and the device performs detection and tracking according to the method in step 3. The center of the rectangular box that is being tracked can be regarded as the control point. The spatial coordinate value of the gesture center is obtained by using the three-dimensional reconstruction principle of stereo vision. After the mouse clicks the calibration button, this point is used as the origin. When the control point leaves the sphere with the control threshold as the radius, the speed command is output, and the size is proportional to the offset distance: V=kd.

(5) In the attitude control mode, the operator appears in the field of view of the camera according to the gesture in step 2, and the device performs detection and tracking according to the method in step 3. In the target rectangular frame, the contour of the gesture is obtained by the background modeling method, and the index finger, middle finger and ring finger are connected with the corresponding fingers by using the convex hull detection and convex hull defect detection algorithms. The coordinate system defined on the palm in advance, solves the rotation matrix according to Carley's theorem, and then converts it into pitch-yaw=roll Euler angles. After clicking the calibration button with the mouse, the current posture is taken as the original posture, and the Euler angular velocity command is output according to the relative rotation angle to control the posture change of the robot.

(6) The operator moves his hand away from the field of vision, or makes a gesture that is not in the template, then the control ends. Repeat steps 3 and 4 to continue the control.

The present invention can also have other various embodiments, without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations are all Should belong to the scope of protection of the appended claims of the present invention.

Claims (9)

1. A multi-gesture robot control method based on binocular vision is characterized by comprising

Step one, setting a binocular camera, and calibrating and correcting;

secondly, performing gesture demonstration in the visual field range of the binocular camera by an operator, manually selecting a rectangular frame containing a gesture in a video shot by a left video camera of the binocular camera, and adding the rectangular frame into a training sample set;

step three, training a nearest neighbor classifier and a Bayes classifier by using a training sample set;

step four, the operator appears in the visual field of the binocular camera according to the gesture in the step two; the processor utilizes a cascade variance classifier according to the image of the left camera, and a Bayes classifier based on a random forest and a nearest neighbor classifier are used for detecting to obtain a target; tracking the target, fusing a tracking result and a detection result, updating samples in the gesture template, detecting and tracking on the polar line of the right camera image after the tracking is successful in the left camera image, and outputting a target rectangular frame if the tracking is successful in the left and right views;

step five, tracking the central point of the target rectangular frame; calculating the offset distance of the central point from the initial point to the target point, and outputting a speed control instruction to enable the robot to perform translational motion;

and step six, extracting characteristic points for describing the gesture outline from the target rectangular frame, and solving a rotation matrix corresponding to the characteristic points to enable the robot to perform posture conversion.

2. The binocular vision-based multi-gesture robot control method according to claim 1, wherein the first step specifically comprises:

step one, setting the distance between a left camera and a right camera in a binocular camera to be 20cm, and horizontally placing the left camera and the right camera;

and step two, calibrating the binocular camera by using a Zhang Zhengyou calibration method, and eliminating distortion and line alignment of the views of the left camera and the right camera so as to enable the imaging origin coordinates of the views of the left camera and the right camera to be consistent, the optical axes to be parallel, the imaging planes to be coplanar and the polar lines to be aligned.

3. The binocular vision-based multi-gesture robot control method according to claim 1, wherein the second step specifically comprises:

secondly, performing gesture demonstration by an operator in the visual field range of the binocular camera;

secondly, manually selecting a rectangular frame containing gestures from a video shot by a left camera of a binocular camera;

secondly, scaling, rotating and affine processing are carried out on the image blocks in the selected rectangular frame, and the scaled, rotated and affine images are normalized into image blocks with the same size to form a positive sample set; selecting a preset number of image blocks in the original image, wherein the distance between the image blocks in the original image and the selected image blocks is greater than a preset threshold value to form a negative sample set; the positive sample set and the negative sample set together constitute a training sample set.

4. The binocular vision-based multi-gesture robot control method of claim 1, wherein in step three,

calculating the posterior probability of the foreground class of the Bayesian classifier by the following formula:

wherein y is ₁ Represents the foreground when y ₁ When =0, y represents that there is no object in the image ₁ 1, indicating that the image contains the target; x is the number of _i An ith feature representing an image; each characteristic of the image is the gray value size relation of two randomly selected points in the image, and the gray value size relation is represented by 0 or 1.

5. The binocular vision-based multi-gesture robot control method of claim 4, wherein in step three,

the number of the Bayesian classifiers is 10; characteristic x _i The corresponding number of positive samples is # p, the number of negative samples is # n, and the total number of samples is # m, then:

solving for p (y) for each Bayesian classifier ₁ |x _i ) And averaging the results, and if the average value is larger than a preset threshold value, determining that the target exists in the image.

6. The binocular vision-based multi-gesture robot control method according to claim 1 or 4, wherein in step three, the nearest neighbor classifier is used for calculating the similarity of two image blocks, and the calculation formula is as follows:

in the formula of ₁ ,μ ₂ ,σ ₁ ,σ ₂ Represents the mean and standard deviation of images P1 and P2; the more similar the two images, the closer the result is to 1; the distance between the two images is defined as:the image slice is considered to contain the object when the distance between the two images is less than a predetermined threshold.

7. The binocular vision based multi-gesture robot control method according to claim 6, wherein the fourth step is specifically:

fourthly, the operator appears in the visual field of the binocular camera according to the gesture in the second step;

step two, manually selecting an initial rectangular frame by an operator;

step two, the processor generates a sliding rectangular frame, a cascade variance classifier is used for filtering out the rectangular frame which does not meet the variance threshold condition, then an image block which possibly contains the foreground is obtained through screening of a Bayesian classifier, and then the similarity between the sliding rectangular frame and the manually selected initial rectangular frame is calculated through a nearest neighbor classifier;

selecting a rectangular frame with the highest overlapping degree as a sample rectangular frame, and calculating Shi-Tomasi angular points in the sample rectangular frame as characteristic points;

fourthly, calculating a forward prediction error, a backward prediction error and a similarity in a sample rectangular frame, and screening out feature points which are smaller than the average value of the forward prediction error and the backward prediction error and are larger than a preset similarity threshold;

step four, calculating the average displacement of the feature points screened from the current frame and the corresponding feature points of the previous frame to obtain the position of the target frame of the current frame, and obtaining the size of the target frame in the current frame according to the ratio of the Euclidean distance between the feature points of the previous frame and the current frame;

and step four, carrying out normalization processing on the target frame obtained in the step four, calculating the similarity between the normalized target frame and all images in the positive sample set, if one similarity is greater than a specified threshold value, effectively tracking, adding the obtained target frame into the sample set, and if not, considering that the tracking is invalid and discarding.

8. The binocular vision-based multi-gesture robot control method according to claim 7, wherein the step five specifically comprises:

fifthly, taking the center point of the target frame obtained in the fourth step and the sixth step as a gesture center point, and calculating a space coordinate value of the gesture center by using a stereoscopic vision principle parallax ranging method, wherein the method specifically comprises the following steps:

where X, Y, Z are the positions of the gesture center points in space, u ₁ For marking the x-coordinate, u, of the sphere in the left camera image coordinate system ₀ Is the x origin, u, of the left camera image coordinate system ₂ Is the x coordinate of the marker sphere in the right camera image coordinate system, d is the translation distance between the two cameras, v ₁ For marking the y-coordinate, v, of the sphere in the left camera image coordinate system ₀ Is the y origin of the left camera image coordinate system, and f is the camera focal length;

step two, an operator sets an origin at any position through a calibration button; when the processor detects that the gesture center point leaves a sphere with a preset control threshold value as a radius, a speed control instruction is output, and the calculation formula is as follows:

V＝kd

v is an output speed control instruction, k is a control coefficient, and d is the distance of the center of the gesture deviating from the initial position; the speed control command is used for controlling the robot to perform translational motion.

9. The binocular vision based multi-gesture robot control method according to claim 8, wherein the sixth step specifically includes:

sixthly, obtaining the outline of the gesture in the target frame obtained in the step IV by a method based on combination of skin color detection and a background difference method, and obtaining 5 feature points of the index finger, the middle finger, the ring finger, the dent of the index finger and the ring finger and the dent of the middle finger and the ring finger by a convex hull detection algorithm and a convex hull defect detection algorithm; obtaining the space coordinates of the 5 characteristic points through the formula in the fifth step;

defining a coordinate system on the palm, taking the root of the middle finger as an origin, defining the direction pointing to the fingertip of the middle finger as the positive direction of the y axis, defining a connecting line parallel to the two dents as the x axis, and defining the direction pointing to the little finger as the positive direction of the x axis;

sixthly, solving a rotation matrix by utilizing 5 characteristic points according to Carley theorem;

and sixthly, converting the rotation matrix into a pitch-yaw-roll Euler angle, acquiring a relative rotation angle of the gesture in the process of converting the current gesture into the original state, and outputting an Euler angular speed instruction according to the relative rotation angle to control the gesture change of the robot.