CN112800905A - Pull-up counting method based on RGBD camera attitude estimation - Google Patents

Abstract

The invention discloses a pull-up counting method based on RGBD camera attitude estimation, which collects RGB and Depth images, and processes the RGB and Depth images to obtain images that suppress most of the background information; Input into the mobi le net series network structure model for effective fusion, and output the key point confidence map and partial affinity field map; evaluate the correlation between the two key points through the integral function, and connect the joint points of each person , get the posture skeleton map of everyone in the image; extract the pull-up motion parameters according to the information of the skeleton key points in the posture skeleton map; judge whether to perform the pull-up action according to the motion parameters, and if so, perform the pull-up count. The present invention can adapt to various motion scenes, and divides the background information through the Depth information, the counting mechanism is more robust, the counting is more accurate, and the calculation efficiency is improved at the same time.

Description

A pull-up counting method based on RGBD camera pose estimation

technical field

The invention relates to a human body counting method, in particular to a pull-up counting method based on RGBD camera attitude estimation.

Background technique

Pull-ups require a certain amount of grip strength and upper body strength, which must overcome one’s own body weight to complete one time. Pull-ups play an important role in developing upper limb suspension strength, shoulder girdle strength and grip strength. Therefore, pull-ups are fitness and exercise. One of the most common projects, for scientific and effective fitness and exercise, when doing pull-ups, it is necessary to count the pull-ups effectively.

At present, the method of counting pull-ups includes manual counting. This counting method relies on the subjective judgment of the counting personnel, which is prone to errors in ambiguous situations, and also wastes manpower;

China Invention Patent Authorization Announcement No. CN105879358B, the authorization announcement date is August 28, 2018, and the patent name is "Pull-up Performance Tester". This method uses a pull-wire displacement sensor. The tested person is inconvenient, and the upper and lower bar sensing unit needs to be installed, and the design is more complicated;

Chinese Invention Patent Publication No. CN 107122798A, published on September 1, 2017, patent title "Pull-up counting detection method and device based on deep convolutional network", the method discloses a pull-up using deep learning. Counting, this method has two problems. First, it collects a lot of data. It needs to collect a large amount of data such as nose passing the rod, head passing the rod but the nose not passing the rod. Material resources, second, cannot handle ambiguous situations very well.

Chinese Invention Patent Publication No. CN 111282248A, published on June 16, 2020, patent title "A system and method for pull-up detection based on key points of bones and faces", the method uses a single frame of the upper arm and forearm There are two problems with this method: First, when a certain key point fails to be detected, these counting mechanisms fail, resulting in counting errors. Second, when there are many people in the RGB image, the key point information of everyone will be detected, and the key point information of multiple people will affect each other, resulting in inaccurate counting; at the same time, detecting the key point information of many people will also bring a certain time. extension.

Therefore, it is necessary to propose a new solution, a pull-up counting method based on RGBD camera pose estimation, which obtains RGB and Depth images, processes the RGB and Depth images, and inputs the processed RGB and Depth images to the In the network model, the posture skeleton map of everyone in the image is obtained, and the corresponding action logic mechanism of the pull-up is designed according to the posture skeleton map, and the plan of counting the pull-up is carried out.

SUMMARY OF THE INVENTION

Aiming at the problem of current pull-up counting, the present invention provides a pull-up counting method based on RGBD camera attitude estimation, and combines the complementary characteristics of RGBD images to design the network structure, so that the network can adaptively integrate RGB and Depth. Image features, the network simultaneously returns to the key points of the human skeleton and associates the key points to obtain the skeleton diagram of the human body posture, and then logically judges and compares the detected key points, designs the pull-up logic and action, and counts the pull-up. .

According to the purpose of the present invention, there is provided a method for counting pull-ups based on RGBD camera attitude estimation, comprising the following steps:

S1: Collect RGB and Depth images, and process RGB and Depth images to obtain images that suppress most of the background information;

S2: Input the processed RGB and Depth images into the mobile net series network structure model for effective fusion, and output the key point confidence map and partial affinity field map; evaluate the correlation between the two key points through the integral function , connect the joint points of each person to get the pose skeleton map of everyone in the image;

S3: Extract the pull-up motion parameters according to the information of the skeleton key points in the posture skeleton diagram;

S4: Determine whether to perform a pull-up action according to the motion parameters, and if so, perform a pull-up count.

Preferably, the step of processing the RGB and Depth images in step S1 includes,

S11: Use a RGBD camera with spatial and temporal consistency to collect RGB and Depth images, and perform background segmentation on the RGB and Depth images respectively;

Specifically, let the coordinates of a pixel in the RGB image be X _R (i,j), and the corresponding pixel coordinates of the depth map be X _D (i, j), and generate a mask map according to the resolution of the depth map. The mask map corresponds to The pixel coordinates are X _M (i, j), and a controllable threshold δ is designed according to the complexity of the scene. For example, the range of the character's activities is used as the threshold standard, and the mask image is optimized by the binarization operation;

S12: Do point multiplication of the optimized mask image with the RGB and Depth images respectively to suppress most of the background information in the RGB and Depth images.

Preferably, in step S2, the step of obtaining the posture skeleton diagram of all people includes:

S21: After passing the processed RGB and depth images through the mask, RGB_f and Depth_f are obtained through the two branches of the network respectively; at the same time, a 1x2 weight vector [W _D , W _R ] is learned, representing the weights of the RGB and Depth modes respectively , Multiply the modal weights [W _D , W _R ] with RGB_f and Depth_f respectively, and then fuse the RGB and Depth feature maps to obtain the fused features;

S22: Input the fused features into the stage1 network structure. The output of each stage has two branches. The two branches output the keypoint confidence map and the affinity field of the keypoint respectively. The input of stage n is stage n-1 Output;

S23: After obtaining the position of the affinity field and the key point, evaluate the correlation of the two key points through the integral function;

S24: Use the Hungarian algorithm to obtain the optimal matching of adjacent key points, and obtain the pose skeleton map of everyone in the image.

Preferably, the network structure of each stage in the mobile net series network structure model adopts 3×3 and 1×1 convolutional layers, and uses hole convolution to increase the receptive field of the network.

Preferably, the true value confidence map is obtained through the maximization operation, and during the test, the positions of the key points are obtained through the maximization operation, and redundant key points are eliminated by non-maximum suppression.

Preferably, a loss function is added to the output of each stage branch, and the loss function is constrained by the L2 norm.

Preferably, the mobile net series network structure model uses a NAS (Neural Architecture Search) search method to balance the accuracy and speed of the network.

Preferably, the pull-up motion parameters in the step S3 include: head position change characteristics, arm position change characteristics; the head position change characteristics refer to the change of head height during the movement process, through The position changes of three key points of nose, ears and eyes are estimated;

The arm variation feature refers to the variation of the arm bending during the pull-up process, which is estimated by the position variation of three key points of the wrist, the elbow and the shoulder.

Preferably, the arm bending change is determined by judging whether the length of the connecting line from the wrist to the shoulder is greater than 0.9 times the sum of the lengths of the elbow to the wrist and the elbow to the shoulder.

The beneficial effects of the present invention are:

1. The present invention does not rely on the subjective judgment of counting personnel, avoids being prone to errors in ambiguous situations, and saves manpower at the same time.

2. The present invention does not need complicated devices, but only needs one RGBD camera, and the price is low.

3. The present invention uses the Depth information to segment the background information, so that the depth value is optimized, and the complementary characteristics of RGBD are fully utilized. Combined with the multi-modal input of RGBD, a robust deep learning algorithm is designed to estimate the key points of the human body, and the Compression of the network model; enabling it to achieve real-time effects on edge devices, designing a corresponding action logic mechanism according to motion attributes, and performing pull-up counting; the present invention can adapt to various motion scenarios, segment background information through Depth information, and count The mechanism is more robust, the counting is more accurate, and the calculation efficiency is improved.

4. When designing the corresponding action logic of the pull-up, the position of multiple key points is comprehensively evaluated to avoid the failure of the counting mechanism because a certain key point cannot be collected.

Description of drawings

Fig. 1 is the flow chart of counting method of the present invention;

Fig. 2 is the flow chart of mobile net series network structure model of the present invention;

Fig. 3 is the network structure of each stage of mobile net series network structure model of the present invention;

Fig. 4 is the key point diagram of human skeleton of the present invention;

Fig. 5 is the state that the pull-up starts counting or the starting state of the next counting;

Fig. 6 is the state that the pull-up starts counting and adds 1;

Description of the drawings in Figure 4: 0: nose; 1: neck; 2: right shoulder; 3: right elbow; 4: right wrist; 5: left shoulder; 6: left elbow; 7: left wrist; 8: right hip; 9: right Knee; 10: Right Ankle; 11: Left Hip; 12: Left Knee; 13: Left Ankle; 14: Right Eye; 15: Left Eye; 16: Right Ear; 17: Left Ear;

Detailed ways

The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings, but these embodiments do not limit the present invention, and those of ordinary skill in the art can make structural, method, or functional transformations according to these embodiments. All are included in the protection scope of the present invention.

As shown in FIG. 1 , a method for counting pull-ups based on RGBD camera attitude estimation disclosed in the present invention includes the following steps:

S1: Collect RGB and Depth images, and process RGB and Depth images to obtain images that suppress most of the background information;

There is a specific embodiment, the step of processing RGB and Depth images in step S1 includes,

S11: Use a RGBD camera with spatial and temporal consistency to collect RGB and Depth images, and perform background segmentation on the RGB and Depth images respectively;

Specifically, let the coordinates of a pixel in the RGB image be X _R (i,j), and the corresponding pixel coordinates of the depth map be X _D (i, j), and generate a mask map according to the resolution of the depth map. The mask map corresponds to The pixel coordinates are X _M (i, j), and a controllable threshold δ is designed according to the complexity of the scene. For example, the range of the character's activities is used as the threshold standard, and the mask image is optimized by the binarization operation; the formula is as follows:

S12: Do point multiplication of the optimized mask image with the RGB and Depth images respectively to suppress most of the background information in the RGB and Depth images. The formula is as follows:

X _R (i,j) = X _M (i, j) · X _R (i, j) (2)

X _D (i,j) = X _M (i, j) · X _D (i, j) (3)

S2: Input the processed RGB and Depth images into the mobile net series network structure model for effective fusion, and output the key point confidence map and partial affinity field map; evaluate the correlation between the two key points through the integral function , connect the joint points of each person to get the pose skeleton map of everyone in the image;

As shown in Figure 2, there is a specific embodiment, and the step of obtaining the pose skeleton diagram of everyone in step S2 includes:

S21: After passing the processed RGB and depth images through the mask, RGB_f and Depth_f are obtained through the two branches of the network respectively; at the same time, a 1x2 weight vector [W _D , W _R ] is learned, representing the weights of the RGB and Depth modes respectively , Multiply the modal weights [W _D , W _R ] with RGB_f and Depth_f respectively, and then fuse the RGB and Depth feature maps to obtain the fused features;

stage n的输入为stage n-1的输出；两个分支都是一轮迭代的预测体系结构，具体迭代公式如下：S22: Input the fused features into the stage1 network structure, the output of each stage has two branches, and the two branches output the key point confidence map S ¹ =ρ ¹ (F) and the affinity field of the key point respectively

The input of stage n is the output of stage n-1; both branches are an iterative prediction architecture, and the specific iteration formula is as follows:

S23: After obtaining the position of the affinity field and the key point, evaluate the correlation of the two key points through the integral function;

S24: Use the Hungarian algorithm to obtain the optimal matching of adjacent key points, and obtain the pose skeleton map of everyone in the image.

There is a preferred solution, as shown in Figure 3, the network structure of each stage in the mobile net series network structure model adopts 3 × 3 and 1 × 1 convolution layers, and uses atrous convolution to increase the receptive field of the network.

There is a preferred solution, the joint point position and the affinity region need to be supervised separately during training, and the loss function is constrained by the L2 norm. In order to avoid the phenomenon of gradient disappearance, a loss function is added to the output of each branch in each stage to play a role of relay supervision.

The loss function of each branch is as follows:

是有J个真实关键点的置信图，

是有C个真实的部分亲和场。W是一个边界标志，当图像位置P的注释消失时，W(P)＝0。这个标记是为了避免无标记部分参与到模型权重的优化。in,

is a confidence map with J real keypoints,

is that there are C real partial affinity fields. W is a boundary marker, W(P)=0 when the annotation of image position P disappears. This mark is to avoid the unmarked part participating in the optimization of model weights.

During training, for each person k's position p, generate a personal keypoint confidence map

The way is:

where X _j,k is the individual k, the location of the true value of the key point j, and σ is the coefficient that controls the peak range. The confidence map of the truth value is obtained by the maximization operation:

During testing, the positions of key points are obtained by maximizing operation, and redundant key points are excluded by non-maximum suppression.

The partial affinity field on the cth limb of individual k is defined as:

where v=(x _j2,k -x _j1,k )/||x _j2,k -x _j1,k || ₂ , x _j,k represents the jth key point position of individual k, whether the pixel P falls on Judgments on the limbs are:

0≤v·(px _j1,k )≤l _c,k &&|v _⊥ ·(px _j1,k )|≤σ _l

Among them, l _{c, k} and σ _l represent the length and width of the limbs, and finally average the limbs of the same category of all people, so that the output channel of the affinity field is equal to the number of limbs:

After obtaining the affinity field and the location d _j of the keypoints, the correlation of the two keypoints is evaluated by the following integral function:

Among them, p(u)=(1-u)d _j1 +ud _j2 After obtaining the edge weights of key points and correlations, the calculation of pose bones is converted into a graph problem.

The Hungarian algorithm is used to obtain the optimal matching of adjacent key points, so as to obtain the pose skeleton map of everyone in the image.

There is a preferred solution, the mobile net series network structure model uses the NAS (Neural ArchitectureSearch) search method to balance the accuracy and speed of the network.

S3: Extract the pull-up motion parameters according to the information of the skeleton key points in the posture skeleton diagram;

As shown in Figure 4, the corresponding positions of the key points of the human skeleton; there is a preferred solution, the pull-up motion parameters in step S3 include: head position change characteristics, arm position change characteristics; the head position change characteristics It refers to the change of the height of the head during the movement, which is estimated by the position changes of the three key points of the nose, ears and eyes;

Specifically, the height change of the head position is mainly obtained by comprehensively considering the position changes of the nose, ears and eyes. The overall movement of the head is defined as follows:

y _head ＝α·y _ears +β·y _eyes +γ·y _nose

(11)

Among them, y _head represents the height of the head at this time, y _ears , y _eyes , and y _nose represent the height of the ears, eyes, and nose, and α, β, and γ represent the corresponding weights.

The arm change feature refers to the change of the arm bending during the pull-up process, which is estimated by the position changes of the three key points of the wrist, elbow and shoulder;

More specifically, the change of arm bending is determined by judging whether the length of the connecting line from the wrist to the shoulder is greater than 0.9 times the sum of the lengths of the elbow to the wrist and the elbow to the shoulder.

The specific formula is as follows:

Among them, x _wrist , x _elbow and x _shoulder represent the abscissa of wrist, elbow and shoulder respectively, y _wrist , y _elbow and y _shoulder respectively represent the ordinate of wrist, elbow and shoulder.

S4: Determine whether to perform a pull-up action according to the motion parameters, and if so, perform a pull-up count.

The specific counting process is as follows:

1) When the arm is in a straight state and the head position changes |y _head -y ₀ |>ε, it means that the body is hanging on the horizontal bar; as shown in Figure 5, the pull-up starts to count or the next count start state;

2) When the arm is in a bent state and the head position changes |y _head -y ₀ |≤ε, it means that the head is over the horizontal bar;

3) When the situation from step 1) to step 2) occurs, increase the count by 1, as shown in Figure 6; when step 2) to step 1) occurs, it means entering the start state of the next count; such a loop count.

Among them, y ₀ represents the height of the pull-up bar, and ε represents a certain distance threshold;

It should be noted that RGB images are greatly affected by challenges such as light, background noise, and motion occlusion, but RGB has rich texture characteristics. The Depth image has the contour information of the target, can distinguish targets with distance differences, and is not very sensitive to light changes, but the Depth image lacks the texture features of the target.

The present invention is based on a pull-up counting method based on RGBD camera attitude estimation, obtains RGB images and Depth images through a self-developed RGBD camera, and processes the RGB images and Depth images and inputs them to the network model for effective fusion of RGBD features, thereby effectively reducing the risk of human body damage. The influence of each challenge factor in the attitude estimation on the performance of the algorithm is obtained. After obtaining the human body pose skeleton diagram, the pull-up motion parameters are extracted according to the information of the athlete's skeletal joint points, and the pull-up is counted. The present invention can adapt to various motion scenes, and divides the background information through the Depth information, the counting mechanism is more robust, the counting is more accurate, and the calculation efficiency is improved at the same time.

Although the preferred embodiments of this invention have been disclosed for illustrative purposes, those of ordinary skill in the art will recognize that various modifications are possible without departing from the scope and spirit of the invention as disclosed in the accompanying claims. , additions and substitutions are possible.

Claims (9)

1. A chin up-counting method based on RGBD camera pose estimation is characterized by comprising the following steps:

s1: collecting RGB and Depth images, and processing the RGB and Depth images to obtain an image with most background information suppressed;

s2: inputting the processed RGB and Depth images into a mobile net series network structure model for effective fusion, and outputting a key point confidence map and a partial affinity field map; evaluating the correlation between the two key points through an integral function, and connecting the joint points of all people to obtain a posture skeleton diagram of all people in the image;

s3: extracting a body upward motion parameter according to the information of skeleton key points in the posture skeleton diagram;

s4: and judging whether to perform pull-up action according to the motion parameters, and if so, performing pull-up counting.

2. The method of claim 1, wherein the method comprises: the step of processing the RGB and Depth images in step S1 includes,

s11, collecting RGB and Depth images by using an RGBD camera with space-time consistency, and respectively carrying out background segmentation on the RGB and Depth images;

specifically, the coordinate of a certain pixel point of the RGB image is X_R(i, j), the corresponding depth map pixel point coordinate is X_D(i, j), generating a mask map according to the resolution of the depth map, wherein the coordinate of a pixel point corresponding to the mask map is X_M(i, j), designing a controllable threshold value delta according to the complexity of the scene, and if the range of the character activity is used as a threshold value standard, carrying out binarization operation optimization on the mask map;

s12: and performing dot multiplication on the optimized mask image and the RGB and Depth images respectively to inhibit most background information in the RGB and Depth images.

3. The method of claim 1, wherein the method comprises: the step of obtaining the posture skeleton map of all persons in the step S2 includes,

s21, passing the processed RGB and Depth images through a mask, and respectively obtaining RGB _ f and Depth _ f through two branched networks; learning a 1x2 weight vector [ W ] simultaneously_D,W_R]Weights for RGB and Depth modes, respectively, are expressed, and the mode weight [ W [ ]_D,W_R]Multiplying the RGB _ f and Depth _ f respectively, and then fusing the RGB and Depth feature maps to obtain fused features;

s22, inputting the fused features into a stage1 network structure, wherein the output of each stage is provided with two branches which respectively output a key point confidence graph and an affinity field of a key point, and the input of stage n is the output of stage n-1;

s23, after the positions of the affinity field and the key points are obtained, evaluating the correlation of the two key points through an integral function;

and S24, obtaining the optimal matching of the adjacent key points by using the Hungarian algorithm to obtain the posture skeleton diagram of all people in the image.

4. The method of claim 3, wherein the method comprises: each stage in the mobile net series network structure model adopts convolution layers of 3 x 3 and 1x 1, and the cavity convolution is used for increasing the receptive field of the network.

5. The method of claim 3, wherein the method comprises: and obtaining a true confidence map through the maximization operation, obtaining the positions of the key points through the maximization operation during testing, and eliminating redundant key points by utilizing non-maximum suppression.

6. The method of claim 3, wherein the method comprises: each stage branch is added with a loss function at the output, and the loss functions are constrained by an L2 norm.

7. The method of claim 3, wherein the method comprises: the mobile net series network structure model uses the method of NAS (neural Architecture search) search to balance the accuracy and speed of the network.

8. The method of claim 1, wherein the method comprises: the parameters of the pull-up motion in the step S3 include: a head position change feature, an arm position change feature; the head position change characteristic refers to the head height change condition in the movement process, and is estimated through the position change of three key points of a nose, ears and eyes;

the arm change characteristics refer to the change situation of arm bending in the process of performing pull-up, and are estimated through the position change of three key points of a wrist, an elbow and a shoulder.

9. The RGBD camera pose estimation based pull-up counting method according to claim 8, wherein: and judging whether the length of a connecting line from the wrist to the shoulder is greater than 0.9 time of the sum of the lengths from the elbow to the wrist and the elbow to the shoulder.

Images