CN111311729A - Natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network - Google Patents

Abstract

The invention discloses a natural scene three-dimensional human body posture reconstruction method based on a bidirectional projection network, which aims at solving the problem that the human body three-dimensional posture reconstruction process in the prior art still needs to be improved. The invention comprises the following steps: firstly, acquiring data by using a camera; secondly, sending the collected video and image data to a two-dimensional posture detector to obtain two-dimensional human body joint point coordinates of corresponding postures; designing two-way projection networks with two structures according to the existence of the three-dimensional attitude data tags in the training process; training the designed network by using a deep-antagonistic learning strategy, minimizing a network loss function, and iterating to finally obtain a trained three-dimensional posture generator; and fifthly, inputting the output result of the two-dimensional posture detector in the step two into the three-dimensional posture generator trained in the step four. The technology is low in cost, can assist VR and AR technologies in the 5G era, establishes portable somatosensory interaction equipment, and realizes large-scale popularization and application of three-dimensional motion reconstruction technologies.

Description

A 3D Human Pose Reconstruction Method in Natural Scenes Based on Bidirectional Projection Network

technical field

The invention relates to the technical field of computer vision, in particular to a method for reconstructing a three-dimensional human body posture in a natural scene based on a bidirectional projection network.

Background technique

In virtual reality technology and somatosensory human-computer interaction, it is usually necessary to accurately capture the movements of the human body and reconstruct a moving three-dimensional human skeleton. Existing methods usually use a professional motion capture device (MOCAP) or some hardware peripherals such as a somatosensory camera (Kinect) to complete the three-dimensional human pose reconstruction. However, these professional equipment are usually expensive and have extremely high requirements on the experimental environment, which hinders the wide-scale promotion and application of 3D pose reconstruction technology. Human 3D pose estimation in monocular images is a difficult task in computer vision, and 3D pose reconstruction based on 2D joints is a thorny ill-posed problem. Most of the existing methods usually rely on paired label data for supervised training of the network, and the model performance is poor in the absence of label data and explicit correspondence. Therefore, using deep learning technology to improve the above-mentioned three-dimensional human pose reconstruction process, the whole process can get rid of the dependence of professional hardware peripherals, and only rely on ordinary mobile phones and cameras to complete the three-dimensional human pose reconstruction in natural scenes.

In the existing deep learning methods, the network usually relies on paired human pose data with labels to train the network. In the absence of 3D labels and clear correspondence, the model is difficult to train and the generalization performance is poor, making it difficult to Reasonable 3D reconstruction of complex and changeable special human postures in the natural environment. Therefore, it is of great significance to design a deep learning scheme that can perform accurate 3D human pose reconstruction in natural scenes and does not rely on label data in the training process. It can replace professional motion capture equipment at a very low cost, and can complete 3D pose reconstruction.

SUMMARY OF THE INVENTION

The invention overcomes the problem that the human body three-dimensional posture reconstruction process still needs to be improved in the prior art, and provides a three-dimensional human body posture reconstruction method in natural scenes based on a bidirectional projection network, which can use a monocular camera to perform three-dimensional reconstruction of human movements in natural scenes. .

The technical solution of the present invention is to provide a three-dimensional human body posture reconstruction method based on a bidirectional projection network with the following steps: including the following steps:

Step 1. Use a camera to collect human motion video or image data in a natural scene;

Step 2, sending the collected video and image data into a two-dimensional attitude detector to obtain the two-dimensional body joint point coordinates of the corresponding attitude;

Step 3: Design bidirectional projection networks with two structures according to whether there are three-dimensional attitude data labels in the training process;

Step 4. Use the deep adversarial learning strategy to train the designed network, minimize the network loss function, and finally obtain the trained 3D pose generator after iteration;

Step 5: Input the output result of the 2D posture detector in Step 2 into the 3D posture generator trained in Step 4, and the output result is the 3D posture data of the person in the video/image.

Preferably, in the first step, an ordinary monocular optical camera or a mobile phone camera is used to complete the collection of the movement data of people in the natural scene, and the data is in the form of pictures or videos.

Preferably, in the second step, the two-dimensional attitude detector is a two-dimensional attitude detection method of OpenPose, StackHourglass or HRNet. When the collected data is a picture, the two-dimensional joint point detection result is obtained by directly inputting the picture. When the collected data is a video When , the two-dimensional joint point detection sequence is obtained by frame-by-frame input.

Preferably, in the third step, according to whether the user has three-dimensional attitude label data, two different structures of A/B bidirectional projection networks are selected. When there is three-dimensional attitude data for use, the bidirectional projection network works in A mode. The network consists of two opposite dual branches, and its network module includes a 3D attitude generator, a 3D attitude discriminator, a 2D attitude projection layer and a 2D attitude discriminator; when there is no 3D attitude data for use, the bidirectional projection network Working in the B mode, the network consists of two projection branches in different directions. The network module includes a 3D pose generator, a 2D pose projection layer and a 2D pose discriminator.

Preferably, in the step 3, the input of the three-dimensional pose generator is two-dimensional joint point coordinates, and the output is three-dimensional joint point coordinates, and its interior includes two deep residual networks and one attitude feature extraction layer, and the deep residual network consists of four The number of neurons in each layer is 1024, and the pose feature extraction layer completes the coding and compression of the pose topology structure; the two-dimensional pose discriminator and the three-dimensional pose discriminator have the same network architecture, which contains two / 3D pose feature extraction layer, deep residual network and a fully connected layer, the input of the 2D/3D discriminator module is the pose vector of different dimensions, and the output is a unary discriminant value; the 2D pose projection layer contains the residual network positive value. There are two branches: direction projection and rotation transformation, respectively project the attitude to different observation angles according to the function. The input of the two-dimensional attitude projection layer module is the three-dimensional attitude data, and the output is the projected two-dimensional attitude data.

Preferably, the step 4 comprises the following sub-steps,

Step 4.1. When there is 3D pose data for network training, select Mode A network architecture for training;

Step 4.1.1. Taking the 2D pose as input, first output an initial depth estimation value through the residual network in the 3D pose generator to obtain the initial estimation result of the 3D pose; then the initial estimation result is passed to the pose feature extraction layer , a feature vector is output through the pose prior topology feature extraction, and the feature vector is passed into the depth residual network again to output the final depth estimation value to generate the final 3D reconstructed pose;

Step 4.1.2. The generated 3D reconstructed pose gets forward projection through the 2D pose projection layer all the way, and calculates the pose error with the input 2D pose, and the other way will send it to the 3D pose discriminator to calculate the distribution error;

Step 4.1.3, take the 3D pose as input, first get the forward projection through the 2D pose projection layer, the forward projection is sent to the 3D pose generator all the way to get the 3D reconstruction result, and calculate the pose error with the input 3D pose , the other is sent to the two-dimensional attitude discriminator to calculate the distribution error;

Step 4.2. When there is no 3D pose data for network training, select Mode B network architecture for training;

Step 4.2.1. Taking the 2D pose as input, first output an initial depth estimation value through the residual network in the 3D pose generator to obtain the initial estimation result of the 3D pose; then the initial estimation result is passed to the pose feature extraction layer , and output a feature vector after the pose prior topology feature extraction, and the feature vector will be passed to the depth residual network again to output the final depth estimation value to generate the final 3D reconstructed pose;

Step 4.2.2. Introduce the 3D reconstructed pose to the 2D pose projection layer to obtain the forward projection and the rotation projection respectively. The forward projection will calculate the attitude error with the input 2D pose, and the rotation projection will be judged by the 2D pose The calculator calculates the two-dimensional distribution error;

Step 4.3. Calculate the loss functions in the two modes of A/B, including the pose loss function and the distribution loss function;

Step 4.3.1. In mode A, the overall loss function of the network is defined as:

loss _A =L _GAN (G _3d ,D _3d )+L _GAN (G _2d ,D _2d )+L _dual (G _2d ,G _3d ), where L _GAN represents the loss function of a generative adversarial network with a gradient penalty term, It reflects the distribution error, and the calculation formula is as follows:

L _dual represents the bidirectional loss of the dual network, which reflects the attitude error. The calculation formula is as follows:

L _dual (G _2d ,G _3d )=||G _2d (G _3d (X _2d ))-X _2d || ₁ +||G _3d (G _2d (X _3d ))-X _3d || ₁

λ is the neural network hyperparameter, G _3d represents the 3D pose generator, G _2d represents the 2D pose projection layer, D _3d and D _2d represent the 3D pose discriminator and 2D pose discriminator, respectively, X _2d and X _3d represent the real 2D pose and 3D pose, A _3d represents the random 3D pose on the line connecting the sampling points of the reconstructed 3D pose distribution and the real 3D pose distribution, A _2d represents the line connecting the sampling points of the projected 2D pose distribution and the real 2D pose distribution A random 2D pose on ;

Step 4.3.2. In mode B, the overall loss function of the network is defined as:

loss _B =L _GAN (G _R2d G _3d ,D _2d )+L _pose (G _K2d G _3d ), where L _GAN represents the loss function of a generative adversarial network with a gradient penalty term, which reflects the distribution error, and the calculation formula is as follows :

L _pose is the reconstruction loss, which reflects the attitude error. The calculation formula is as follows:

L _pose (G _K2d G _3d )=||G _K2d G _3d (X _2d )-X _2d || ₁

λ is the neural network hyperparameter, G _3d represents the 3D pose generator, G _R2d represents the rotation projection transformation of the 2D pose projection layer, G _K2d represents the forward projection transformation of the 2D pose projection layer, D _2d represents the 2D pose discriminator, X _2d represents the real 2D pose data, A _2d represents the random 2D pose on the line connecting the sampling points between the projected 2D pose distribution and the real 2D pose distribution;

Step 4.4: Use the neural network optimizer to adjust the network parameters to minimize the error function, and after iterating for 20-40 EPOCH, the loss function converges, and the trained 3D pose generator is obtained.

The step 5 includes the following sub-steps,

Step 5.1. Pass the video or image data collected by the ordinary camera into the two-dimensional attitude detector, and first obtain the two-dimensional joint point data;

Step 5.2, normalize the output result of the two-dimensional attitude detector, so that it can be directly used as the input of the three-dimensional attitude generator; the normalization processing has the following steps:

Step 5.2.1. Use the detected coordinates of the left and right shoulder joints to reconstruct the center neck coordinates:

Among them: (x _T , y _T ) represents the center neck coordinate, (x _ls , y _ls ) represents the left shoulder coordinate, (x _rs , y _rs ) represents the right shoulder coordinate;

Step 5.2.2. Use the detected left and right shoulder and hip joints to reconstruct the central spine coordinates:

Where: (x _S , y _S ) represents the center spine coordinate, (x _ls , y _ls ) represents the left shoulder coordinate, (x _rs , y _rs ) represents the right shoulder coordinate, (x _lh , y _lh ) represents the left hip coordinate, ( x _rh , y _rh ) represents the coordinates of the right hip;

Step 5.3. Pass the normalized 2D pose data into the 3D pose generator, and the output result is the reconstructed 3D pose. When the input data is image data, the output result is the 3D human pose skeleton; when the input data is video data , the output result is the 3D human skeleton action.

Compared with the prior art, the three-dimensional human body posture reconstruction method based on the bidirectional projection network of the present invention has the following advantages: (1) By training the deep neural network in a data-driven manner, the low-cost human posture can be directly realized through the neural network. 3D reconstruction does not require any expensive hardware equipment, only ordinary cameras or mobile phones can collect data, and based on the visual method to reconstruct the 3D posture of the moving human body, it can replace the professional hardware peripherals to complete the 3D reconstruction of the human body posture. Low cost and easy to use, it can help VR and AR technologies in the 5G era, build portable somatosensory interactive devices, and realize large-scale promotion and application of 3D motion reconstruction technology.

(2) A unique neural network training method is adopted, which makes full use of the physiological structure characteristics of human posture data to add new constraints to the network. Therefore, the network training process does not depend on specific data labels and 3D data sets. The unlabeled deep learning training process, and the trained model has good generalization performance, which can achieve complex 3D human pose estimation tasks in natural scenes.

(3) The present invention designs a bidirectional projection network by studying the two major characteristics of human body posture. It adds the pose prior knowledge contained in the dataset as a new constraint to the training process of the network, which reduces the model's dependence on real 3D data during training, and enables network training without relying on label data. And it can achieve accurate 3D human pose reconstruction in natural scenes.

Description of drawings

1 is a schematic diagram of a bidirectional projection network A mode network structure in the present invention;

Fig. 2 is the network structure schematic diagram of bidirectional projection network B mode in the present invention;

3 is a schematic diagram of the internal structure of a bidirectional projection network component module in the present invention;

Fig. 4 is the overall flow chart in the present invention;

FIG. 5 is an effect diagram of three-dimensional human body posture reconstruction in a natural scene according to the present invention.

Detailed ways

The method for reconstructing three-dimensional human body pose in natural scenes based on a bidirectional projection network of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

1. Introduction of related technologies

Reconstructing human gestures and actions in three-dimensional space is one of the main goals of computer vision. In the last century, some scholars have studied this problem [1]. In order to get rid of the dependence on specialized equipment, some of the early methods are mostly based on feature engineering, reconstructing 3D poses by motion physiology modeling of human skeletal joints [2, 3], or search-based methods, using 3D skeleton's The database dictionary performs nearest neighbor lookup and outputs the corresponding 3D pose for the 2D pose [4, 5]. With the development of deep learning, researchers try to directly output the 3D pose of the human body from RGB images by building an end-to-end model [6, 7, 8, 9], but the complex background of images in natural scenes usually interferes with the end-to-end The 3D pose reconstruction process of the end. In recent years, inferring 3D human pose from monocular vision systems has attracted great attention, and this technology can be widely used in animated movies, virtual reality, behavior recognition, and human-computer interaction. Because recovering 3D pose from 2D observations is inherently an ill-conditioned problem, this is very challenging in computer vision tasks. In natural scenes, affected by factors such as illumination, angle and complex background, it is very difficult to directly infer the 3D pose of the human body in the image. Some previous works have divided this problem into two parts: first, through various advanced 2D human body The keypoint detector estimates the 2D pose from the image, and then performs 3D human pose reconstruction based on the obtained 2D pose. Among them [10] first proposed a simple baseline algorithm, which regards 3D pose reconstruction as a regression task from 2D joint points to 3D coordinate points, and uses neural networks to complete high-quality 3D pose reconstruction. [11] further represented the pose as a distance matrix and transformed this problem into a 2D to 3D distance matrix regression. [12] regarded human pose as a special kind of topological graph data, and designed a Semantic Graph Convolutional Network (SemGCN) to complete the regression task of graph-structured data. However, these methods using 3D labeled data to train networks have two serious limitations: (1) Because 3D pose data is very demanding on experimental conditions, it is usually necessary to use expensive multi-angle motion capture equipment to capture 3D human motion indoors. Therefore, it is usually difficult to obtain a large amount of 3D human pose data for training in real scenes; (2) The strict correspondence in the training process of labeled data will lead to overfitting on a single data set. On the one hand, the model cannot be generalized to other special angles or 2D poses that have never been seen before. On the other hand, the network can only generate 3D pose data in the training set, and cannot make reasonable gestures for complex poses in natural scenes. reconstruction. Both of these limitations are due to the reliance of the training process on 3D labeled data.

In recent years, the accuracy of 2D human joint detection algorithms has been increasing, and it has been able to achieve real-time 2D pose estimation in natural scenes. Therefore, more and more researchers are devoted to using these easily obtained 2D joint point data for 3D pose reconstruction, which is divided into two steps: first use advanced 2D human joint detector to obtain 2D pose from the image, and then use advanced 2D human joint detector to obtain 2D pose from the image. These 2D poses are elevated to 3D. The key point of solving ill-conditioned problems is to combine the characteristics of the problem and add reasonable prior information as constraints. In traditional methods, such constraints are provided by regular terms designed manually, and usually only a single problem can be solved. In the era of deep learning, using the network to automatically learn prior information constraints from data can be regarded as a new way to solve ill-conditioned problems, and a class of problems can be solved by a model trained with a large amount of data.

Therefore, it is the main contribution of the present invention to abstract the important characteristics of the pose data as the constraints of the network. Through the research on the physiological structure characteristics of attitude data, the present invention uses deep learning technology to design a bidirectional projection network, which has two working modes, A/B, and can train the network separately under the condition of three-dimensional data label and unlabeled data. , the trained network can complete complex 3D human pose reconstruction tasks in natural scenes.

2. The proposed method

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Embodiment 1: Referring to Figures 1-5, the overall flow chart of a three-dimensional human body pose reconstruction method in natural scenes based on a bidirectional projection network is shown in Figure 4. After taking pictures or videos with a monocular camera, it first goes through a two-dimensional process. The pose detection network (OpenPose, HRNet, StackHourglass) obtains the corresponding two-dimensional human pose, and obtains the two-dimensional joint point detection results. Before sending the data into the 3D pose generator, it is necessary to select the corresponding mode to train the 3D pose generator according to whether it has 3D label data. When there is 3D pose label data, the bidirectional projection network works in A mode, and when there is no 3D pose label data, the bidirectional projection network works in B mode. Users can select the corresponding mode to train the network according to whether they have 3D human pose data.

When mode A is selected, the bidirectional projection network training process has the structure shown in Figure 1, and the 2D attitude data and the 3D attitude data are respectively sent to the two branches of the bidirectional network. In the first branch, the input 2D pose first generates a 3D pose reconstruction result through the 3D pose generator, and the reconstruction result regenerates the 2D projection result through the 2D pose projection layer; in the second branch, The input 3D pose is first passed to the 2D pose projection layer, and then the output is sent to the 3D pose generator to complete the second reconstruction. The two branches complete a dual operation process. In the two processes, the error of the reconstructed attitude needs to be calculated separately. The error is divided into distribution error and attitude error. The loss function of the entire network is:

loss _A =L _GAN (G _3d ,D _3d )+L _GAN (G _2d ,D _2d )+L _dual (G _2d ,G _3d ),

Among them, _LGAN represents the loss function of the generative adversarial network with the gradient penalty term, which reflects the distribution error, and the calculation formula is as follows:

L _dual represents the bidirectional loss of the dual network, which reflects the attitude error. The calculation formula is as follows:

L _dual (G _2d ,G _3d )=||G _2d (G _3d (X _2d ))-X _2d || ₁ +||G _3d (G _2d (X _3d ))-X _3d || ₁

λ is the neural network hyperparameter, G _3d represents the 3D pose generator, G _2d represents the 2D pose projection layer, D _3d and D _2d represent the 3D pose discriminator and 2D pose discriminator, respectively, X _2d and X _3d represent the real 2D pose and 3D pose, A _3d represents the random 3D pose on the line connecting the sampling points of the reconstructed 3D pose distribution and the real 3D pose distribution, A _2d represents the line connecting the sampling points of the projected 2D pose distribution and the real 2D pose distribution A random 2D pose on ;

When mode B is selected, the bidirectional projection network training process has the structure shown in Figure 2. Mode B does not require any label data. The input 2D pose is firstly reconstructed by the 3D pose generator, and then the 3D pose will be reconstructed through the 3D pose generator. The two-dimensional attitude projection layer completes two projection changes. One branch projects the three-dimensional attitude to the forward observation perspective to obtain the forward two-dimensional projection result, and the other branch performs the rotation projection transformation of the three-dimensional attitude result to obtain the observation results from other perspectives. At this time, the two branches have completed two different observation processes. The attitude error and distribution error can also be obtained by applying two constraints to these two observation results. The loss function of the entire network is:

loss _B =L _GAN (G _R2d G _3d ,D _2d )+L _pose (G _K2d G _3d ),

Among them, _LGAN represents the loss function of the generative adversarial network with the gradient penalty term, which reflects the distribution error, and the calculation formula is as follows:

L _pose is the reconstruction loss, which reflects the attitude error. The calculation formula is as follows:

L _pose (G _K2d G _3d )=||G _K2d G _3d (X _2d )-X _2d || ₁

λ is the neural network hyperparameter, G _3d represents the 3D pose generator, G _R2d represents the rotation projection transformation of the 2D pose projection layer, G _K2d represents the forward projection transformation of the 2D pose projection layer, D _2d represents the 2D pose discriminator, X _2d represents the real 2D pose data, A _2d represents the random 2D pose on the line connecting the sampling points between the projected 2D pose distribution and the real 2D pose distribution;

During the training process, the A/B modes of the bidirectional projection network share the same set of network modules. The network components are shown in Figure 3, including a 3D pose generator, a 2D/3D pose discriminator, and a 2D pose projection. Floor.

The three-dimensional pose generator contains two deep residual networks and one pose feature extraction layer. The deep residual network is composed of four residual blocks stacked, and the number of neurons in each layer is 1024. The input two-dimensional pose first passes through the residual The network outputs an initial depth estimation value to obtain the initial estimation result of the 3D pose, and then the initial estimation result is passed to the pose feature extraction layer. After the pose prior topology feature extraction, the 3D pose will be encoded into a space angle and depth. The feature vector of the information, this feature vector will be passed to the depth residual network again to output the final depth estimation value to generate the final 3D reconstructed pose.

The two-dimensional pose discriminator and the three-dimensional pose discriminator have the same network architecture, and the main difference lies in the feature extraction layer. The poses of the two dimensions are first encoded into a feature vector containing the topology of the motion pose through the corresponding pose feature extraction layer. Then, the final discriminant value is output through the deep residual network and the fully connected layer to complete the calculation of the difference between the two distributions.

The two-dimensional attitude projection layer includes two branches, which can respectively project the attitude to different angles. The observation of the forward perspective is completed by the deep residual network connected by multiple residual blocks, and the observation of other rotating perspectives is done by comparing The pose rotation transformation layer is implemented.

The transformation process of the forward projection is as follows:

X _2d = G _2d (X _3d )

The transformation process of the rotation projection is as follows:

X _2d = G _R2d X _3d

where X _2d represents the 2D pose, X _3d represents the 3D pose, G _2d represents the depth residual network projection transformation, and G _R2d represents the rotation transformation.

where the rotation transformation matrix:

Through the combination of the above modules, two training modes, A and B, of the bidirectional projection network can be formed. According to the actual situation, the corresponding mode is selected to train the network, and the error function is continuously minimized by iteration. After 20-40 EPOCH network training, the final result can be obtained. A trained 3D pose generator.

The previously detected 2D pose is then normalized as follows:

1. Use the detected coordinates of the left and right shoulder joints to reconstruct the center neck coordinates:

Among them: (x _T , y _T ) represents the center neck coordinate, (x _ls , y _ls ) represents the left shoulder coordinate, (x _rs , y _rs ) represents the right shoulder coordinate;

2. Use the detected left and right shoulder and hip joints to reconstruct the central spine coordinates:

Where: (x _S , y _S ) represents the center spine coordinate, (x _ls , y _ls ) represents the left shoulder coordinate, (x _rs , y _rs ) represents the right shoulder coordinate, (x _lh , y _lh ) represents the left hip coordinate, ( x _rh , y _rh ) represents the coordinates of the right hip;

The normalized 2D human pose is sent to the trained 3D pose generator, and the 3D pose generator will output a 3D human skeleton conforming to the topology of the human pose according to the 2D detection results, and connect the skeleton sequence of each frame of video. The reconstruction of the three-dimensional human posture in the video can be realized. Because the estimation result of the two-dimensional posture detector is directly used in the method of the present invention, the model has strong generalization performance and can be used for some special postures in natural scenes. A reasonable 3D reconstruction effect can be obtained. The reconstruction effect of the invented method is shown in FIG. 5 .

3. References, the numbers in parentheses in the application documents refer to the literatures corresponding to the numbers below.

[1] H.-J.Lee and Z.Chen.Determination of 3d human body postures from asingle view.Computer Vision,Graphics,and Image Processing,30(2):148–168,1985.

[2] V. Ramakrishna, T. Kanade, and Y. Sheikh. Reconstructing 3d human pose from 2d image landmarks. In European Conference on Computer Vision (ECCV), pages 573–586. Springer, 2012.

[3]C.Ionescu,J.Carreira,and C.Sminchisescu.Iterated second-orderlabel sensitive pooling for 3d human pose estimation.In Conference onComputer Vision and Pattern Recognition(CVPR),pages 1661–1668,2014.2

[4] H. Jiang. 3d human pose reconstruction using millions of exemplars. In International Conference on Pattern Recognition (ICPR), pages 1674–1677. IEEE, 2010.

[5]C.-H.Chen and D.Ramanan.3D human pose estimation=2D poseestimation+matching.In Conference on Computer Vision and Pattern Recognition(CVPR),pages 5759–5767,2017.

[6]S.Li and A.B.Chan.3d human pose estimation from monocular images with deep convolutional neural network.In Asian Conference on Computer Vision(ACCV),pages 332–347.Springer,2014.

[7]D.Mehta,S.Sridhar,O.Sotnychenko,H.Rhodin,M.Shafiei,H.-P.Seidel,W.Xu,D.Casas,and C.Theobalt.Vnect:Real-time 3d human pose estimation with asingle rgb camera.volume 36,72017.

[8]B.Tekin,I.Katircioglu,M.Salzmann,V.Lepetit,and P.Fua.Structured prediction of 3d human pose with deep neural networks.In British MachineVision Conference(BMVC),2016.

[9] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-fine volumetric prediction for single-image 3d human pose. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1263–1272. IEEE , 2017.

[10]J.Martinez,R.Hossain,J.Romero,and J.J.Little.A simple yet effective baseline for 3d human pose estimation.In ICCV,2017.

[11]F.Moreno-Noguer.3d human pose estimation from a single image viadistance matrix regression.In Proceedings of the Conference on ComputerVision and Pattern Recognition(CVPR),2017.1

[12] Zhao L, Peng X, Tian Y, et al. Semantic Graph Convolutional Networks for 3D Human Pose Regression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019:3425-3435.

Claims (7)

1. a natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network is characterized in that: contain the following steps: Step 1. Use a camera to collect human motion video or image data in a natural scene; Step 2, sending the collected video and image data into a two-dimensional attitude detector to obtain the two-dimensional body joint point coordinates of the corresponding attitude; Step 3: Design bidirectional projection networks with two structures according to whether there are three-dimensional attitude data labels in the training process; Step 4. Use the deep adversarial learning strategy to train the designed network, minimize the network loss function, and finally obtain the trained 3D pose generator after iteration; Step 5: Input the output result of the 2D posture detector in Step 2 into the 3D posture generator trained in Step 4, and the output result is the 3D posture data of the person in the video/image. 2. the natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network according to claim 1, is characterized in that: in described step 1, adopts common monocular optical camera or mobile phone camera to complete the character movement data under natural scene. Acquisition, the data is in the form of pictures or videos. 3. the natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network according to claim 1, is characterized in that: in described step 2, two-dimensional posture detector is the two-dimensional posture detection method of OpenPose, StackHourglass or HRNet, When the collected data is a picture, directly input the picture to obtain the two-dimensional joint point detection result; when the collected data is a video, input the two-dimensional joint point detection sequence frame by frame. 4. the natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network according to claim 1, it is characterized in that: in described step 3, according to whether the user has three-dimensional posture label data, chooses two kinds of different structures of A/B. Bidirectional projection network, when there is 3D attitude data for use, the bidirectional projection network works in A mode. At this time, the network consists of two opposite dual branches. Its network modules include a 3D attitude generator, a 3D attitude discriminator, a 3D attitude projection layer and 2D attitude discriminator; when there is no 3D attitude data for use, the bidirectional projection network works in B mode. At this time, the network consists of two projection branches in different directions, and its network module includes a 3D attitude generator. , 2D pose projection layer and 2D pose discriminator. 5. the natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network according to claim 1, is characterized in that: the input of three-dimensional posture generator in described step 3 is two-dimensional joint point coordinates, and the output is three-dimensional joint point coordinates , which contains two deep residual networks and an attitude feature extraction layer. The deep residual network is composed of four residual blocks stacked, and the number of neurons in each layer is 1024. The attitude feature extraction layer completes the encoding and compression of the attitude topology. ; The 2D pose discriminator and the 3D pose discriminator have the same network architecture, which includes a 2D/3D pose feature extraction layer, a deep residual network and a fully connected layer. The input of the 2D/3D discriminator module is different dimensions The attitude vector of , and the output is a unary discriminant value; the two-dimensional attitude projection layer contains two branches: forward projection and rotation transformation of the residual network, and the attitude is projected to different observation angles according to the function. The input of the projection layer module is the three-dimensional attitude data, and the output is the projected two-dimensional attitude data. 6. The natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network according to claim 1, is characterized in that: described step 4 comprises the following sub-steps, Step 4.1. When there is 3D pose data for network training, select Mode A network architecture for training; Step 4.1.1. Taking the 2D pose as input, first output an initial depth estimation value through the residual network in the 3D pose generator to obtain the initial estimation result of the 3D pose; then the initial estimation result is passed to the pose feature extraction layer , a feature vector is output through the pose prior topology feature extraction, and the feature vector is passed into the depth residual network again to output the final depth estimation value to generate the final 3D reconstructed pose; Step 4.1.2. The generated 3D reconstructed pose gets forward projection through the 2D pose projection layer all the way, and calculates the pose error with the input 2D pose, and the other way will send it to the 3D pose discriminator to calculate the distribution error; Step 4.1.3, take the 3D pose as input, first get the forward projection through the 2D pose projection layer, the forward projection is sent to the 3D pose generator all the way to get the 3D reconstruction result, and calculate the pose error with the input 3D pose , the other is sent to the two-dimensional attitude discriminator to calculate the distribution error; Step 4.2. When there is no 3D pose data for network training, select Mode B network architecture for training; Step 4.2.1. Taking the 2D pose as input, first output an initial depth estimation value through the residual network in the 3D pose generator to obtain the initial estimation result of the 3D pose; then the initial estimation result is passed to the pose feature extraction layer , and output a feature vector after the pose prior topology feature extraction, and the feature vector will be passed to the depth residual network again to output the final depth estimation value to generate the final 3D reconstructed pose; Step 4.2.2. Introduce the 3D reconstructed pose to the 2D pose projection layer to obtain the forward projection and the rotation projection respectively. The forward projection will calculate the attitude error with the input 2D pose, and the rotation projection will be judged by the 2D pose The calculator calculates the two-dimensional distribution error; Step 4.3. Calculate the loss functions in the two modes of A/B, including the pose loss function and the distribution loss function; Step 4.3.1. In mode A, the overall loss function of the network is defined as: loss _A =L _GAN (G _3d ,D _3d )+L _GAN (G _2d ,D _2d )+L _dual (G _2d ,G _3d ), where L _GAN represents the loss function of a generative adversarial network with a gradient penalty term, It reflects the distribution error, and the calculation formula is as follows:

L _dual represents the bidirectional loss of the dual network, which reflects the attitude error. The calculation formula is as follows: L _dual (G _2d ,G _3d )=||G _2d (G _3d (X _2d ))-X _2d || ₁ +||G _3d (G _2d (X _3d ))-X _3d || ₁ λ is the neural network hyperparameter, G _3d represents the 3D pose generator, G _2d represents the 2D pose projection layer, D _3d and D _2d represent the 3D pose discriminator and 2D pose discriminator, respectively, X _2d and X _3d represent the real 2D pose and 3D pose, A _3d represents the random 3D pose on the line connecting the sampling points of the reconstructed 3D pose distribution and the real 3D pose distribution, A _2d represents the line connecting the sampling points of the projected 2D pose distribution and the real 2D pose distribution A random 2D pose on ; Step 4.3.2. In mode B, the overall loss function of the network is defined as: loss _B =L _GAN (G _R2d G _3d ,D _2d )+L _p o _se (G _K2d G _3d ), where L _GAN represents the loss function of the generative adversarial network with the gradient penalty term, which reflects the distribution error and calculates The formula is as follows:

L _pose is the reconstruction loss, which reflects the attitude error. The calculation formula is as follows: L _pose (G _K2d G _3d )=||G _K2d G _3d (X _2d )-X _2d || ₁ λ is the neural network hyperparameter, G _3d represents the 3D pose generator, G _R2d represents the rotation projection transformation of the 2D pose projection layer, G _K2d represents the forward projection transformation of the 2D pose projection layer, D _2d represents the 2D pose discriminator, X _2d represents the real 2D pose data, A _2d represents the random 2D pose on the line connecting the sampling points between the projected 2D pose distribution and the real 2D pose distribution; Step 4.4: Use the neural network optimizer to adjust the network parameters to minimize the error function, and after iterating for 20-40 EPOCH, the loss function converges, and the trained 3D pose generator is obtained. 7. The natural scene three-dimensional human body posture reconstruction method based on bidirectional projection network according to claim 1, is characterized in that: described step 5 comprises the following sub-steps, Step 5.1. Pass the video or image data collected by the ordinary camera into the two-dimensional attitude detector, and first obtain the two-dimensional joint point data; Step 5.2, normalize the output result of the two-dimensional attitude detector, so that it can be directly used as the input of the three-dimensional attitude generator; the normalization processing has the following steps: Step 5.2.1. Use the detected coordinates of the left and right shoulder joints to reconstruct the center neck coordinates:

Among them: (x _T , y _T ) represents the center neck coordinate, (x _ls , y _ls ) represents the left shoulder coordinate, (x _rs , y _rs ) represents the right shoulder coordinate; Step 5.2.2. Use the detected left and right shoulder and hip joints to reconstruct the central spine coordinates:

Where: (x _S , y _S ) represents the center spine coordinate, (x _ls , y _ls ) represents the left shoulder coordinate, (x _rs , y _rs ) represents the right shoulder coordinate, (x _lh , y _lh ) represents the left hip coordinate, ( x _rh , y _rh ) represents the coordinates of the right hip; Step 5.3. Pass the normalized 2D pose data into the 3D pose generator, and the output result is the reconstructed 3D pose. When the input data is image data, the output result is the 3D human pose skeleton; when the input data is video data , and the output result is the 3D human skeleton action.

Images