CN111339870A - A Human Shape and Pose Estimation Method for Object Occlusion Scenarios - Google Patents

Abstract

The invention discloses a human body shape and attitude estimation method for object occlusion scenes. The calculated weak perspective projection parameters are converted into camera coordinates to obtain a UV image containing human body shape information without occlusion; The image is occluded by random object pictures, and the human body mask under occlusion is obtained; the obtained virtual occlusion data is used to train the UV map repair network of the encoding-decoding structure; the color image of the human body occluded by the real object is input, and the mask image is used as the ground truth to construct Saliency detection network of encoding-decoding structure; use latent space features obtained from encoding to supervise training of human encoding network; input color image of occluded human body to obtain a complete UV image; use the corresponding relationship between UV image and vertices of human 3D model to recover the occlusion situation 3D model of the human body under. The invention converts the estimation of the shape of the occluded human body into the image repairing problem of the two-dimensional UV map, thereby realizing the real-time and dynamic reconstruction of the human body in the occlusion scene.

Description

A Human Shape and Pose Estimation Method for Object Occlusion Scenarios

technical field

The invention belongs to the field of computer vision and three-dimensional vision, and in particular relates to a method for estimating human body shape and posture under the scene of object occlusion.

Background technique

Estimating the shape and pose of a 3D human body from a single image is a research hotspot in the field of 3D vision in recent years. It plays an important role in the application of virtual reality technologies such as human motion capture, virtual fitting and human animation. In recent years, deep learning technology has simplified the solution method of recovering the overall shape of the human body from a single image, especially after the SMPL model was proposed and widely used, the 3D human body shape and pose estimation from monocular images has undergone multiple stages of vigorous development , including (1) optimizing and solving the SMPL parameters by matching two-dimensional visual features; (2) directly regressing the SMPL parameters with a convolutional neural network (CNN); (3) using the two-dimensional UV map to represent the three-dimensional points of the SMPL surface, and then the 3D human shape estimation into a CNN-based image translation problem. Deep neural networks have become the mainstream method for 3D human shape estimation due to their accuracy and operational efficiency, and they can obtain better reconstruction results in specific scenarios. However, most of the existing methods do not consider the common phenomenon of occlusion between people and objects. Without explicit consideration of occlusion, such methods cannot be directly transferred to deal with human estimation in occluded scenarios. This makes them very sensitive to scenes with occlusion or even slight object occlusion, and it is difficult to meet real-world needs.

The three-dimensional shape and pose estimation of the human body in occlusion scenes has always been a difficult problem in the field. The main reasons are: (1) Object occlusion will introduce serious ambiguity in network training, and lead to a significant reduction in the image features that can be directly used. , thereby affecting the complete 3D human shape estimation effect. (2) Due to the ubiquity and randomness of occluded objects, it is difficult for the network to accurately segment the pixels where the human body and the occluded objects are located in the image, resulting in disturbed reconstruction results.

SUMMARY OF THE INVENTION

Purpose of the invention: Aiming at the problem of human body shape and attitude estimation in occlusion scenes, the present invention proposes a human body shape and attitude estimation method for object occlusion scenes, which converts the occluded human body shape estimation into the image restoration problem of two-dimensional UV maps, and then realizes the Real-time, dynamic reconstruction of the human body in occlusion scenes.

Technical solution: A method for estimating human body shape and posture for object occlusion scenes described in the present invention includes the following steps:

(1) In the data preparation stage, the weak perspective projection parameters are calculated by using the correspondence between the three-dimensional joint points of the human body and the two-dimensional joint points of the three-dimensional human data set;

(2) According to the calculated weak perspective projection parameters, the three-dimensional model of the human body is converted into the camera coordinates through three-dimensional rotation and translation;

(3) Normalize the x, y, and z coordinate values of the vertices of the 3D model of the human body under the camera coordinates to the range of -0.5 to 0.5 and store them in the R, G, and B channels of the UV map to obtain the situation without occlusion UV map containing the shape information of the human body;

(4) Add random object picture occlusion to the two-dimensional image of the human body, and obtain the human body mask under the occlusion condition;

(5) Repeat step (3), the 3D point that falls outside the mask area after using weak perspective projection is the 3D point under visual occlusion, and its x, y, and z coordinates are fixed to -0.5, and the corresponding 3D point under occlusion is obtained. UV map;

(6) In the training phase, the UV map repair network of the encoding-decoding structure is trained based on the virtual occlusion data obtained in steps (1) to (5); the repair network is based on the L1 loss between the complete human body UV map, The Laplacian smoothing term between adjacent pixels and the consistency of UV connections are used as constraints;

(7) The saliency detection network of encoding-decoding structure is constructed by using the real object occluded human color image as input, and the mask image as the ground truth;

(8) Connect the color image of the occluded human body with the saliency map and send it to the human body encoding network. At the same time, encode the UV map under the corresponding occlusion using the repair network trained in step (6), and use the encoded latent space features to supervise the human body. coding network training;

(9) In the test phase, input the color image of the occluded human body, and through the saliency detection network and the human body encoding network, decode the latent space eigenvalues encoded by the human body encoding network using the decoder of the repair network to obtain a complete UV image;

(10) Using the UV map and the corresponding relationship between the vertices of the three-dimensional model of the human body to restore the three-dimensional model of the human body under occlusion.

Further, the UV map inpainting network described in step (6) uses ResNet as the encoder and the stacked deconvolution layers as the decoder.

Further, described step (6) is realized by following formula:

L=L ₁ +λL _tv + μL _p

Among them, λ and μ are the weights, L _tv is the Laplace smoothing term, and L _p is the consistency constraint at the UV connection:

Among them, V _b is the model vertex point set corresponding to multiple UV pixels, and P(v) is the UV pixel value corresponding to the model vertex v.

Further, the human coding network described in step (8) uses the VGG-19 structure.

Further, the color image in step (9) is a preprocessed human occlusion image obtained from a monocular color camera.

Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are: 1. Use a large amount of virtual occlusion data to train the image repair network, so that the overall framework has better robustness to various occlusions; 2. Use saliency detection , reduce the interference of invalid image features such as occlusion and background on the reconstruction, enhance the robustness to the human body and occlusion edges in the image, and avoid the problem of inaccurate segmentation; 3. Use the method of latent space consistency to convert the three-dimensional shape of the human body. The estimation is transformed into an image inpainting problem, which reduces the complexity of the solution; 4. A UV connection consistency constraint is proposed to improve the smoothness of the reconstruction results in the method of human reconstruction with UV maps.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Fig. 2 is a schematic diagram of human body information UV map generation;

Fig. 3 is the UV map of human body shape information;

4 is a schematic diagram of a three-dimensional model of a human body;

Fig. 5 is a network structure diagram of saliency detection;

FIG. 6 is a schematic diagram of a reconstruction result of the present invention.

Detailed ways

The present invention will be described in further detail below with reference to the accompanying drawings. As shown in FIG. 1 , the implementation process of a method for estimating human body shape and posture for object occlusion scenes according to the present invention is as follows:

As shown in Figure 2, the generation method of the human body information UV map is as follows: in the data preparation stage, first use the projection relationship between the joint points of the 3D model of the human body and the 2D joint points in the 3D human data set to calculate the weak perspective projection parameters, and then use the 3D translation to calculate the weak perspective projection parameters. , rotation and other operations to convert the human body model to the camera coordinates, normalize the x, y, and z coordinates of the vertices of the human body 3D model under the camera coordinates to [-0.5, 0.5] and store them in the R, G, B three of the UV map channel, thereby obtaining the UV map containing the human body shape information without occlusion as shown in Figure 3. In order to obtain the UV map of the occluded human body, a random object image is added to the 2D image of the human body, and the human body mask under the occlusion condition is obtained. Weak perspective projection is performed on the human body 3D model to the human body mask through the projection parameters. The three-dimensional points that fall outside the mask area are the three-dimensional points under the visual occlusion, and their x, y, and z coordinates are fixed to -0.5, and the three-dimensional coordinates of the vertices are still stored in the mask area, so as to obtain as shown in Figure 4. The UV map under the corresponding occlusion shown. Since both the occlusion UV map and the complete UV map in this step have nothing to do with the background of the color image, virtual occlusion can be used to generate a large amount of occlusion UV data to enhance the robustness of the network.

Using the obtained large number of occluded UV maps and full UV maps, an image inpainting network with ResNet-50 as the encoder and stacked deconvolutional layers as the decoder is trained. The network is able to encode occlusion UV maps into high-dimensional human features, and decode the complete human shape UV maps from high-dimensional features. The network is constrained by the L1 loss with the full body UV map, the Laplacian smoothing term between adjacent pixels, and the consistency of UV connections.

Its specific formula is:

L=L ₁ +λL _tv + μL _p

Among them, λ and μ are the weights, L _tv is the Laplace smoothing term, and L _p is the consistency constraint at the UV connection:

Among them, V _b is the model vertex point set corresponding to multiple UV pixels, and P(v) is the UV pixel value corresponding to the model vertex v. This constraint enables the parts of the UV map as shown in Figure 3 to be smoothly connected.

The saliency detection network of the encoding-decoding structure is constructed with the real object occluding the color image of the human body as the input and the mask image as the ground truth. After the saliency map detection network shown in Figure 5, the human saliency map of the occluded image is obtained. . The color image of the occluded human body is connected with the saliency map and sent to the human encoding network. At the same time, the UV map under the corresponding occlusion is encoded by the trained repair network, and the encoded latent space features are used to supervise the training of the human encoding network. Here, input the human coding network with VGG-19 as the basic structure. Using the occlusion UV map corresponding to the color image, the high-dimensional features obtained by the encoder of the image inpainting network are used as the supervision of the human encoding network. At the same time, as shown in Figure 5, the two networks are trained end-to-end with different scaled human masks as the supervision of the saliency network.

After the network training is completed, the human occlusion images are directly obtained from the monocular color camera and preprocessed such as cropping and scaling. The preprocessed color images are input into the network, directly passed through the saliency detection network, and the human body encoding network to obtain high-dimensional human body features. The latent space feature value encoded by the human body encoding network is decoded by the decoder of the inpainting network to obtain high-dimensional features, and then the decoder of the image inpainting network is used to decode the complete UV image. Through the correspondence between the UV map and the 3D model of the human body, the 3D model of the human body with the corresponding shape can be directly recovered from the UV map of the human body shape. Figure 6 shows the reconstruction results of the occluded human color image through this method.

Claims (5)

1. a human body shape and attitude estimation method for object occlusion scene, is characterized in that, comprises the following steps: (1) In the data preparation stage, the weak perspective projection parameters are calculated by using the correspondence between the three-dimensional joint points of the human body and the two-dimensional joint points of the three-dimensional human data set; (2) According to the calculated weak perspective projection parameters, the three-dimensional model of the human body is converted into the camera coordinates through three-dimensional rotation and translation; (3) Normalize the x, y, and z coordinate values of the vertices of the 3D model of the human body under the camera coordinates to the range of -0.5 to 0.5 and store them in the R, G, and B channels of the UV map to obtain the situation without occlusion UV map containing the shape information of the human body; (4) Add random object picture occlusion to the two-dimensional image of the human body, and obtain the human body mask under the occlusion condition; (5) Repeat step (3), the 3D point that falls outside the mask area after using weak perspective projection is the 3D point under visual occlusion, and its x, y, and z coordinates are fixed to -0.5, and the corresponding 3D point under occlusion is obtained. UV map; (6) In the training phase, the UV map repair network of the encoding-decoding structure is trained based on the virtual occlusion data obtained in steps (1) to (5); the repair network is based on the L1 loss between the complete human body UV map, The Laplacian smoothing term between adjacent pixels and the consistency of UV connections are used as constraints; (7) The saliency detection network of encoding-decoding structure is constructed by using the real object occluded human color image as input, and the mask image as the ground truth; (8) Connect the color image of the occluded human body with the saliency map and send it to the human body encoding network. At the same time, encode the UV map under the corresponding occlusion using the repair network trained in step (6), and use the encoded latent space features to supervise the human body. coding network training; (9) In the test phase, input the color image of the occluded human body, and through the saliency detection network and the human body encoding network, decode the latent space eigenvalues encoded by the human body encoding network using the decoder of the repair network to obtain a complete UV image; (10) Using the UV map and the corresponding relationship between the vertices of the three-dimensional model of the human body to restore the three-dimensional model of the human body under occlusion. 2. a specific kind of human body shape and attitude estimation method for object occlusion scene according to claim 1, is characterized in that, the UV map repairing network described in step (6) uses ResNet as encoder, with stacked reverse Convolutional layers act as decoders. 3. a specific kind of human body shape and attitude estimation method for object occlusion scene according to claim 1, is characterized in that, described step (6) is realized by following formula: L=L ₁ +λL _tv + μL _p Among them, λ and μ are the weights, L _tv is the Laplace smoothing term, and L _p is the consistency constraint at the UV connection:

Among them, V _b is the model vertex point set corresponding to multiple UV pixels, and P(v) is the UV pixel value corresponding to the model vertex v. 4 . A specific method for estimating human body shape and posture for an object occlusion scene according to claim 1 , wherein the human coding network described in step (8) uses a VGG-19 structure. 5 . 5. a specific kind of human body shape and attitude estimation method for object occlusion scene according to claim 1, is characterized in that, the color image described in step (9) is the preprocessed obtained from monocular color camera Human occlusion image.

Images