CN111476901B

CN111476901B - Three-dimensional human body shape representation method

Info

Publication number: CN111476901B
Application number: CN202010279674.5A
Authority: CN
Inventors: 张举勇; 江博艺
Original assignee: University of Science and Technology of China USTC
Current assignee: University of Science and Technology of China USTC
Priority date: 2020-04-10
Filing date: 2020-04-10
Publication date: 2023-04-07
Anticipated expiration: 2040-04-10
Also published as: CN111476901A

Abstract

The invention discloses a three-dimensional human body shape representation method. On the one hand, by processing the collected human body grid data set, a large amount of labeled training data can be obtained, which increases the robustness of the model; on the other hand, by using Deformation representation, and defines a deformation representation based on the approximate rigid block of the human body, which can have higher accuracy and robustness than directly using the Euclidean distance. Meanwhile, for the hinged deformation of the human body, a hierarchical reconstruction network is designed, which makes full use of the shape prior of the human body and improves the accuracy of the model.

Description

Three-dimensional human body shape representation method

Technical Field

The invention relates to the technical field of human body three-dimensional reconstruction, in particular to a three-dimensional human body shape representation method.

Background

The parameterized human body model has wide application in the fields of computer graphics and computer vision, including three-dimensional human body tracking, three-dimensional human body reconstruction and attitude estimation. Since the human body has various attributes including gender, race, posture, body type, and the like, which contain abundant geometric deformation, high-precision reconstruction of the human body is a challenging task. In recent years, with the development of deep learning, geometric reconstruction using a neural network has become a trend.

In the past, human reconstruction was primarily based on the skeletal skinning method. The method expresses the identity deformation space of a human body by a group of simple linear basis deformations, and then deforms the human body to a specific posture by using a skeleton. The method is simple and high in efficiency in calculation, decoupling of the identity and the posture is achieved, high-frequency change of a human identity deformation space is ignored, and geometric reconstruction accuracy is limited. In addition, motion has a relative motion representation of skeletal joint points, without encoding a prior distribution of human motion, and thus may produce an abnormal human model. The other main method is based on deformation expression, typically, a three-dimensional point coordinate is replaced by triangular deformation, and decoupling parametric modeling of a human body is realized by decomposing the deformation into parts related to various attributes such as identity, posture and the like. The method has higher geometric reconstruction accuracy, but the calculation process is more complex, the parameter quantity is large, and the speed is slower.

Recently, learning some shapes like human faces using neural networks has been developed. Through end-to-end training, the network maps the shape manifold to a low-dimensional nonlinear space, which shows higher precision. But the human body has significant large-scale deformation caused by different poses compared to the human face. A network architecture which achieves good accuracy in face modeling is directly applied to a human body grid, and a good result cannot be obtained. This is because the european coordinate is not easy to maintain the original deformation mode for large-scale deformation, and is easy to cause distortion. On the other hand, the network architecture is not designed for a special skeleton hinge structure of a human body, and the precision is not improved by using the prior of the shape of the human body.

Disclosure of Invention

The invention aims to provide a three-dimensional human body shape representation method which can improve the reconstruction precision of a three-dimensional human body model.

The purpose of the invention is realized by the following technical scheme:

a three-dimensional human body shape representation method comprising:

preprocessing the collected human body grid data set, deforming based on a standard posture, and calculating ACAP deformation representing and describing human body deformation characteristics of rigid block deformation to form a training data set;

constructing an encoder network and a hierarchical reconstruction network to form an end-to-end network structure, and training the network structure by using a training data set; in the training process, the ACAP deformation expression is coded through a coder network to obtain an identity attribute and an action attribute, a three-dimensional human body model is reconstructed through the reconstruction network by utilizing the identity attribute and the action attribute, and the coder network and a hierarchical reconstruction network are trained by utilizing an error between a reconstruction result and input training data;

and after the training is finished, inputting the identity attribute and the action attribute into a trained hierarchical reconstruction network to obtain a three-dimensional human body model reconstruction result.

According to the technical scheme provided by the invention, 1) a large amount of labeled training data can be obtained by processing the acquired human body grid data set, so that the robustness of the model is improved. 2) By using the deformation representation and defining a deformation representation based on human body approximate rigid blocks, higher precision and robustness can be achieved compared with the mode that Euclidean distance is directly used. 3) Aiming at the hinge type deformation of the human body, a hierarchical reconstruction network is designed, the shape prior of the human body is fully utilized, and the precision of the model is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.

Fig. 1 is a flowchart of a method for representing a three-dimensional human body shape according to an embodiment of the present invention;

FIG. 2 is a reference body grid for calculating ACAP (consistent deformation) features and rigid blocks defined by calculating large-scale features g of a body according to an embodiment of the present invention;

fig. 3 is a schematic visualization diagram of reconstruction of each phase of a neutral posture and an original posture generated by a reconstruction network according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.

In the field of human body parametric representation, the traditional skeleton skin-based method has high model speed and strong action expression capability, but the reconstruction precision is limited because simple linear dimension reduction is only carried out on the deformation space of the identity; on the other hand, the representation of the motion is directly based on the relative motion of the joint points and does not limit the rationality of the human motion. To this end, an embodiment of the present invention provides a method for representing a three-dimensional human body shape, as shown in fig. 1, the method mainly includes:

and 11, preprocessing the collected human body grid data set, deforming based on the standard posture, and calculating the ACAP deformation expression and the human body deformation characteristics describing the deformation of the rigid blocks to form a training data set.

The preferred embodiment of this step is as follows:

1) And carrying out standardization processing on the collected human body grid data set to obtain human body grid data with unified topology, and deforming to obtain a neutral human body grid corresponding to each human body grid data through the defined standard posture.

In the embodiment of the invention, the original human body grid data can be obtained from the network, including SCAPE, FAUST, dyna, MANO and the like. For collected human mesh data sets of different sources, the mesh representations of the collected human mesh data sets are inconsistent and need to be converted into a standard topology G = { V, E }, where V is a vertex set and E is an edge set. In this embodiment, a source data is used as a standard topology (e.g., SCAPE); calculating corresponding ACAP (consistent deformation) deformation representation by using action grids in standard topology (for example, seventy action grids in SCAPE), obtaining a group of priori deformation representation bases C of human body actions, and recovering the deformation representation of a human body by using a group of parameters w; and (3) using the linear space of Cw as a priori space of human body deformation, then optimizing a group of vertex coordinates p and rigid transformation parameters of standard topology, namely a rotation parameter R and a translation parameter t, and standardizing human body grid data sets from different sources.

The normalization process is to solve the following optimization problem:

wherein λ is ₁ 、λ ₂ 、λ ₃ Are all set weights; | w | charging ₁ Is a sparse regularization constraint on parameter w;

E _prior is a human body deformation prior term determined by a prior deformation representation base C, so that the optimized grid vertex conforms to the human body shape as much as possible, and is represented as follows:

wherein, T _i (w) is the prior deformation of a neighborhood of the ith vertex in the standard topology, the prior deformation represents that the base C is multiplied by w to obtain an ACAP characteristic, and then the ith vertex component of the ACAP characteristic is converted to obtain T _i (w)；q _i Is the position of the ith vertex in the deformed reference grid under the standard topology; relative to the deformation reference grid, the position of the ith vertex of the grid to be optimized under the standard topology is p _i (ii) a N (i) refers to a neighborhood vertex index set of the ith vertex under the standard topology, j refers to the jth vertex in the N (i), and the positions of the corresponding vertices in the deformation reference grid and the grid to be optimized are respectively represented as q _j 、p _j ；c _ij Is an edge weight value calculated on the deformation reference grid, called cotangent weight, and is specifically q _j And q is _i The edge weight value of (2). In the embodiment of the invention, the deformation reference grid is a human body grid required for calculating ACAP deformation expression, and can be selected according to actual conditions or experience; as shown in fig. 2, the pre-selected warped reference grid is on the left. Those skilled in the art will appreciate that the deformed reference mesh corresponds one-to-one to all vertices of the standard topology, except for the different locations of the vertices; in the embodiment of the invention, the deformation reference gridFor reference, optimizing a grid under a standard topology; that is, q _i Is stationary, p _i Is optimized and the value is changed.

E _icp Is a point-to-plane registration energy term with the nearest neighbor of the target mesh such that the optimization result is close to the target, expressed as:

d is an index set of the corresponding point pair selected by dynamic calculation under the standard topology; v. of _l(i) Is and p _i The corresponding point on the corresponding target grid,

representing point v _l(i) Normal direction of (2); the target grid refers to human body grid data sets of different sources to be optimized;

E _lan the L2 loss energy terms of a group of manually marked sparse corresponding points of the standard topology and the target grid are expressed as follows to reduce sliding errors and avoid local minimum values:

where L is a pre-selected set of corresponding points in a standard topology, the above equation limits the spatial location of the points in the pre-selected set of corresponding points L to be as small as possible.

In the embodiment of the invention, the human body grid data fitted based on the mode has consistent standard topology, and has tolerable registration error with the original data.

The human body model data constructed on the basis also needs to define the corresponding neutral human body grids. Illustratively, a consistent a-gesture in the SPRING dataset may be used as the neutral gesture. And for all the human body data of each identity, selecting a human body with the minimum error with the SPRING average grid as a candidate human body grid, and then converting the candidate human body grid into the posture A by using an ARAP (as rigid as possible) deformation method to be used as a neutral human body grid of the human body of the identity.

2) Calculating ACAP deformation representation of each human body grid data and corresponding neutral human body grid, and recording as f and f _s (ii) a And respectively calculating human body deformation characteristics g and g of the human body grid and the corresponding neutral human body grid describing rigid block deformation according to the hinge type deformation characteristics of the human body based on the skeleton _s To obtain a set of training data { f, f } _s ,g,g _s And correspondingly processing the collected data to obtain a training data set.

By the method, the human body grid data with the uniform topology are obtained, and each identity human body has a neutral posture grid. In the embodiment of the invention, the deformation expression of the geometric shape of the ACAP is used for replacing the original Euclidean coordinate shape expression to enhance the modeling precision of large-scale deformation and obtain better performance compared with a general linear model.

The formula for calculating the human body grid data and the ACAP deformation expression of the corresponding neutral human body grid is as follows:

similarly, i represents a vertex in the standard topology, and the above is standardized, so the standard topology can be any standard topology after being standardized or a previously selected standard topology. The other parameters have the same meanings as above and are not described in detail.

T in the above formula _i Refers to the affine transformation matrix of the ith vertex in the standard topology, the affine transformation matrix T _i The method comprises the steps of transforming a local umbrella-shaped structure of a neighborhood of a deformation reference grid into various deformation information of a structure of a computational grid; by polar decomposition, T _i Decomposition to rigid R _i And a non-rigid part S _i (determined by 3 and 6 degrees of freedom, respectively), after disambiguation of the rigid deformation part of each vertex, the ACAP deformation of the computational mesh is obtainedRepresenting; the dimensions of the representation are 9 times the number of vertices, each vertex having 3 and 6 parametric record rigid and non-rigid deformations, respectively.

Taking each human body grid data and corresponding neutral human body grid as calculation grid to be substituted into the formula to obtain corresponding ACAP deformation expressions f and f _s 。

In addition, some approximately rigid parts, such as the lower arm, the head, etc., are defined on the human body in consideration of the skeleton-based hinged deformation characteristics of the human body. On the basis, a deformation characteristic g of a large scale of a human body for describing the deformation of the rigid block is defined, and the calculation formula is as follows:

wherein v is _k Is the set of vertices of the kth rigid block, q _i' 、p _i' Respectively representing the positions of the ith' vertex on the kth rigid block in the deformation reference grid and the calculation grid;

and &>

Respectively averaging the k-th rigid block on the deformation reference grid and the calculation grid; by means of radiation deformation of a rigid block>

And performing polar decomposition and parameterization to obtain the human body deformation characteristic, wherein the dimension of the human body deformation characteristic is 9 times of the number of approximate rigid blocks of the human body.

Similarly, each human body grid data and corresponding neutral human body grid are taken as calculation grids and are substituted into the formula to obtain deformation characteristics g and g _s 。

As shown in fig. 2, the rigid blocks defined for calculating the deformation reference grid (left side of fig. 2) and the large-scale features of the human body (right side of fig. 2) are represented for ACAP deformation.

Step 12, constructing an encoder network and a hierarchical reconstruction network to form an end-to-end network structure, and training the network structure by using a training data set; in the training process, the ACAP deformation expression is coded through a coder network to obtain identity attributes and action attributes, the identity attributes and the action attributes are utilized to reconstruct a three-dimensional human body model through a reconstruction network, and errors between a reconstruction result and input training data are utilized to train the coder network and the hierarchical reconstruction network.

In the embodiment of the invention, the encoder network can adopt a standard variational self-encoder structure to learn the identity attribute e from the ACAP deformation expression f of the human body grid data _s And attitude attribute e _p Then, an end-to-end network structure is formed by combining the reconstructed network for training. Training data { f, f _s ，g，g _s Only f is used as the input of the network and the rest of the data is used to calculate the reconstruction error.

In the embodiment of the invention, the hierarchical reconstruction network is based on human body geometric prior and comprises two parts, wherein the first part is used for hierarchically reconstructing a main part of a three-dimensional human body model, the second part is used for reconstructing a difference part, and the reconstructed main part and the reconstructed difference part are added to obtain a reconstruction result; the reconstructed network is described as:

wherein,

representation utilization identity attribute e _s And attitude attribute e _p In conjunction with the reconstruction result of (a), based on the result of (b)>

Representation utilization identity attribute e _s The reconstruction result of (2);

Indicates a reconstruction result->

The main part b in (1), W represents a skin layer;

Represents the result of a reconstruction>

The difference part d in (1);

Indicates a reconstruction result->

Main part b of (1) _s ，

Representing a skin layer;

represents the result of a reconstruction>

The difference part d in _s ；

Reconstruction of the main parts b and b _s The following equation is used to reconstruct the human deformation characteristics

And &>

Wherein,

are independent mapping transformations, which may be modeled, for example, using a multi-tier perceptron.

Then, the skin layer is utilized

Reconstruction of the main parts b and b _s ：

Wherein the skin layer

In the form of a matrix, is selected>

Represents->

The xth row and the y column; y is the number of rigid blocks, e.g., Y =16; the above principle is that the deformation of each vertex is obtained by linear convex combination of the relative rigid block deformation of the vertex, and the coefficient of the convex combination is determined by learning training and is not set manually.

Difference parts d and d _s Expressed as:

wherein,

independent mapping transformations, illustratively, these mapping transformations may be modeled using a multi-tier perceptron.

As shown in fig. 3, an example of the visualization of the various computed portions of the reconstructed network over one reconstructed instance is given. B (e) in FIG. 3 _s ，e _p )、B(e _s 0) i.e. main parts b and b as mentioned herein _s 。

In the embodiment of the invention, the network can be trained end to end through the training data set, and after the training is finished, the decoupled low-dimensional hidden layer can be used for representing e _s ，e _p The method can be used for reconstructing a human body (or applied to aspects of human body editing, motion migration and the like), can also be applied to aspects of human body editing, motion migration and the like, and has wide application prospects in the fields of video live broadcast, virtual fitting, somatosensory games and the like.

During training, L1 mode loss between reconstruction and input features and distribution regularization loss of hidden layer variables can be adopted as loss functions.

The L1 mode loss can be expressed as:

in the above-mentioned formula, the compound has the following structure,

all are the reconstruction results of the relevant data in the training data, and the specific values 9 (number of deformation features) and 16 (number of rigid blocks) involved in the above formula) Are by way of example only and are not limiting.

The regular partial loss of the hidden layer parameter distribution is:

E _sKL ＝D _KL (q(e _s |f)||p(e _s ))

E _pKL ＝D _KL (q(e _p |f)||p(e _p ))

the two losses are KL divergence losses of which the distribution of the standard constraint hidden layer in the variational self-encoder meets the prior distribution.

And step 13, after training, inputting the identity attribute and the action attribute into a trained hierarchical reconstruction network to obtain a three-dimensional human body model reconstruction result.

After the network training is completed through the step 12, the encoder network can be abandoned, and the reconstructed network is directly used as a decoupled low-dimensional human body parameterized model. The model reconstructs a complete human body grid from two groups of decoupling parameters respectively representing identity and posture. Specifically, the trained hierarchical reconstruction network may reconstruct the identity attribute and the posture attribute of the input data according to the method introduced in the training phase

And/or>

Both are represented by ACAP deformation, and the corresponding human body grid and the neutral human body grid can be obtained through a simple conversion. The conversion method referred to herein can be referred to in the prior art, and is not described in detail.

Compared with the traditional human body parameterized model representation method, the scheme of the embodiment of the invention mainly has the following advantages:

1) The characteristics of input and output are represented by nonlinear deformation, the traditional Euclidean coordinates are replaced, the precision is higher, and the deformation with large scale is more robust.

2) The reconstruction accuracy of the model is further improved by utilizing the strong fitting capability of the neural network and combining the framework design of human body deformation prior.

3) The obtained posture hidden layer representation has certain semantics by utilizing the learning of a large number of human body models with various postures, namely embedding reasonable actions of the human body into a low-dimensional space. However, the posture parameters of the conventional model often have no semantics and may generate unreasonable human body actions.

Through the description of the above embodiments, it is clear to those skilled in the art that the above embodiments may be implemented by software, or by software plus a necessary general hardware platform. With this understanding, the technical solutions of the embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.), and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods according to the embodiments of the present invention.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for representing a three-dimensional human body shape, characterized in that it includes:

The collected human mesh dataset is preprocessed and deformed based on standard poses. Then, the ACAP deformation representation and the human deformation features describing the deformation of rigid blocks are calculated to form the training dataset.

An encoder network and a hierarchical reconstruction network are constructed to form an end-to-end network structure, and the network structure is trained using a training dataset. During the training process, the encoder network encodes the ACAP deformation representation to obtain identity attributes and action attributes. The reconstruction network then uses the identity attributes and action attributes to reconstruct a 3D human body model, and uses the error between the reconstruction result and the input training data to train the encoder network and the hierarchical reconstruction network.

After training, the identity attributes and action attributes are input into the trained hierarchical reconstruction network to obtain the 3D human body model reconstruction result.

The preprocessing of the collected human mesh dataset to obtain the training dataset consisting of ACAP deformation representation and human deformation features includes:

First, the collected human mesh dataset is standardized to obtain human mesh data with a unified topology. Then, the neutral human mesh corresponding to each human mesh data is obtained by deformation through the defined standard pose.

Then, calculate the ACAP deformation representation of each human body mesh and the corresponding neutral human body mesh, denoted as f and _fs ; and, based on the hinge-like deformation characteristics of the human body based on the skeleton, calculate the human body deformation features g and _gs describing the rigid block deformation of the human body mesh and its corresponding neutral human body mesh, thereby obtaining a set of training data {f, _fs , g, _gs }. The collected data is then processed accordingly to obtain the training dataset.

The formula for calculating the human body deformation characteristic g, which describes the deformation of a rigid block, is:

Where _{v_k} is the set of vertices of the k-th rigid block, and _{q_i'} and _{p_i'} represent the i'-th vertex on the k-th rigid block of the deformable reference mesh and the computational mesh, respectively;

and

These are the average values of the k-th rigid block on the deformation reference mesh and the computational mesh, respectively; through radial deformation of the rigid block...

Polar decomposition and parameterization are performed to obtain the human body deformation characteristics;

Substituting each human body mesh data and its corresponding neutral human body mesh as the computational mesh into the above formula yields the deformation characteristics g and _gs ;

When training the network structure using the training dataset, the L1 modulus loss between the reconstruction and input features, as well as the distribution regularization loss of the hidden layer variables, are used as loss functions.

The hierarchical reconstruction network is a hierarchical reconstruction network based on human geometric priors, consisting of two parts: the first part reconstructs the main parts of the 3D human model hierarchically, and the second part reconstructs the differential parts. The reconstruction result is obtained by adding the reconstructed main parts and the reconstructed differential parts. The reconstruction network is described as follows:

in,

This represents the reconstruction result using the identity attribute _es and the pose attribute _ep .

This represents the reconstruction result using the identity attribute e _s ;

Indicates the reconstruction result

The main part b,

Indicates the skin layer;

Indicates the reconstruction result

The difference in part d;

Indicates the reconstruction result

The main part of it is b _s .

Indicates the skin layer;

Indicates the reconstruction result

The difference in _ds ;

When reconstructing the main parts b and _bs , the following formula is used to reconstruct the human body deformation features.

and

in,

It is an independent mapping transformation;

Then, using the skin layer

Reconstructing the main parts b and b _s :

Among them, the skin layer

In matrix form,

express

The element in the x-th row and y-th column; Y is the number of rigid blocks;

The difference between d and _ds is represented as follows:

in,

Independent mapping transformations.

2. The method for representing a three-dimensional human body shape according to claim 1, characterized in that the standardization process of the collected human body mesh dataset includes:

For human body mesh datasets collected from different sources, a standard topology is used based on data from one source. Using the action mesh in the standard topology, the corresponding ACAP deformation representation is calculated to obtain a set of prior deformation representation basis C for human actions. Then, a set of parameters w is used to recover the deformation representation of a human body. The linear space of Cw is used as the prior space for human body deformation. Then, the vertex coordinates p and rigid transformation parameters of a set of standard topologies, namely rotation parameters R and translation parameters t, are optimized to standardize human body mesh datasets from different sources.

3. The method for representing a three-dimensional human body shape according to claim 2, characterized in that the standardization process involves solving the following optimization problem:

Where _λ1 , _λ2 , and _λ3 are all set weights; ||w|| ₁ is a sparse regularization constraint on the parameter w;

E _prior is the a priori term of human body deformation determined by the a priori deformation representation basis C, expressed as:

Where _Ti (w) is the prior deformation of a neighborhood of the i-th vertex under the standard topology, q _i is the position of the i-th vertex in the deformed reference grid under the standard topology, and the deformed reference grid is pre-selected; relative to the deformed reference grid, the position of the i-th vertex in the grid to be optimized under the standard topology is p _i ; N(i) refers to the set of neighboring vertex indices of the i-th vertex under the standard topology, j refers to the j-th vertex in N(i), and the positions of the corresponding vertices in the deformed reference grid and the grid to be optimized are represented as q _j and pj _ , respectively; c _{ij} is an edge weight value calculated on the deformed reference grid;

E _{icp} is the registered energy term from the nearest neighbor point of the target grid to the plane, expressed as:

Where D is the set of indices for the corresponding points selected in the dynamic calculation under the standard topology; v _{l(i)} is the corresponding point on the target mesh corresponding to p_i .

The normal vector of point v _l(i) is represented; the target mesh refers to the human body mesh dataset from different sources to be optimized.

E _{lan} is a set of manually labeled sparse corresponding points in the standard topology and the target mesh, expressed as:

Where L is the set of corresponding points pre-selected in the standard topology.