CN103971394A - Facial animation synthesizing method - Google Patents

Abstract

A facial animation synthesizing method comprises the step of obtaining a facial front image on which the animation synthesis needs to be carried out, the step of carrying out three-dimensional human face model estimation based on the front image of the human face, the step of carrying out the front human face carton effect synthesis based on the facial feature analysis to obtain a front facial carton effect composite graph, the step of carrying out the multi-posture facial carton effect synthesis based on the human face three-dimensional model estimation and the front facial carton effect composite graph, the step of embedding the synthesized multi-posture facial carton effect synthesis in the preset animation video material to obtain the animation video of the human face cartoon effect. According to the facial animation synthesizing method, the traditional artificial animation creation of the facial carton effect is avoided, achieving is easy and fast, and different individuation requirements of different users can be met.

Description

Synthesis Method of Human Face Animation

technical field

The invention relates to the field of image processing, in particular to a method for synthesizing human face animation with a cartoon effect.

Background technique

With the vigorous development of multimedia carriers such as mobile phones and the Internet, the digital cartoon and animation industry has grown rapidly and has gradually become a mainstream cultural product in the information age.

Although personalized cartoon animation has a good application prospect, so far almost all personalized cartoon animations are hand-created by animators. The output is low and the cost is high, which cannot meet the growing demand of consumers for personalized consumption.

Contents of the invention

In view of this, it is necessary to provide a fast and personalized face animation synthesis method.

A method for synthesizing human face animation, comprising the steps of:

Obtain the frontal image of the face to be synthesized by animation;

Carrying out 3D face model estimation based on the face frontal image;

Based on the face feature analysis, the frontal face cartoon effect is synthesized, and the frontal face cartoon caricature effect composite map is obtained;

Carrying out multi-pose face cartoon effect synthesis based on the three-dimensional face model estimation and the front face cartoon caricature effect synthesis map;

Synthesizing and embedding the synthesized multi-posture human face cartoon effect into the preset animation video material to obtain an animation video with human face cartoon effect.

Preferably, the step of estimating a three-dimensional face model based on the front face image further includes:

Construct a training set {s ₁ , s ₂ ,…,s _m } containing m three-dimensional face shape models, let the feature points p _t on the three-dimensional model s _j (1≤j≤m) and the face image = (u _t , v _t ) The vertex corresponding to ^T is v _tj , where 1≤t≤k, k is the number of feature points on the face image.

Preferably, the optimized energy function is used:

f(v)=argmin(||p _t -(v _tj ) _1:2 ||),

Find the optimal three-dimensional point so that the two-dimensional coordinates of its vertices are closest to p _t .

Preferably, based on the above energy function, use the optimization algorithm based on the sparse linear model to treat all the feature points as a whole and combine them into a sparse shape vector, and then use the prior knowledge in the training library to optimize the sparse The vector has missing data for overall estimation, so as to obtain the quasi-3D coordinates of feature points.

Preferably, after the three-dimensional coordinate-like feature points are obtained, the three-dimensional face model is reconstructed using a three-dimensional face model reconstruction method based on feature point deformation technology.

Preferably, the method of described three-dimensional facial model reconstruction comprises:

Let l vertices of the model form a three-dimensional point set: Q={x ₁ ,x ₂ ,…,x _l }, where the subscripts within k are the feature points that are estimated to have similar three-dimensional information, and the features of similar three-dimensional information The optimal estimated value of the point is: Y={y ₁ ,y ₂ ,…,y _k }, using the following deformation technology to estimate the 3D face model:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; \&Phi;</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ,</span>$

Among them, Φ() is the radial basis function, and w _i is the weight.

Preferably, the step of synthesizing the frontal face cartoon effect based on facial feature analysis includes:

The face feature point positioning method based on the active shape model technology is adopted, and the scale and rotation transformation are performed according to the feature point of the human eye to achieve normalization, and the frontal face cartoon image with the line drawing effect is obtained through the connection of the Bezier curve .

Preferably, the step of synthesizing a multi-pose human face cartoon effect based on the 3D model estimation of the human face and the cartoon caricature effect synthesis map of the frontal face comprises:

Carry out a Delaunay triangulation scheme on the synthetic image of the cartoon caricature effect of the front face and the three-dimensional face model respectively, and after obtaining the result of the Delaunay triangulation, combine the face feature points and the three-dimensional face in the composite picture with the caricature effect of the front face The face feature points in the model map the frontal face cartoon image as a texture onto the estimated 3D face model;

Using the geometric rigid body transformation matrix combined with the preset parameter projection matrix, the three-dimensional face model is transformed and projected into a two-dimensional image, so as to obtain the synthesis result of multi-pose face cartoon caricature.

Preferably, when performing the Delaunay triangulation scheme, first project the points of the three-dimensional face model onto a two-dimensional image plane, then perform Delaunay triangulation on the two-dimensional image plane, and finally determine the three-dimensional space according to its connection relationship Midpoint connections.

Preferably, after obtaining the synthesis result of the multi-pose face cartoon effect, select the corresponding face cartoon comic effect synthesis result to replace the original protagonist according to the facial posture of the main character in the preset animation video material, so as to obtain the result of the face cartoon effect Animated video.

Preferably, using the minimized objective function:

${\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}_{\mathrm{<span\; class="notranslate">\; I</span>}}<span\; class="notranslate">\; \&Integral;</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&dtri;</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>$

Make the replaced avatar and background blend smoothly, where I is the smoothed image, and v is the gradient field of the original cartoon animation image.

Compared with the prior art, the human face animation synthesis method generates a three-dimensional model of the human face according to the frontal image of the human face and performs synthesis of the cartoon effect of the frontal face, and performs multi-pose synthesis according to the three-dimensional model of the human face and the cartoon effect of the frontal face Face cartoon effect synthesis, and finally synthesize the synthesized multi-pose face effect with preset animation video materials, avoiding the traditional artificial creation of face cartoon effect animation, simple and fast to achieve, and different users can according to different needs Synthesis of animation with cartoon effect on faces, so it can meet various individual needs.

Description of drawings

FIG. 1 is a schematic flowchart of a method for synthesizing human face animation according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

Please refer to Fig. 1, the human face animation synthesis method of the embodiment of the present invention comprises the following steps:

Step 01: Obtain the frontal image of the human face to be synthesized with animation. In this embodiment, the frontal image of the human face is obtained by a digital camera.

Step 02, perform 3D face model estimation based on the face frontal image.

Specifically, first construct a training set {s ₁ , s ₂ ,…,s _m } containing m 3D face shape models, and let the features on the 3D model s _j (1≤j≤m) and the face image The vertex corresponding to the point p _t =(u _t , v _t ) ^T (1≤t≤k, k being the number of feature points on the face image) is v _tj . Find the optimal three-dimensional point by optimizing the following basic functions so that the two-dimensional coordinates of its vertices are closest to p _t .

f(v)＝argmin(||p _t -(v _tj ) _1:2 ||) (1)

Based on the above energy function, using the optimization algorithm based on the sparse linear model, all the feature points are regarded as a whole, and combined into a sparse shape vector, and then the sparse vector is missing by using the prior knowledge in the training library The data is estimated as a whole, so as to obtain the three-dimensional coordinates of the feature points. In this way, relatively accurate and stable estimation results can be obtained.

After the three-dimensional coordinate-like feature points are obtained, the three-dimensional face model is reconstructed by using the three-dimensional face model reconstruction method based on the feature point deformation technology. Specifically, let the l vertices of the model form a 3D point set as: Q={x ₁ ,x ₂ ,…,x _l }, where the subscripts within k are the feature points that are estimated to have class-like 3D information. The optimal estimated value of the feature points of the 3D-like information is: Y={y ₁ ,y ₂ ,...,y _k }.

The 3D face model is estimated using the following deformation techniques:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; \&Phi;</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

In formula (2), Φ() is the radial basis function, and w _i is the weight.

Step 03: Based on the face feature analysis, the cartoon effect synthesis of the frontal face is carried out to obtain the composite picture of the cartoon caricature effect of the frontal face.

In this embodiment, a face feature point positioning method based on Active Shape Model (ASM) technology is adopted, and scale and rotation transformations are performed according to the feature point of human eyes to realize normalization. On the basis of obtaining the feature points of the face, a frontal face cartoon image with a line drawing effect is obtained through a Bezier curve (Bézier curve) connection.

Step 04, performing multi-pose face cartoon effect synthesis.

Specifically, a Delaunay triangulation scheme is performed on the front face cartoon caricature composite image and the three-dimensional face model. In this embodiment, the points of the three-dimensional face model are first projected onto the two-dimensional image plane, and the Delaunay triangulation is performed on the two-dimensional image plane, and finally the connection relationship of the points in the three-dimensional space is determined according to the connection relationship, which can avoid Delaunay triangulation under traditional 3D data is a very complex operation. After obtaining the result of Delaunay triangulation, texture mapping is realized by combining the face feature points in the frontal face cartoon caricature synthesis image and the face feature points in the 3D face model, that is, the frontal face cartoon image is used as The texture is mapped to the estimated 3D face model. Finally, the geometric rigid body transformation matrix combined with the camera internal parameter projection matrix is used to transform the pose of the 3D face model and project it into a 2D image to obtain the synthesis result of multi-pose face cartoon caricature.

Step 05: Synthesize and embed the synthesized multi-pose face cartoon effect into the preset animation video material. Said synthesis is based on said personalized cartoon animation.

Specifically, after obtaining the synthesis result of multi-pose human face cartoon effect, the original protagonist can be replaced by the corresponding synthesis result of human face cartoon caricature effect selected according to the facial posture of the main character in the preset animation video material.

In this embodiment, a space fusion scheme based on image gradient domain processing is used to smoothly fuse the replaced avatar with the background, eliminate the abrupt contrast between the face cartoon caricature synthesis and the background, and improve the realism of the video image. The minimization objective function used is as follows:

${\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}_{\mathrm{<span\; class="notranslate">\; I</span>}}<span\; class="notranslate">\; \&Integral;</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&dtri;</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Among them, I is the smoothed image, and v is the gradient field of the original cartoon animation image. The above minimization problem can be transformed into solving the following Poisson equation:

ΔI–div(v)=0 (4)

The above Poisson equation can be numerically solved by solving a large-scale system of linear equations.

The human face animation synthesis method generates a three-dimensional model of a human face according to a frontal image of a human face and synthesizes a frontal human face cartoon effect, and synthesizes a multi-pose human face cartoon effect according to the three-dimensional model of a human face and the frontal human face cartoon effect synthesis, Finally, the synthesized multi-pose face effect is synthesized with the preset animation video material, avoiding the traditional manual creation of cartoon face animation, and the realization is simple and fast, and different users can perform cartoon face animation according to different needs Synthetic, so it can meet many different individual needs.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention are still within the scope of this invention. The protection scope of the technical solution of the invention.

Claims (11)

1. a human face animation synthetic method, comprises the steps:

Obtain and want to carry out the synthetic people's face direct picture of animation;

Based on described people's face direct picture, carry out three-dimensional face model estimation;

Based on Facial Feature Analysis, carry out front face cartoon effect and synthesize, obtain front face cartoon effect composite diagram;

Based on described human face three-dimensional model estimate and described front face cartoon effect composite diagram to carry out multi-pose Face cartoon effect synthetic;

By in the default animated video material of the synthetic embedding of described synthetic multi-pose Face cartoon effect, obtain the animated video of human face cartoon effect.

2. human face animation synthetic method as claimed in claim 1, is characterized in that: described step of carrying out three-dimensional face model estimation based on described people's face direct picture further comprises:

Build a training set { s who comprises m three-dimensional face shape ₁, s ₂..., s _m, make three-dimensional model s _j(1≤j≤m) unique point p above and on facial image _t=(u _t, v _t) ^tcorresponding summit is v _tj, 1≤t≤k wherein, k is the number of unique point on described facial image.

3. human face animation synthetic method as claimed in claim 2, is characterized in that: adopt optimization energy function:

f(v)＝argmin(||p _t-(v _tj) _1:2||)，

Find two-dimensional coordinate and p that optimum three-dimensional point makes its summit _tthe most approaching.

4. human face animation synthetic method as claimed in claim 3, it is characterized in that: take above-mentioned energy function as basis, the optimized algorithm of utilization based on sparse linear model, all unique points are regarded as to an integral body, and be combined into a sparse shape vector, then utilize the priori in training storehouse to have the data of disappearance to carry out overall estimation to sparse vector, thereby obtain the class three-dimensional coordinate of unique point.

5. human face animation synthetic method as claimed in claim 4, is characterized in that: after obtaining class three-dimensional coordinate unique point, utilize the three-dimensional face model method for reconstructing based on unique point deformation techniques to carry out three-dimensional face model reconstruction.

6. human face animation synthetic method as claimed in claim 5, is characterized in that: the method that described three-dimensional face model is rebuild comprises:

If the l of model summit forms a three-dimensional point set and is: Q={x ₁, x ₂..., x _l, being wherein marked on down k and take interiorly as estimating at the unique point of class three-dimensional information, the optimization estimated value of the unique point of class three-dimensional information is: Y={y ₁, y ₂..., y _k, utilize following deformation techniques to estimate three-dimensional face model:

$F\left(x_j\right)=\underset{i=1}{\overset{k}{\mathrm{\&Sigma;}}}{w}_i\mathrm{\&Phi;}\left(\right||x_j-y_i|\left|\right)+x_j,$

Wherein, Φ () is radial basis function, w _ifor weight.

7. human face animation synthetic method as claimed in claim 1, is characterized in that: describedly based on Facial Feature Analysis, carry out the synthetic step of front face cartoon effect and comprise:

The man face characteristic point positioning method of employing based on active shape model technology, and according to human eye feature anchor point carry out yardstick, rotational transform realizes normalization, the mode connecting by Bezier obtains the front face cartooning image of stick figure effect.

8. human face animation synthetic method as claimed in claim 1, is characterized in that: describedly based on described human face three-dimensional model, estimate and described front face cartoon effect composite diagram carries out the synthetic step of multi-pose Face cartoon effect and comprises:

Described front face cartoon effect composograph and three-dimensional face model are carried out respectively to Delaunay triangulation scheme, obtain after Delaunay triangulation result, in conjunction with the face characteristic in human face characteristic point and three-dimensional face model in front face cartoon effect composite diagram name a person for a particular job front face cartooning image as texture to the three-dimensional face model of estimating to obtain;

Thereby utilize how much rigid body translation matrixes in conjunction with parameter preset projection matrix, three-dimensional face model to be carried out to posture changing and project into two dimensional image obtaining the synthetic result of multi-pose Face cartoon effect.

9. human face animation synthetic method as claimed in claim 8, it is characterized in that: when carrying out Delaunay triangulation scheme, first the spot projection of three-dimensional face model is arrived to two dimensional image plane, in described two dimensional image plane, carry out again Delaunay triangulation, finally according to its annexation, determine the annexation of three dimensions mid point.

10. human face animation synthetic method as claimed in claim 1, it is characterized in that: after obtaining the synthetic result of multi-pose Face cartoon effect, according to leading role's facial pose in described default animated video material, select the synthetic result of corresponding human face cartoon caricature effect to replace original leading role, obtain the animated video of human face cartoon effect.

11. human face animation synthetic methods as claimed in claim 10, is characterized in that: utilize and minimize objective function:

${\arg \min }_I\&Integral;{\left|\right|\&dtri;I-v\left|\right|}^2,$

Head portrait and the background of replacing are smoothly merged, and wherein I be the image after smoothly, and v is the gradient fields of former cartoon image.