CN116311426A - A Children's Expression Scoring System - Google Patents

Abstract

The invention relates to the technical field of face recognition, in particular to a pediatric expression scoring system, which comprises: the face detection module receives an externally input expression picture, and detects a first face area from the expression picture; the feature extraction module is used for extracting a plurality of action units in the first face area to obtain a first feature group; and the score generation module is used for generating a similarity score according to the first feature group and the second feature group which are generated in advance. The beneficial effects are that: aiming at the problem that the evaluation process of dance expressions is not accurate enough mainly by teachers according to subjective, the scheme is characterized in that the first characteristic group is built by extracting corresponding action units from input expression pictures by constructing the system, so that better quantitative characterization of facial expressions is realized, and then, a better quantitative evaluation effect is realized by calculating with a second characteristic group acquired in advance.

Description

A Children's Expression Scoring System

technical field

The invention relates to the technical field of face recognition, in particular to a children's facial expression scoring system.

Background technique

Dance is a kind of performing arts, which is formed by a series of movements, which can be impromptu or professionally choreographed; dance is divided into street dance, modern dance, folk dance, street dance, ballet and so on. In order to achieve a better expressive effect, specific dance expressions are usually added to the dance movements to achieve better emotional expression.

In the prior art, the evaluation process for children's dance expressions mainly relies on teachers demonstrating the expressions corresponding to specific dance movements, and students imitate them, and then teachers give corresponding evaluations.

However, in the actual implementation process, the inventors found that the above solution evaluates the similarity of expressions manually, which leads to the problem that the evaluation process is relatively subjective and it is difficult to accurately quantify.

Contents of the invention

Aiming at the above-mentioned problems existing in the prior art, a children's facial expression scoring system is now provided.

The specific technical scheme is as follows:

A children's facial expression scoring system, comprising:

A face detection module, the face detection module receives an externally input expression picture, and detects the first facial region from the expression picture;

A feature extraction module, the feature extraction module is connected to the face detection module, and the feature extraction module extracts a plurality of action units in the first facial area to obtain a first feature group;

A score generation module, the score generation module is connected to the feature extraction module, and the score generation module generates a similarity score according to the first feature group and the pre-generated second feature group.

On the other hand, the face detection module uses a pre-trained face detection model to detect the expression pictures;

The face detection model includes:

A feature extraction network, the feature extraction network extracts feature map combinations from the input facial expression pictures;

A feature pyramid, the feature pyramid is connected to the feature extraction network, and the feature pyramid fuses the combination of feature maps to obtain a feature fusion result;

A single-stage detection module, the single-stage detection module generates the first facial region according to the feature fusion result.

On the other hand, the single-stage detection module includes:

a convolutional layer, the convolutional layer extracts image features from the feature fusion result;

a context network that generates associated features from the feature fusion results;

A merging layer, the merging layer is respectively connected to the convolutional layer and the context network, and the merging layer generates a merging feature according to the image features and the associated features;

A face classification layer, the face classification layer is connected to the merging layer, and the face classification layer generates a face classification result according to the merging feature;

A coordinate regression layer, the coordinate regression layer is connected to the merging layer, and the coordinate regression layer generates a coordinate displacement prediction result according to the merging feature;

Deformable convolutional network, the deformable convolutional network is respectively connected to the face classification layer and the coordinate regression layer, and the deformable convolutional network generates the first One facial area.

On the other hand, the feature extraction module includes:

an element extraction module, the element extraction module identifies a plurality of facial elements from the first facial region;

A vector combination module, the vector combination module is connected to the element extraction module, the vector combination module selects the facial elements and combines the facial elements into sub-vectors;

An action unit prediction module, the action unit prediction module is connected to the vector combination module, and the action unit prediction module obtains the action unit according to the sub-vector prediction and adds it to the first feature group.

On the other hand, the action unit prediction module includes:

a preprocessing layer, the preprocessing layer receives the input sub-vector and generates a preprocessing vector;

The first fully connected layer, the first fully connected layer is connected to the preprocessing layer, and the first fully connected layer generates a vector score for each of the preprocessing vectors;

A weight generation layer, the weight generation layer is connected to the first fully connected layer, and the weight generation layer generates the vector weights of each of the preprocessing vectors respectively;

A feed-forward layer, the feed-forward layer is connected to the first fully connected layer and the weight generation layer, and the feed-forward layer is obtained from the sub-vectors according to the vector score and the vector weight The action unit.

On the other hand, the preprocessing layer includes a second fully connected layer, a batch normalization layer and a linear correction layer connected in sequence.

On the other hand, before collecting the expression picture, a reference picture is also collected in advance;

The face detection module detects a second face area from the reference picture, and the feature extraction module extracts a plurality of reference units in the second face area to obtain the second feature group.

On the other hand, it also includes an action unit measurement module, the action unit measurement module is connected to the feature extraction module, and the action unit measurement module selects the reference unit to filter out the ones that are actually added to the second feature group. the reference unit.

On the other hand, the action unit measurement module includes:

a node generation module, the node generation module obtains the reference unit and creates a tree node;

a connection module, the connection module is connected to the node generation module, and the connection module creates a decision tree according to the reference unit and the tree node;

a measurement module, the measurement module is connected to the connection module, the measurement module changes the tree nodes, and calculates the variance of the decision tree to generate a measurement result;

A screening module, the screening module is connected to the measurement module, and the screening module selects a plurality of reference units that have the greatest influence on the variance of the decision tree according to the measurement results and the pre-configured extraction quantity, as the first The element of the reference in the set of two features.

On the other hand, in the score generating module, the mean square error is calculated respectively for the action unit in the first feature group and the reference unit in the second feature group, as the similarity score.

The above technical solution has the following advantages or beneficial effects:

Aiming at the problem that the evaluation process of dance expressions in the prior art is not accurate enough based on the teacher’s subjective judgment, this program builds the above system and extracts the corresponding action units from the input expression pictures to form the first feature group. A better quantitative representation of the expression is then calculated with the pre-collected second feature group, thereby achieving a better quantitative evaluation effect.

Description of drawings

Embodiments of the present invention are more fully described with reference to the accompanying drawings. However, the accompanying drawings are for illustration and illustration only, and do not limit the scope of the present invention.

Fig. 1 is the overall schematic diagram of the embodiment of the present invention;

Fig. 2 is the schematic diagram of human face detection model in the embodiment of the present invention;

3 is a schematic diagram of a single-stage detection model in an embodiment of the present invention;

Fig. 4 is a schematic diagram of a feature extraction module in an embodiment of the present invention;

5 is a schematic diagram of an action unit prediction module in an embodiment of the present invention;

FIG. 6 is a schematic diagram of a pretreatment layer in an embodiment of the present invention;

Fig. 7 is a schematic diagram of an action unit weighing module in an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

It should be noted that, in the case of no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but not as a limitation of the present invention.

The present invention includes:

A kind of children's expression scoring system, as shown in Figure 1, comprises:

Face detection module 1, face detection module 1 receives the facial expression picture of external input, detects and obtains the first face region from the facial expression picture;

The feature extraction module 2, the feature extraction module 2 is connected to the face detection module 1, and the feature extraction module 2 extracts a plurality of action units in the first face area to obtain the first feature group;

Before collecting the expression pictures, there are also pre-collected reference pictures;

The face detection module 1 detects the second face area from the reference picture, and the feature extraction module 2 extracts a plurality of reference units in the second face area to obtain the second feature group;

Score generation module 3, the score generation module 3 is connected to the feature extraction module 2, and the score generation module 3 generates a similarity score according to the first feature group and the pre-generated second feature group;

Specifically, in view of the problem that the expression evaluation process in the prior art is relatively subjective and difficult to quantify, in this embodiment, by constructing the above-mentioned children's expression scoring system, the face detection module 1 is used to intercept the input expression picture and extract the obtained Corresponding to the first facial area of face in the facial expression picture, the action unit corresponding to each facial muscle action in the first facial area is then extracted as the first feature group by the feature extraction module 2, which realizes the comparison of facial expressions. Good characterization effect; through the children's facial expression scoring system, the reference picture made by the teacher is collected in advance for the second facial area, and the reference unit is extracted to form the second feature group, which can also achieve a good characterization effect on standard expressions, and then make the score The generation module 3 can calculate the similarity score according to the quantized first feature group and the second feature group.

During the implementation process, the facial expression picture refers to the facial photo collected by the student to be evaluated, and the reference photo refers to the facial photo collected by the teacher or obtained by intercepting the teaching image. The facial photo is taken by an image sensor and may contain Face, background and part of the upper body and limbs; the first facial area and the second facial area are facial images corresponding to the actual human face area detected and intercepted by the face detection module 1, which are compared to the face of the original collection. This photo has extraneous background and edges removed. Action units and reference units refer to multiple detectable parts on the face corresponding to facial muscle movements, such as eyebrow raising, lip corner pulling, eyebrow lowering, inner eyebrow raising, etc., which can be realized by detecting the action unit and reference unit Detection of more complex facial expressions.

In one embodiment, the face detection module uses a pre-trained face detection model to detect facial expressions;

As shown in Figure 2, the face detection model includes:

Feature extraction network 11, feature extraction network 11 extracts feature map combination to the expression picture of input;

The feature pyramid 12, the feature pyramid 12 is connected to the feature extraction network 12, and the feature pyramid 12 fuses the combination of feature maps to obtain a feature fusion result;

A single-stage detection module 13. The single-stage detection module 13 generates the first face region according to the feature fusion result.

Specifically, in order to achieve a better recognition effect, in this embodiment, by constructing the above-mentioned face detection model, by passing the input facial expression picture through the feature extraction network 11 with a depth-separable convolution structure, the recognition of image features is realized. extraction, and the feature maps of the last three convolutional layers in the feature extraction network 11 are used as output to form a feature map combination; subsequently, the three effective feature layers in the feature map combination are performed through the 1X1 convolution in the feature pyramid 12 The number of channels is adjusted, and then the feature fusion of the three effective feature layers is performed by means of upsampling and feature value superposition to obtain the feature fusion result, so that the single-stage detection module 13 can predict the actual face position according to the feature fusion result. As the first facial area or the second facial area, a better detection effect on human faces can be achieved.

In one embodiment, as shown in Figure 3, the single-stage detection module 13 includes:

Convolutional layer 131, the convolutional layer 131 extracts image features from the feature fusion result;

Context network 132, context network 132 generates associated features from feature fusion results;

Merging layer 133, the merging layer 133 connects the convolutional layer 131 and the context network 132 respectively, and the merging layer 133 generates a merged feature according to image features and associated features;

The face classification layer 134, the face classification layer 134 is connected to the merge layer 133, and the face classification layer 134 generates a face classification result according to the merge feature;

The coordinate regression layer 135, the coordinate regression layer 135 is connected to the merging layer 133, and the coordinate regression layer 135 generates a coordinate displacement prediction result according to the merging feature;

The deformable convolutional network 136 is connected to the face classification layer 134 and the coordinate regression layer 135 respectively, and the deformable convolutional network 136 generates the first facial area according to the face classification result and the coordinate displacement prediction result.

Specifically, in order to achieve a better prediction effect on the face area, in this embodiment, the feature extraction of the feature fusion result is realized by adding a 1×1 convolutional layer 131, and the context network 132 is used to contextualize the feature fusion result. Recognition of mutual information, and then generate associated features to achieve a better recognition effect on the edge of the face; then, through the merge layer 133, the image features and associated features are concat processed to obtain the merged features, which are input into the face classification layer 134 and the face classification layer 134 respectively. In the coordinate regression layer 135 . Wherein, the output vector dimension of the face classification layer 134 is W/S*H/S*2K, which can discriminate and classify the face parts in the image features; Corrected coordinate regression, the output vector dimension is W/S*H/S*4K, in the coordinate regression layer 135, by predicting the relative scaling and displacement of each face and ground truth at each sliding point to realize this process. Subsequently, after realizing the generation of the face classification results and coordinate location prediction results, the deformation convolution network 136 is further added as a deformation layer to simulate geometric deformation, so as to realize the difference between children's expressions and adult expressions. A better simulation effect, and then extract the first facial region.

In one embodiment, as shown in Figure 4, the feature extraction module 2 includes:

Element extraction module 21, element extraction module 21 obtains a plurality of facial elements from the identification of the first facial region;

Vector combination module 22, the vector combination module 22 is connected to the element extraction module 21, the vector combination module selects face elements and combines the face elements into sub-vectors;

The action unit prediction module 23, the action unit prediction module 23 is connected to the vector combination module 22, the action unit prediction module 23 obtains the action unit according to the sub-vector prediction and adds it to the first feature group.

Specifically, in order to achieve a better generation process of action units, in this embodiment, after the face region is extracted and the feature layer is generated by the face detection module 1, the element extraction module 21 is further used to extract the first face The facial elements are identified in the area, and the vector combination module 22 assembles a specific number of facial elements into W*H sub-vectors, each sub-vector may represent an action unit. Subsequently, each sub-vector is predicted by the action unit prediction module 23 to determine the actual action unit and add it to the first feature group.

In one embodiment, as shown in Figure 5, the action unit prediction module 23 includes:

Preprocessing layer 231, the preprocessing layer 231 receives the input sub-vector and generates a preprocessing vector;

The first fully connected layer 232, the first fully connected layer 232 is connected to the preprocessing layer 231, and the first fully connected layer 232 generates the vector score of each preprocessing vector respectively;

Weight generating layer 233, weight generating layer 233 is connected to the first fully connected layer 232, and weight generating layer 233 generates the vector weight of each preprocessing vector respectively;

Feedforward layer 234. The feedforward layer 234 connects the first fully connected layer 232 and the weight generation layer 233. The feedforward layer 234 obtains multiple action units from the sub-vectors according to the vector score and vector weight.

Specifically, in order to achieve a better prediction effect on the action unit, in this embodiment, the input sub-vectors are preprocessed by setting the preprocessing layer 231, and then each preprocessing vector is processed in the first fully connected layer 232 The vector scores are calculated separately, and the vector weight of each preprocessing vector is calculated by the weight generation layer 233 through the Softmax function, so that the feedforward layer 234 can calculate the actual action unit based on the sigmoid activation function.

In one embodiment, as shown in FIG. 6 , the preprocessing layer 231 includes a second fully connected layer 2311 , a batch normalization layer 2312 and a linear correction layer 2313 connected in sequence.

Specifically, in order to achieve a better preprocessing effect on the sub-vectors, in this embodiment, the second fully connected layer 2311 and the batch normalization layer 2312 are sequentially set in the pre-processing layer 231 to perform normalization processing on the sub-vectors (BatchNormalization) and The linear correction layer 2313 with the ReLU function achieves better vector preprocessing effect.

In one embodiment, as shown in Figure 7, it also includes an action unit measurement module 4, the action unit measurement module 4 is connected to the feature extraction module 2, and the action unit measurement module 4 selects the reference unit to filter out the actual addition to the second feature the reference unit of the group;

Action unit measurement module 4 includes:

Node generation module 41, node generation module 41 obtains reference unit and creates tree node;

Connection module 42, connection module 42 connection node generation module 41, connection module 42 creates decision tree according to reference unit and tree node;

A measurement module 43, the measurement module 43 is connected to the connection module 42, the measurement module 43 changes the tree nodes, and calculates the variance of the decision tree to generate a measurement result;

The screening module 44, the screening module 44 is connected to the measurement module 43, and the screening module 44 selects a plurality of reference units that have the greatest impact on the variance of the decision tree according to the measurement results and the pre-configured extraction quantity, as the reference units in the second feature group.

Specifically, in order to achieve a more significant evaluation effect on the similarity of expressions, in this embodiment, in the process of collecting reference pictures, the action unit measurement module 4 is further set up to measure the effect of the reference units that have a greater impact on the expression. Partially screened, and only this part of the reference units is added to the second feature group as the reference units that will be selected during the actual evaluation. Through this method, the range of the similarity score can be increased in the process of calculating the similarity between each action unit and the reference unit, so as to achieve a better measurement effect.

In the implementation process, in order to achieve a better screening effect, in this embodiment, a decision tree is used to measure the influence of each reference unit on the expression. Specifically, the node generation module 41 uses each reference unit as a tree node and a leaf, and the connection module 42 connects the tree nodes according to the association between the reference nodes to create a decision tree. Subsequently, the tree nodes are randomly changed by the measurement module 43, and the overall variance of the decision tree is calculated to measure the purity of the decision tree, so that the screening module 44 can filter according to the variation of the variance and the variation of the corresponding reference unit Get the TopN reference units that have the greatest influence on the expression. In one embodiment, the screening module 44 makes statistics on the influence of the variation of each reference unit on the variance, and screens out the 10 reference units with the greatest influence as the reference units in the second feature group.

In one embodiment, in the score generation module 3 , the mean square errors are respectively calculated for the action units in the first feature group and the reference units in the second feature group as similarity scores.

Specifically, in order to achieve a better similarity calculation effect, in this embodiment, the mean square error is selected as the scoring method to measure the similarity of expressions, thereby achieving a better evaluation effect on the similarity between each action unit and the reference unit .

The above are only preferred embodiments of the present invention, and are not intended to limit the implementation and protection scope of the present invention. For those skilled in the art, they should be able to realize the equivalent replacement and The solutions obtained by obvious changes shall all be included in the protection scope of the present invention.

Claims (10)

1. a children's facial expression scoring system, is characterized in that, comprises: A face detection module, the face detection module receives an externally input expression picture, and detects the first facial region from the expression picture; A feature extraction module, the feature extraction module is connected to the face detection module, and the feature extraction module extracts a plurality of action units in the first facial area to obtain a first feature group; A score generation module, the score generation module is connected to the feature extraction module, and the score generation module generates a similarity score according to the first feature group and the pre-generated second feature group. 2. children's facial expression scoring system according to claim 1, is characterized in that, described facial detection module adopts a face detection model trained in advance to detect described facial expression picture; The face detection model includes: A feature extraction network, the feature extraction network extracts feature map combinations from the input facial expression pictures; A feature pyramid, the feature pyramid is connected to the feature extraction network, and the feature pyramid fuses the combination of feature maps to obtain a feature fusion result; A single-stage detection module, the single-stage detection module generates the first facial region according to the feature fusion result. 3. children's facial expression scoring system according to claim 2, is characterized in that, described single-stage detection module comprises: a convolutional layer, the convolutional layer extracts image features from the feature fusion result; a context network that generates associated features from the feature fusion results; A merging layer, the merging layer is respectively connected to the convolutional layer and the context network, and the merging layer generates a merging feature according to the image features and the associated features; A face classification layer, the face classification layer is connected to the merging layer, and the face classification layer generates a face classification result according to the merging feature; A coordinate regression layer, the coordinate regression layer is connected to the merging layer, and the coordinate regression layer generates a coordinate displacement prediction result according to the merging feature; Deformable convolutional network, the deformable convolutional network is respectively connected to the face classification layer and the coordinate regression layer, and the deformable convolutional network generates the first One facial area. 4. children's expression scoring system according to claim 1, is characterized in that, described feature extraction module comprises: an element extraction module, the element extraction module identifies a plurality of facial elements from the first facial region; A vector combination module, the vector combination module is connected to the element extraction module, the vector combination module selects the facial elements and combines the facial elements into sub-vectors; An action unit prediction module, the action unit prediction module is connected to the vector combination module, and the action unit prediction module obtains the action unit according to the sub-vector prediction and adds it to the first feature group. 5. children's facial expression scoring system according to claim 4, is characterized in that, described action unit prediction module comprises: a preprocessing layer, the preprocessing layer receives the input sub-vector and generates a preprocessing vector; The first fully connected layer, the first fully connected layer is connected to the preprocessing layer, and the first fully connected layer generates a vector score for each of the preprocessing vectors; A weight generation layer, the weight generation layer is connected to the first fully connected layer, and the weight generation layer generates the vector weights of each of the preprocessing vectors respectively; A feed-forward layer, the feed-forward layer is connected to the first fully connected layer and the weight generation layer, and the feed-forward layer is obtained from the sub-vectors according to the vector score and the vector weight The action unit. 6. children's facial expression scoring system according to claim 5, is characterized in that, described preprocessing layer comprises the second fully connected layer, batch normalization layer and linear correction layer that are connected successively. 7. children's facial expression scoring system according to claim 1, is characterized in that, before collecting described facial expression picture, also gathers reference picture in advance; The face detection module detects a second face area from the reference picture, and the feature extraction module extracts a plurality of reference units in the second face area to obtain the second feature group. 8. children's facial expression scoring system according to claim 7, is characterized in that, also comprises action unit measurement module, and described action unit measurement module connects described feature extraction module, and described action unit measurement module carries out to described reference unit Select to filter out the reference cells that are actually added to the second feature set. 9. children's facial expression scoring system according to claim 8, is characterized in that, described action unit measuring module comprises: a node generation module, the node generation module obtains the reference unit and creates a tree node; a connection module, the connection module is connected to the node generation module, and the connection module creates a decision tree according to the reference unit and the tree node; a measurement module, the measurement module is connected to the connection module, the measurement module changes the tree nodes, and calculates the variance of the decision tree to generate a measurement result; A screening module, the screening module is connected to the measurement module, and the screening module selects a plurality of reference units that have the greatest influence on the variance of the decision tree according to the measurement results and the pre-configured extraction quantity, as the first The element of the reference in the set of two features. 10. children's facial expression scoring system according to claim 1, is characterized in that, in the described scoring generation module, to the reference unit in the described action unit in the described first characteristic group and the described second characteristic group respectively Calculate the mean square error as the similarity score.

Images