KR20160053749A - Method and systems of face expression features classification robust to variety of face image appearance - Google Patents

Abstract

The present invention relates to a facial expression feature classification method and system. A facial feature classification method performed in a facial feature classification system for facial expression recognition includes generating a facial expression class variation image corresponding to a variation component in a facial expression class using training facial images constructed for each facial expression class for a query facial image Calculating an image difference between the change image in the facial expression class and the query face image and defining the difference as a facial feature, and classifying the facial feature into a specific facial expression class by applying a rare expression, . As described above, by using the facial feature extraction and classification method that removes the variation components in the facial expression class that are not related to the facial expressions appearing in the query facial image, there is an advantage that stable facial recognition performance is exhibited even if there are various changes in the query facial image.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and a system for classifying facial expressions that are robust against diversity of facial images,

The present invention relates to a facial expression feature classification method and system, and more particularly, to a method and system for feature extraction and classification for facial expression recognition.

Facial Expression Recognition extracts facial features related to facial expression from a query facial image and classifies the facial features to determine whether the query facial image corresponds to a certain facial expression (eg, neutral, smile, surprise) It is decided through the classification process. Many facial expression recognition methods use a facial feature extraction method that extracts texture information corresponding to the entire area or local contour or wrinkles of the face.

At this time, the method of recognizing the facial expression is based on the face image of the block unit, and by measuring the similarity degree between the query face image and the registered face image, the face of the user inputted through the camera is compared with the registered face image, To find out. In this regard, Korean Unexamined Patent Publication No. 2009-0021279 (published on Sep. 22, 2010) discloses a technique for improving face recognition performance even when the number of registered face images is small, such as a robot environment, A variety of combinations of images are created by judging the degree of similarity. Thus, a method for increasing the diversity of registered face images is disclosed.

However, if the person corresponding to the query face image does not exist in the training face images, confusion between the facial identity of the characters and facial expression occurs, .

In order to solve these drawbacks, facial expression recognizing method which computes the image difference between the query facial image and the neutral expression facial image of the query imitated person to remove the characteristic information of the person has been studied.

However, there are two problems with the facial expression recognizing method that removes the unique information of the person. First, in the real face facial expression recognition, the person corresponding to the query face image does not exist in the training face images frequently, which is not applicable in this case. Second, when the acquired illumination condition is different between the query face image and the no face image, the recognition difference is deteriorated according to the illumination change when calculating the image difference.

Korean Patent Laid-Open Publication No. 2009-0021279 (published Sep. 22, 2010) discloses a " block-based face recognition method and apparatus ".

Embodiments of the present invention provide a facial expression feature classification method and system that are robust to diversity of facial images that are not related to facial expressions existing in a facial image. It should be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to one aspect of the present invention, there is provided a facial feature classification method performed in a facial feature classification system for facial expression recognition, comprising the steps of: A step of generating a change image in a facial expression class corresponding to a change component in a facial expression class using images, a step of calculating an image difference between the change image in the facial expression class and the query facial image, And classifying the facial feature into a specific facial expression class by applying a rare expression and determining a facial expression class corresponding to the query facial image.

Here, the change image in the facial expression class may be obtained by approximating the query face image using a linear combination of the training face images.

The variation image in the facial expression class can be generated using each weight vector and each training face image by obtaining a weight vector representing a weight for linear combination of each training face image by applying a normalized least squares method have.

The step of determining the facial expression class may include a step of obtaining a plurality of sparse coefficient vectors for expressing facial features of the query face image using a dictionary composed of facial features of the training facial image, And determining a facial expression class of the query facial image by finding a facial expression class in which a sparse coefficient is most concentrated in the fused sparse coefficient vector.

In addition, the dictionary may define a face feature as many as the number of facial expressions using the image difference between the training face image and the change image in the facial expression class.

According to another aspect of the present invention, there is provided a facial feature classification system for recognizing a facial expression, comprising: training face images composed of each facial expression class for a query facial image, An image difference calculation unit for calculating an image difference between the change image in the facial expression class and the query face image to define the face difference as a face feature; And an image classifier for classifying the query facial image and determining a facial expression class corresponding to the query facial image.

Here, the change image in the facial expression class may be obtained by approximating the query face image using a linear combination of the training face images.

The variation image in the facial expression class can be generated using each weight vector and each training face image by obtaining a weight vector representing a weight for linear combination of each training face image by applying a normalized least squares method have.

The image classification unit may further include: a sparse expression applying unit that obtains a plurality of sparse coefficient vectors for expressing facial features of the query face image using a dictionary composed of facial features of the training face image; And a classifier for finding a facial expression class in which a sparse coefficient is most concentrated in the fused sparse coefficient vector to determine a facial expression class of the query facial image.

In addition, the dictionary may define a face feature as many as the number of facial expressions using the image difference between the training face image and the change image in the facial expression class.

According to the embodiment of the present invention, in order to achieve robust facial expression recognition of diversity of facial images irrelevant to facial expression existing in a facial image, when there is no facial expression image of a query facial image character on training facial images, It is possible to overcome the problems of the prior art in which recognition performance is degraded due to the diversity of the query face image by reducing the illumination change in the query face image.

As described above, by using the facial feature extraction and classification method that removes the intra-class variation components unrelated to the facial expression appearing in the query facial image, the facial feature recognition performance is stable even if there are various changes in the query facial image .

1 is a block diagram of a facial expression feature classification system according to an embodiment of the present invention.
FIG. 2 is a detailed block diagram of an image classification unit of a facial expression feature classification system according to an embodiment of the present invention. Referring to FIG.
FIG. 3 is a flowchart illustrating a facial expression feature classification method by the facial expression feature classification system according to an embodiment of the present invention.
FIG. 4 illustrates an example of a process of extracting facial features in the facial expression feature classification method according to the facial expression feature classification system according to the embodiment of the present invention.
5 and 6 illustrate a variation image in a facial expression class that is generated using a query facial image and a training facial image by the facial expression feature classification method by the facial expression feature classification system according to the embodiment of the present invention .
FIG. 7 is a diagram illustrating a query facial image q, a facial expression class variation image h _i ^q , and a facial feature y _i ^q , which are used by the facial expression feature classification method by the facial expression feature classification system according to the embodiment of the present invention. ). ≪ / RTI >

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "including" an element, it is to be understood that the element may include other elements as well as other elements, And does not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a facial expression feature classification system (apparatus) according to an embodiment of the present invention.

As described above, the facial expression feature classification system 100 according to the embodiment includes a change image generation unit 110, an image difference calculation unit 120, an image classification unit 130, and the like.

The change image generation unit 110 generates a change image in the facial expression class corresponding to the change component in the facial expression class using the training facial images configured for each facial expression class with respect to the query facial image.

Here, the change image in the facial expression class can be obtained by approximating a query face image using a linear combination of training face images. In addition, by applying the normalized least squares method, each weight vector representing a weight for a linear combination of training face images is obtained, and a variation image in a facial expression class can be generated using each weight vector and each training face image.

The image difference calculation unit 120 calculates an image difference between the change image in the facial expression class and the query face image to define the feature as a facial feature.

The image classifying unit 130 classifies the facial feature into a specific facial expression class by applying the rare facial expression to determine the facial expression class corresponding to the query facial image. The detailed configuration of the image classifying unit 130 is shown in FIG.

2 is a detailed block diagram of the image classifying unit 130 constituting the facial expression feature classification system 100 according to an embodiment of the present invention.

As shown, the image classification unit 130 includes a sparse representation application unit 131, a fusion unit 133, and a classification unit 135.

The sparse expression application unit 131 obtains a plurality of sparse coefficient vectors for expressing facial features of the query face image using a dictionary composed of face features of the training face image.

Here, the dictionary defines facial features as many as the number of facial expressions using the difference between the training facial image and the variation image in the facial expression class.

The fusion unit 133 fuses a plurality of sparse coefficient vectors to obtain a fused sparse coefficient vector.

The classifying unit 135 finds a facial expression class in which the sparse coefficient is most concentrated in the fused sparse coefficient vector and determines the facial expression class of the query facial image.

3 is a flowchart illustrating a facial expression feature classification method by a facial expression feature classification system (apparatus) according to an embodiment of the present invention.

As shown in the figure, the facial expression feature classification method according to the embodiment includes the steps of generating a variation image in a facial expression class corresponding to a variation component in a facial expression class using training facial images configured for each facial expression class, (S201 to S203).

Next, the image difference between the change image in the facial expression class and the query face image is calculated and defined as a face feature (S205).

The method further includes a step (S207 to S211) of classifying the facial feature into a specific facial expression class by applying the rare expression and determining a facial expression class corresponding to the query facial image.

Finally, the step S213 of outputting the label of the determined facial expression class is further included.

Hereinafter, the facial expression feature classification method by the facial expression feature classification system (apparatus) according to the embodiment of the present invention will be described in more detail with reference to FIG. 1 to FIG.

First, when a query face image is input to the facial expression feature classification system 100 (S201), the change image generation unit 110 generates i-th (i = 1 , ..., C) intra-class variation images are generated one by one. Through this process, the change images in C facial classes corresponding to the total of C facial classes are obtained.

For this purpose, we define the query face image q∈R ^N and define the whole training face image set as Φ = [Φ ₁ , Φ ₂ , ..., Φ _C ] ∈ R ^NxM . Where N and M are the number of dimensions of the vectorized face image and the number of total training face images, respectively. In addition, Φ = [t _{i , 1} , t _{i , 2} , ..., t _{i , Mi} ] ∈ R ^NxMi is the training face image set of the ith facial expression class, where t _ij ∈ R ^N is the j Second training face image.

In order to extract facial features of the query facial image q, a change image h _i ^q in the facial expression class is generated using Φ _i (S203). h _i ^q is obtained by approximating q using a linear combination of training face images contained in Φ _i . Therefore, we apply the normalized least squares method to obtain the weight vector w _i ^q = [w ₁ , w ₂ , ..., w _Mi ] ^T ∈ R ^Mi representing the weight of each training face image The optimization problem defined in Equation (1) must be solved.

를 안정화시키기 위한 정규화 항이다.Here, || ₂ is the L2 norm of the vector. || q-Φ _i w _i ^q || ₂ ² is a reconstruction error and || w _i ^q || ₂ ² years

Lt; / RTI >

를 이용하여 표정 클래스 내 변화 영상 h_i ^q은 수학식 2와 같이 표현된다.The weight vector obtained using Equation (1)

The change image h _i ^q in the facial expression class is expressed by Equation (2).

의 각 요소(element)는 해당하는 훈련 얼굴 영상의 선형 조합에 대한 가중치를 나타낸다. h_i ^q는 i번째 표정 클래스의 훈련 얼굴 영상 셋 Φ_i 를 이용하여 생성되었으므로 i번째 표정 클래스에 해당하는 표정을 나타낸다.here,

Each element of the training image represents a weight for a linear combination of corresponding training face images. Since h _i ^q is generated using the training face image set Φ _i of the i-th facial expression class, the facial expression corresponding to the i-th facial expression class is shown.

Then, the image difference calculation unit 120 calculates an image difference between the i-th (i = 1, ..., C) facial expression variation image and the query face image, and defines it as a facial feature. Through this process, C facial features corresponding to a total of C facial expression classes are obtained (S205).

Fig. 5 and Fig. 6 show three training images (three rows in height) having the highest weight when generating query face images (leftmost column) q and h _i ^q when i = Neutral and i = Surprise, (Rightmost column) in the facial expression class. It can be seen that the generated h _i ^q is similar to the query face image q and the facial identity when compared with the facial identity.

The change image in C expression classes included in the entire training face image set Φ h _i ^q (i = 1, ..., C), the face feature y _i ^q is defined as shown in Equation (3).

Fig. 7 shows the facial feature y _i ^q obtained using Equation 3 when i = Neutral, i = Smile, i = Surprise. Since h _i ^q and q are similar to each other, the face feature y _i ^q obtained by the image difference suppresses the influence of elements not related to the facial expression, and the expression ^q and h _i ^q And the effect of emphasizing the difference of the reference facial expression.

Next, the image classifying unit 130 classifies the facial feature into a specific facial expression class by applying the rare facial expression, and determines the facial expression class corresponding to the query facial image. That is, the face feature and the dictionary obtained from the change image in the i-th facial expression class are input and a rare expression is applied. This process obtains C sparse representation coefficient vectors corresponding to a total of C expression classes.

To this end, the sparse expression applying unit 131 obtains a plurality of sparse coefficient vectors for expressing facial features of the query face image using a dictionary composed of face features of the training face image. Here, the dictionary defines face features as many as the number of facial expressions using the image difference between the training face image and the change image in the facial expression class (S207).

The classification process using the rare expression classification will be described using the facial features y _i ^s ∈ R ^N (i = 1, ..., C) of the C facial expression classes obtained by the equation (3).

로부터 훈련 얼굴 특징들을 추출하기 위해 훈련 얼굴 영상 셋 Φ(i=1,…,C)과 수학식 1과 수학식 2 및 수학식 3을 이용하여 얻어진다. 이는 쿼리 얼굴 영상에서 얼굴 특징을 추출하는 방법과 완전히 동일하다. 구체적으로, 한 개의 훈련 얼굴 영상 s에서 얼굴 특징을 추출할 때 표정 클래스 내 변화 영상 h_i ^s 를 생성하고 s와 h_i ^s의 영상 차를 이용하여 얼굴 특징 y_i ^s이 표정의 개수(=C) 만큼 정의된다. 도 4를 참고하여, i번째 표정 클래스 내 변화 영상 h_i ^s를 이용하여 얻어진 M개의 훈련 얼굴 특징들 y_i ^s로 구성된 딕셔너리를 A_i∈R^NXM라고 하자.First, we need to define a dictionary consisting of trained facial expression features for classifying rare expressions. Training face images

(I = 1, ..., C) and the equations (1), (2) and (3) to extract the training facial features from the training facial image set. This is exactly the same as the method of extracting face features from a query face image. Specifically, when extracting facial features from one training facial image s, we generate a change image h _i ^s in the facial class and use the image difference of s and h _i ^s to determine the face feature y _i ^s by the number of facial expressions (= C ). 4, the change in the i-th facial expression class Let A _i ^∈ R ^NXM be a dictionary consisting of M training face features y _i ^s obtained using the image h _i ^s .

를 구해야 한다.A sparse coefficient vector for expressing the query face feature y _i ^q using the dictionary A _i by solving the L1 norm minimization problem defined in Equation 4 to determine the facial expression class of the query facial image q,

.

를 얻을 수 있다. In this equation, ε is a noise term with a small amount of energy. Using C change images in the facial expression class according to Equation (4), C sparse coefficient vectors

Can be obtained.

Next, the fusion unit 133 fuses the plurality of rare coefficient vectors to obtain the fused rare-coefficient vector (S209).

의 상호 보완적인(complementary) 정보를 인식에 활용하기 위해 수학식 5와 같이 C개의 희소 계수 벡터

를 융합하여 융합된 희소 계수 벡터

를 얻을 수 있다.Sparse coefficient vectors obtained by using change images in different facial classes

To compute the complementary information of the C sparse coefficient vector < RTI ID = 0.0 >

Lt; RTI ID = 0.0 > fused < / RTI >

Can be obtained.

에 의해 각

는

를 생성할 때 동일한 기여를 한다.In this equation,

Respectively,

The

The same contribution is made.

Next, the classifying unit 135 finds the facial expression class in which the sparse coefficient is most concentrated in the fused sparse coefficient vector, and determines the facial expression class of the query facial image (S211).

에서 가장 희소 계수가 집중된 표정 클래스를 찾아 쿼리 얼굴 영상 q의 표정 클래스를 결정한다.As shown in Equation (6), the fused sparse coefficient vector

The facial expression class of the query face image q is determined by finding the facial expression class in which the most rare coefficients are concentrated.

에서 i번째 표정 클래스 내 j번째 훈련 얼굴 영상에 관계된 희소 계수값이다.Where x ^com _{i , j} is the fused sparse coefficient vector

Is the sparse coefficient value related to the jth training face image in the ith face expression class.

Finally, the classifying unit 135 outputs the label of the determined facial expression class (S213).

As described above, by using the facial feature extraction and classification method that removes the variation components in the facial expression class that are not associated with facial expressions appearing in the query facial image, the facial feature recognition performance is stable even if there are various changes in the query facial image.

Combinations of each step of the flowchart and each block of the block diagrams appended to the present invention may be performed by computer program instructions. These computer program instructions may be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus so that the instructions, which may be executed by a processor of a computer or other programmable data processing apparatus, And means for performing the functions described in each step are created. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

The foregoing description is merely illustrative of the technical idea of the present invention and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (10)

A facial feature classification method performed in a facial feature classification system for facial expression recognition,
Generating a variation image in a facial expression class corresponding to a variation component in a facial expression class using training facial images configured for each facial expression class for a query facial image;
Calculating an image difference between the change image in the facial expression class and the query face image to define the face feature;
And classifying the facial features into a specific facial expression class by applying a rare expression to determine a facial expression class corresponding to the query facial image. The method according to claim 1,
Wherein the change image in the facial expression class is obtained by approximating the query face image using a linear combination of the training face images. 3. The method of claim 2,
The variation image within the facial expression class is generated by using a normalized least squares method to obtain each weight vector representing a weight for a linear combination of each training face image and using each weight vector and each training face image. Facial Expression Feature Classification Method for Facial Expression Recognition. The method according to claim 1,
The step of determining the facial expression class includes:
Obtaining a plurality of sparse coefficient vectors for expressing facial features of the query face image using a dictionary composed of face features of the training face image;
Obtaining a fused sparse coefficient vector by fusing the plurality of sparse coefficient vectors;
Determining a facial expression class in which a sparse coefficient is most concentrated in the fused sparse coefficient vector as a facial expression class of the query facial image. 5. The method of claim 4,
Wherein the dictionary defines a face feature as many as the number of facial expressions using an image difference between the training facial image and the change image in the facial expression class. A change image generation unit for generating a change image in a facial expression class corresponding to a change component in a facial expression class using training facial images configured for each facial expression class,
An image difference calculation unit for calculating an image difference between the variation image in the facial expression class and the query face image to define the image difference as a face feature;
And an image classifier for classifying the facial features into a specific facial expression class by applying a rare expression and determining a facial expression class corresponding to the query facial image. The method according to claim 6,
Wherein the change image in the facial expression class is obtained by approximating the query face image using a linear combination of the training face images. 8. The method of claim 7,
The variation image within the facial expression class is generated by using a normalized least squares method to obtain each weight vector representing a weight for a linear combination of each training face image and using each weight vector and each training face image. Facial Expression Feature Classification System for Facial Expression Recognition. The method according to claim 6,
The image classification unit may include:
A sparse expression applying unit for obtaining a plurality of sparse coefficient vectors for expressing a face feature of the query face image using a dictionary composed of face features of the training face image,
A fusion unit for fusing the plurality of sparse coefficient vectors to obtain a fused sparse coefficient vector,
And a classifier for finding a facial expression class in which the sparse coefficient is most concentrated in the fused sparse coefficient vector and determining the facial expression class of the query facial image as the facial expression class of the query facial image. 10. The method of claim 9,
Wherein the dictionary defines face features as many as the number of facial expressions using an image difference between the training face image and the change image in the facial expression class.

Images