JP2020091559A - Expression recognition device, expression recognition method, and program - Google Patents

Abstract

To provide an expression recognition device that can recognize expression with high accuracy even during utterance.SOLUTION: An expression recognition device 1 comprises: a face image obtaining unit 10 which obtains a face image; a voice obtaining unit 12 which obtains speech voice; a speech voice recognition unit 13 which obtains phonemes constituting the obtained voice; a lip shape change DB 16 which stores data regarding a lip shape change corresponding to phonemes; a lip shape change calculation unit 15 which reads data corresponding to phonemes obtained by the speech voice recognition unit 13 from the lip shape change DB 16, and generates a face image in which a lip shape change corresponding to the phonemes has been canceled from the face image obtained by the face image obtaining unit 10; and an expression identifier 17 which estimates expression based on a face image obtained by the lip shape change calculation unit 15.SELECTED DRAWING: Figure 1

Description

The present invention relates to a facial expression recognition device, a facial expression recognition method, and a program for recognizing human facial expressions.

Various products have already been put into practical use as technology for recognizing human facial expressions based on image information. Many of them extract image feature points from face images, and use corresponding discriminators that have been trained with a large number of face images in advance to correspond to corresponding facial organ points (eyes, nose, mouth, which were previously determined for face recognition. The position in the image (identification points such as both ends) is detected to detect the orientation of the face and the state of opening and closing the mouth, opening and closing of the eyes, and the face image prepared as learning data (joy, anger, sadness, etc.) Emotion estimation is performed based on which emotion data in the data set is closest.

As another application example, instead of using a general visible light image (RGB image) as image data, a near infrared camera image is used to make it robust against changes in external ambient light, and a plurality of cameras. Using an image (stereo camera, etc.), depth information is estimated from the parallax of each image, and depth information is added to the feature points extracted from the image data for three-dimensional identification processing, or a random dot pattern. There is a device using a depth camera sensor that irradiates an image pattern such as with infrared light and acquires more accurate depth information from the parallax change of the pattern reflected on the face. In addition, although the types of input image data, etc. are different, the basic configuration such as the subsequent recognition of changes in face position and organ points by the discriminator, similarity/classification determination processing based on learning data including emotion label data, etc. will change greatly. Absent.

JP, 2017-120609, A

When recognizing facial expressions, especially when recognizing facial expressions while interacting with the user, changes in the mouth area due to utterances affect the accuracy of face recognition. In particular, the learning data used for learning the face recognition discriminator is based on emotion labels (joy, anger, sadness, etc.) given to the still image as correct values, and changes in mouth shape during utterance. Since the data corresponding to is not included in the learning data, the facial expression cannot be correctly recognized due to the subtle changes in the mouth.

To solve this problem, the learning image is divided into the entire face and the image excluding the mouth area, and when utterance is detected, the model is switched to the learned model that uses only the upper half of the face as the learning image. A technique for recognizing emotions has been proposed (Patent Document 1). However, if the information on the lips is cut off uniformly, the emotional information shown on the lips will not be used.

Therefore, the present invention provides a facial expression recognition device that can recognize facial expressions with high accuracy even during utterance.

The facial expression recognition device of the present invention includes a face image acquisition unit that acquires a face image, a voice acquisition unit that acquires a spoken voice, a voice recognition unit that obtains phonemes that form the acquired voice, and a lip shape corresponding to the phoneme. A database that stores data relating to changes, and data corresponding to phonemes obtained by the speech recognition unit are read from the database, and from the face image acquired by the face image acquisition unit, changes in lip shape corresponding to the phonemes. A lip shape change calculation unit that generates a face image that has been canceled, and a facial expression discriminator that estimates a facial expression based on the face image obtained by the lip shape change calculation unit.

With this configuration, in addition to the image information being acquired by the face image acquisition unit, the speech acquisition unit simultaneously acquires the utterance voice, and the utterance phoneme recognition by the voice signal processing and the face shape change data corresponding to the utterance phoneme. The facial image obtained from the input image can be recognized with high accuracy even during the utterance by performing facial expression recognition processing after applying the operation to suppress the change and deformation due to these utterances to the face image obtained from the input image. ..

The facial expression recognition device of the present invention may include a data pipeline that integrates a face image acquired by the face image acquisition unit and a time series of phoneme data obtained by the speech recognition unit. With this configuration, it is possible to synchronize the face image with the uttered voice and appropriately cancel the movement of the lips corresponding to the utterance.

In the facial expression recognition device of the present invention, the lip shape change calculation unit may interpolate data read from the database in a section in which the phoneme does not exist, and cancel the change in lip shape using the interpolated data. .. Here, as the data interpolation, linear interpolation or interpolation using a spline curve may be performed. As a result, the movement of the lips corresponding to the utterance can be canceled even in the section in which no phoneme exists.

In the facial expression recognition apparatus of the present invention, the database stores data relating to changes in the default lip shape and coefficients applied to the data, and the coefficients are set for each user by initial calibration. Good. As a result, the movement of the lips can be canceled according to the movement of the lips that differs depending on the user.

In the facial expression recognition device of the present invention, the database stores data relating to a change in the default lip shape, and a coefficient applied to the data, and the coefficient is a face image in which the change in the lip shape is canceled. It may be corrected with a policy of not causing an error. Accordingly, even if the user does not perform calibration in advance, it is possible to set a coefficient that matches the movement of the lips that differs depending on the user.

In the facial expression recognition apparatus of the present invention, the database may store data corresponding to a bigram including data of either preceding or following phonemes or a trigram including data of both preceding and following phonemes. Even if the same phoneme is uttered, the movement of the lips may differ depending on the front and back phonemes, but the configuration of the present invention can appropriately cancel the movement of the lips including the front and back phonemes.

In the facial expression recognition device of the present invention, the voice acquisition unit is a microphone array having a plurality of microphones, and when face images of a plurality of people are acquired by the face image acquisition unit, a person associated with each face image utters. The voice may be identified by a microphone array, and the facial expression identification unit may identify the facial expression of the face image using the identified voice. With this configuration, the movement of the lips can be canceled even when a plurality of people are speaking.

The facial expression recognition method of the present invention is a method for recognizing an facial expression from a facial image by a facial expression recognition device, wherein the facial expression recognition device acquires a facial image, and the facial expression recognition device acquires a speech voice. And a step in which the facial expression recognition device obtains a phoneme that forms the acquired voice, and the facial expression recognition device stores data corresponding to the obtained phoneme from a database that stores data relating to changes in the lip shape corresponding to the phoneme. The step of generating a face image in which the change in the lip shape corresponding to the phoneme is read out from the face image, and the facial expression recognition device estimates the facial expression based on the face image in which the change in the lip shape is canceled. And performing steps.

The program of the present invention is a program for recognizing facial expressions from a face image, and includes a step of obtaining a face image, a step of obtaining a speech voice, a step of obtaining a phoneme configuring the obtained voice, Data corresponding to the phoneme is read from a database that stores data related to changes in the lip shape corresponding to the phoneme, from the face image, a step of generating a face image in which the change in the lip shape corresponding to the phoneme is canceled, And a step of estimating a facial expression based on the face image in which the change in the lip shape is canceled.

According to the present invention, facial expressions can be appropriately recognized even during utterance.

It is a figure which shows the structure of the facial expression recognition apparatus of 1st Embodiment. (A) It is a figure which shows the result of unifying the time series of facial feature point data and phoneme data. (B) is a diagram showing an example of a case where there is a request for facial expression recognition processing data in a section where there is no phoneme data. It is a figure which shows the data memorize|stored in lip shape change DB of 1st Embodiment. It is a figure which shows the change of the lip shape at the time of uttering two phonemes. It is a flow chart which shows operation of the facial expression recognition device of a 1st embodiment. It is a figure which shows the structure of the facial expression recognition apparatus of 2nd Embodiment. It is a figure which shows the data memorize|stored in lip shape change DB of 3rd Embodiment.

Hereinafter, the facial expression recognition device according to the embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing the configuration of the facial expression recognition device 1. The facial expression recognition device 1 includes a face image capturing unit 10, a facial feature point recognition unit 11, a speech input unit 12, a speech phoneme recognition unit 13, a data pipeline 14, a lip shape change calculation unit 15, and a lip. A shape change database (hereinafter referred to as “lip shape change DB”) 16 and a facial expression identification unit 17 are provided.

The face image capturing unit 10 is a camera having a function of capturing a user whose facial expression is to be recognized. Specifically, it is a visible light camera, a near infrared camera, a camera with depth information by a plurality of cameras, a depth camera, or the like. The face image capturing unit 10 captures a face image and sends the face image data to the next face feature point recognition unit 11.

The facial feature point recognizing unit 11 uses the face image data acquired by the face image capturing unit 10 to identify the feature points indicating the positions of facial organ points (eyes, nose, mouth, etc.) and the predefined organ point positions such as the contour portion. The position of is estimated and extracted. Each of the obtained facial feature points includes a three-dimensional coordinate position. Here, a method of acquiring the three-dimensional coordinate position will be described.

When measured as a three-dimensional point including depth in advance such as a depth camera, the three-dimensional position of the feature point can be obtained without performing the three-dimensional estimation. On the other hand, when only the position on the two-dimensional image data is obtained at the time of estimating and detecting the organ point according to the face image capturing device, the three-dimensional posture is estimated from the position of each organ point mapped on the two-dimensional image. Then, a three-dimensional position including the depth direction of each organ point is obtained.

The face feature point recognizing unit 11 records a recognized point group and a face shape in advance when a face is in a horizontal direction or when a part of the face is hidden by a shield and an organ point position is not included in the image. An estimated position is obtained based on the positional relationship information between the feature points thus obtained, and is used as an output result. The face feature point recognition unit 11 estimates the face orientation of the entire face based on the face contour and the feature point position of each organ point, and also outputs the result as an output result.

The uttered voice input unit 12 has a function of inputting the uttered voice of the user whose facial expression is to be recognized. Specifically, it is a microphone, for example. The utterance voice input unit 12 transmits voice data to the next utterance phoneme recognition unit 13.

The utterance phoneme recognition unit 13 has a function of estimating a utterance phoneme symbol string from the utterance speech signal and outputting the symbol string. As for a method of estimating a phoneme symbol string from a speech, a method of estimating from a formant in a speech signal processing, a method of estimating based on a probability statistical model (GMM-HMM), and learning and identification of phoneme features using a deep learning model. Can be used (DNN_HMM) or the like (Keiichi Tokuda, “Voice Recognition and Speech Synthesis by Hidden Markov Model”, IPSJ Magazine Vol.45 No.10 Oct 2004). The present embodiment does not limit the specific method for recognizing a phoneme. It should be noted that the definition of the phoneme of the recognition result may be any as long as it corresponds to the phoneme data stored in the lip shape change DB 16 described later.

The data pipeline 14 integrates the output results from the facial feature point recognition unit 11 and the speech phoneme recognition unit 13 in time series, and holds the data for a certain period of time. The data of the facial feature points obtained from the face image and the data of the phonemes obtained from the uttered speech signal have different recognition processing times, so that the recognition result may not always be obtained at the same time. When applying the following process, it is necessary to process each recognition result as a pair corresponding at the same time in each real time. The data pipeline 14 records and saves the recognition results of the facial feature points and the speech phonemes for a certain period of time. As a result, both data corresponding to each designated time can be acquired in the latter stage processing.

FIG. 2A is a diagram showing a result of unifying the time series of the facial feature point data and the phoneme data. In FIG. 2A, the shaded area is the time when the data of the recognition result exists. As shown in FIG. 2A, the data pipeline 14 saves data by synchronizing the times at which the recognition results are obtained.

The lip shape change calculation unit 15 has a function of canceling the lip shape change due to the utterance from the face image and calculating the face image when it is determined that the utterance is not made. Specifically, the positions of the facial feature points recognized from the facial image are corrected by the amount of lip movement based on the utterance. The data stored in the lip shape change DB 16 is used for this calculation.

FIG. 3 is a schematic diagram showing the data stored in the lip shape change DB 16. The lip shape change DB 16 is a database that stores the amount of change in facial feature points such as lips and jaws (referred to as “lip shape change data” in this document) corresponding to speech phonemes. The lip shape change data is data for canceling the movement of the lip corresponding to the phoneme.

For example, when the phoneme is “a”, the pronunciation is made with the mouth wide open, so that the characteristic points of the lower lip and the chin move downward when viewed from the state of not speaking. In order to cancel the movement of the lip of the utterance "a" and correct the face image when not uttered, the characteristic points of the lower lip and the jaw are moved upward as shown in FIG. Further, when the phoneme is "mu", the upper and lower lips are closed tightly to produce a pronunciation, so that the characteristic points of the upper and lower lips are moving closer to each other when viewed from a state where no utterance is made. In order to cancel the movement of the lip caused by “mu” and correct the face image when not speaking, move the upper lip slightly upward and move the lower lip slightly downward as shown in FIG. ..

The lip shape change calculation unit 15 acquires the phoneme data at the time corresponding to the facial feature point from the data stored by the data pipeline 14, and reads the lip shape change data corresponding to the read phoneme from the lip shape change DB 16. By applying the read lip shape change data to the facial feature points, the facial feature point data in which the movement of the lips is canceled is obtained.

The lip shape change data recorded in advance for each phoneme is in the front direction, and cannot be directly applied to face images having different face orientations, the size of the entire face, and the like. The lip shape change calculation unit 15 calculates a personal face size from the face orientation and front and rear edges, and left and right edges based on the feature points acquired at the time of face feature identification, and when applying the shape change, the lip shape change calculation unit 15 The conversion is applied to the lip change amount read from the lip shape change DB 16, and then the lip shape deformation is subtracted from the original facial feature points.

As shown in FIG. 2A, regarding the phoneme data, there is a section where there is no recognized phoneme data in the intermediate state of the front and rear recognized phonemes. For example, as shown in FIG. 2B, there is a case where there is a request for facial expression recognition processing data in a section where there is no phoneme data. In that case, the lip change amount data in the intermediate state is compared with the preceding and following phoneme data. Generate it in the original. Here, the process of obtaining the intermediate state will be described.

FIG. 4 is a diagram showing changes in the lip shape when uttering two phonemes. Since the utterance is not uttered for each phoneme but uttered continuously, the lip shape between the recognized phonemes is an intermediate state in which the lip shape is gently changed from the lip shape of the immediately preceding phoneme. Therefore, the lip shape state based on the phoneme of each time frame is synthesized and used as the intermediate state of the phonemes recognized before and after. When an intermediate state is created from two shape states (three-dimensional coordinate positions of feature points), the amount of change in the coordinate position of the preceding and following shape states divided by the number of frames corresponding to the change time is appropriately added. Can be generated by For example, when the phonemes before and after are “A” and “Mu”, the lip shape change data corresponding to “A” stored in FIG. 3 and the lip shape change data of “Mu” are interpolated, The lip shape change data of the lip movement in the intermediate state can be calculated.

As for the method of interpolation of this lip shape change data, in addition to the method of simply linearly dividing it (linear interpolation), the amount of change before and after the start is defined in advance as a change between phonemes with a spline curve, etc. There are several ways to achieve this. Which method is used can be selected by defining and recording a data set that defines the lip shape corresponding to the phoneme.

The facial expression identification unit 17 identifies facial expressions by inputting feature point data of the entire face, which is obtained by subtracting the facial shape deformation due to the lip shape change by the lip shape change calculation unit 15. As an identification method, there are a method based on a statistical method, a method using a classifier learned by deep learning, etc., but in the present embodiment, any method can be used as long as it corresponds to the input facial feature point data. Such a method may be used.

FIG. 5 is a flowchart showing the operation of the facial expression recognition device 1 according to the first embodiment. In the facial expression recognition device 1, the face image capturing unit 10 captures a face image of the recognition target user (S10), and recognizes the feature points of the facial organs from the captured face image (S11). Further, the facial expression recognition apparatus 1 collects and inputs the utterance at the same time as the face image is captured by the utterance voice input unit 12 (S12). Then, the facial expression recognition device 1 recognizes a phoneme included in the input speech voice (S13). The facial expression recognition device 1 stores the facial organ feature points and the phonemes in synchronization with each other based on the time when the feature points of the facial organs are obtained and the time when the phonemes of the speech are obtained by the data pipeline 14 (S14). ).

Next, the facial expression recognition device 1 identifies the data of the phoneme uttered when the face image for which the facial expression is desired to be captured is captured, and the lip shape change data corresponding to the read phoneme is read from the lip shape change DB 16 (S15). .. Here, when there is no phoneme data uttered when the face image is captured (the case shown in FIG. 2B), the phoneme data before and after is read out, and the lip shape corresponding to the data of the phonemes before and after is read out. The change data is interpolated to obtain lip shape change data. The facial expression recognition device 1 cancels the influence of the movement of the lips caused by utterance by applying the lip shape change data to the facial feature points (S16).

Subsequently, the facial expression recognition device 1 recognizes the facial expression by using the face image in which the influence of the movement of the lips caused by the utterance is canceled (S17).

The configuration of the facial expression recognition device 1 according to the present embodiment has been described above, but the example hardware of the facial expression recognition device 1 includes a CPU, a RAM, a ROM, a hard disk, a display, a keyboard, a mouse, a communication interface, and the like. A computer, a camera for capturing face images, and a microphone for inputting speech. The facial expression recognition device 1 described above is realized by storing a program having a module that realizes each of the functions described above in a RAM or a ROM and executing the program by the CPU. Such a program is also included in the scope of the present invention.

The facial expression recognition apparatus 1 according to the present embodiment obtains the lip shape change from the uttered voice and recognizes the facial expression based on the face image from which the lip shape change data is subtracted, so that the facial expression can be accurately recognized even during utterance. .. Further, by adopting a configuration in which the facial image in which the change in the lip shape has been canceled is passed to the facial expression identification unit 17, the conventional facial expression recognition method can be used as the configuration of the facial expression identification unit 17.

(Second embodiment)
FIG. 6 is a diagram showing the configuration of the facial expression recognition device 2 according to the second embodiment. There are individual differences in the movement of the lips due to vocalization. The facial expression recognition device 2 of the second exemplary embodiment reflects the individual difference and more accurately cancels the influence of the movement of the lips caused by utterance.

Although the basic configuration of the facial expression recognition device 2 of the second exemplary embodiment is the same as that of the first exemplary embodiment, the facial expression recognition device 2 of the second exemplary embodiment has a correction coefficient database (hereinafter, referred to as “correction coefficient database”). 18). The correction coefficient DB 18 is a coefficient that determines how strongly the lip shape change data stored in the lip shape change DB 16 is reflected for each user when canceling the influence of lip movement due to utterance. I remember.

This coefficient can be set in the initial setting of the facial expression recognition device 2 by shooting an image in which the user speaks a phoneme and using the image. If the initial setting is complicated, the coefficients may be calibrated gradually while using the facial expression recognition device 2. If the correction coefficient is not appropriate, when the lip shape change data multiplied by the correction coefficient is applied to the face image of the user, the face image may become an error. An easy-to-understand example is that, as a result of moving the lower lip feature point upward, the lower lip feature point is located above the upper lip feature point. When such an error occurs, the correction coefficient is adjusted based on the policy of reducing the degree of moving the characteristic point of the lower lip upward. In this way, it is possible to gradually perform calibration while recognizing facial expressions. The other configurations and operations of the facial expression recognition device 2 of the second embodiment are the same as those of the first embodiment.

The facial expression recognition device 2 according to the second embodiment obtains the lip shape change of each user from the uttered voice, and recognizes the facial expression based on the face image from which the lip shape change data is subtracted, so that the facial expression recognition device 2 can be used even during utterance. Can recognize facial expressions with high accuracy.

In the present embodiment, the configuration in which the general lip shape change DB 16 and the correction coefficient DB 18 are combined to deal with the user's individual difference has been described, but each user may have the lip shape change DB 16.

(Third Embodiment)
Next, the facial expression recognition device 3 according to the third embodiment will be described. The facial expression recognition device 3 of the third embodiment has the same basic configuration as the facial expression recognition device 1 of the first embodiment, but the data stored in the lip shape change DB 16 is different.

FIG. 7 is a diagram showing an example of data stored in the lip shape change DB 16 included in the facial expression recognition device 3 according to the third embodiment. In the first embodiment, the lip shape change DB 16 stores the lip shape change data corresponding to one phoneme, but in the present embodiment, the middle phoneme of three consecutive phonemes (trigrams) is stored. The lip shape change data corresponding to is stored. Even for the same phoneme, the movement of the lip differs depending on the front and back phonemes.Therefore, by having the lip shape change data including the front and back phonemes, it is possible to specify the lip shape change data corresponding to the more accurate phoneme. it can. Other configurations and operations of the facial expression recognition device 3 of the third embodiment are the same as those of the first embodiment.

The facial expression recognition device 3 according to the third embodiment obtains the lip shape change of each user from the uttered voice, and recognizes the facial expression based on the face image from which the lip shape change data is subtracted, so that the facial expression recognition device 3 can be used even during utterance. Can recognize facial expressions with high accuracy.

Here, the trigram is taken as an example, but as a modified example, the lip shape change data corresponding to the latter phoneme of the two consecutive phonemes, or the previous phoneme of the two consecutive phonemes is supported. It may have the lip shape change data for

Although the facial expression recognition apparatus of the present invention has been described in detail above with reference to the embodiments, the present invention is not limited to the above-described embodiments.

It is also assumed that the image captured by the face image capturing unit 10 includes a plurality of face images. In such a case, it is necessary to specify the uttered voice of the recognition target user so that the facial expression recognition device can cancel the influence of the movement of the lip from the face image of the recognition target user. The facial expression recognition device of the present invention may use a microphone array including a plurality of microphones as the utterance voice input unit 12. The direction of the sound source can be identified and the speech can be separated by the microphone array, and the speech of the recognition target user can be identified. As a result, similar to the above-described embodiment, by obtaining lip shape change data from the uttered voice and canceling it from the face image, it is possible to perform accurate facial expression recognition even when there are multiple users.

In the above-described embodiment, the lip shape change is detected from the uttered voice, but in addition to this, the data obtained by adding the phoneme phoneme and the lip change in the intermediate state to the data set of the still image for facial expression recognition are used. The accuracy may be further improved by generating it as learning data and using this to generate a model for estimating lip changes.

INDUSTRIAL APPLICABILITY The present invention is useful as a device for recognizing its facial expression from a face image.

1, 2 facial expression recognition device, 10 face image capturing unit, 11 face feature point recognition unit,
12 speech input unit, 13 speech phoneme recognition unit, 14 data pipeline,
15 lip shape change calculation unit, 16 lip shape change DB, 17 facial expression identification unit,
18 Correction coefficient DB.

Claims (10)

A face image acquisition unit that acquires a face image,
A voice acquisition unit that acquires the utterance voice,
A speech recognition unit that obtains phonemes that compose the acquired speech;
A database that stores data on changes in lip shape corresponding to phonemes,
Lip corresponding to the phoneme obtained by the voice recognition unit is read from the database, and a face image obtained by canceling the change in the lip shape corresponding to the phoneme is generated from the face image obtained by the face image obtaining unit. A shape change calculation unit,
An expression discriminator that estimates an expression based on the face image obtained by the lip shape change calculation unit,
Facial expression recognition device. The facial expression recognition device according to claim 1, further comprising a data pipeline that integrates a face image acquired by the face image acquisition unit and a time series of phoneme data obtained by the voice recognition unit. The said lip shape change calculation part interpolates the data read from the said database in the area where the said phoneme does not exist, and cancels the change of a lip shape using the interpolated data. Facial expression recognition device. The facial expression recognition device according to claim 3, wherein the lip shape change calculation unit performs linear interpolation or interpolation using a spline curve. 5. The database stores data on changes in default lip shape and coefficients applied to the data, and the coefficients are set for each user by initial calibration. Facial expression recognition device described in. The database stores data on changes in the default lip shape and coefficients applied to the data, and the coefficient is designed to prevent a face image in which changes in the lip shape have been canceled from becoming an error. The facial expression recognition device according to claim 1, which is corrected. The facial expression recognition device according to claim 1, wherein the database stores data corresponding to a trigram including data of phonemes before and after. The voice acquisition unit is a microphone array having a plurality of microphones,
When face images of a plurality of people are acquired by the face image acquisition unit, a voice uttered by a person related to each face image is specified by a microphone array,
The facial expression recognition device according to any one of claims 1 to 7, wherein the facial expression identification unit identifies the facial expression of the face image using the identified voice. A method of recognizing a facial expression from a facial image by a facial expression recognition device,
The facial expression recognition device acquires a facial image;
The facial expression recognition device acquires a speech voice,
The facial expression recognition device obtains phonemes forming the acquired voice;
The facial expression recognition device reads out the data corresponding to the obtained phoneme from the database that stores the data regarding the change in the lip shape corresponding to the phoneme, and the face in which the change in the lip shape corresponding to the phoneme is canceled from the face image. Generating an image,
The facial expression recognition device estimates a facial expression based on the facial image in which the change in the lip shape has been canceled;
Facial expression recognition method. It is a program that recognizes facial expressions from face images.
A step of acquiring a face image,
A step of acquiring a speech voice,
A step of obtaining phonemes that compose the acquired speech,
Data corresponding to the obtained phoneme is read from a database that stores data relating to changes in the lip shape corresponding to the phoneme, and a step of generating a face image in which the change in the lip shape corresponding to the phoneme is canceled from the face image, ,
A step of estimating a facial expression based on the face image in which the change in the lip shape is canceled,
A program to execute.

Images