JP2001043345A - Facial expression recognition device, medication control system using it, arousal level evaluation system, recovery evaluation system - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To obtain a facial expression recognition device capable of recognizing a degree of facial expression such as strength and intensity of a predetermined human expression such as pain and vitality. SOLUTION: An expression transition map generating means 12 for generating an expression transition map 11 composed of a plurality of face images in which the expression changes stepwise for an expression which changes according to a predetermined emotion or a human state, Face image input means 14 for inputting a real face image 15 to be evaluated for the degree of facial expression
A matching unit 16 for matching a real image input from the face image input unit with a face image at each stage of a previously generated expression transition map; and a face image that is the most similar from the matching result of the matching unit. And a facial expression determining means 17 for determining a degree of a predetermined facial expression.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for comparing a real facial image of a target person with a facial expression transition map described for each degree of a certain facial expression and performing a judgment process on the predetermined facial expression of the person from the real facial image. The present invention relates to a facial expression recognition device for recognizing a degree, and a medication control system, an awakening level evaluation system, and a recovery evaluation system using the facial expression recognition device.

[0002]

2. Description of the Related Art FIG.
FIG. 9 is a block diagram showing a conventional expression recognition device disclosed in Japanese Patent Publication No. 8-No. In the figure, 1 is a measurement module for measuring the movement of the epidermis such as a face, and 2 is this measurement module 1
Is a converter for converting the measurement result of the above into a moment. Reference numeral 3 denotes a processing unit that processes the output of the converter 2 to calculate a feature vector, and 4 denotes a storage unit that stores the feature vector. 5 is the distance between the feature vector calculated by the processing unit 3 at the time of recognition and the feature vector stored in the storage unit 4,
It is a distance calculation unit that recognizes what facial expression.

Next, the operation will be described. A time-series face image of a face presenting a predetermined expression is input to the measurement module 1, and the movement of the epidermis related to the generation of the expression is extracted as optical flow data. At the time of learning, the data extracted by the measurement module 1 is input to the converter 2 to convert the temporal change pattern of the movement of the epidermis into first and second moments. Next, this converter 2
The first and second moments converted in step (1) are input to the processing unit 3, and a feature vector for identifying a facial expression is determined based on the first and second moments, and is stored in the storage unit 4.

On the other hand, at the time of recognition, a feature vector is calculated from a given facial expression time-series face image to be recognized in a similar procedure, and the calculated feature vector is input to the distance calculation unit 5. The distance calculation unit 5 compares the feature vector of the recognition target from the processing unit 3 with the feature vector stored in the storage unit 4 and selects a standard expression close to the recognition target based on the distance between the vectors. I do. The distance calculation unit 5 outputs a recognition result of the facial expression time-series face image to be recognized based on the selected standard facial expression.

As described above, in the conventional expression recognition apparatus disclosed in Japanese Patent Application Laid-Open No. 4-32078, a minute movement of a muscle is measured over the entire face, its temporal change is patterned, and based on the pattern. A technique for recognizing facial expressions representing emotions is disclosed.

In addition, as a conventional expression recognition device, Japanese Patent Laid-Open No. 3-25277 discloses a method of recognizing an expression by patterning the movement of a muscle and comparing it with a standard expression pattern.
"Face impression analysis using average face" which extracts a parameter indicating an impression from an average face of a plurality of persons having the same occupation type or a plurality of persons having the same occupation type (Nagata, Kaneko,
Harashima: IEICE Transactions A, Vol. 80-
A, No. 8, pp. 1266-1272).

[0007]

Since the conventional facial expression recognition device is configured as described above, when recognizing the facial expression of a person, most of the facial expression recognition devices also include facial expressions such as laughter, anger, joy, and sadness. It deals with emotions that are significantly different (such expression recognition is referred to as expression recognition by category). For example, if it is desired to grasp the state of a patient during rehabilitation, such expression by category is used. It is necessary to recognize not the recognition but the strength and intensity of a person's predetermined facial expression such as painfulness (this is called recognition of the degree of facial expression). Although there is a recognition device, there is no conventional expression recognition device for recognizing the degree of the expression, and the degree in each state is determined at a certain moment. It may be difficult to determine only with boric.

[0008] Furthermore, an effective conventional technique for judging and describing the state of each degree of facial expression using a facial expression recognition device and reducing the labor of describing a plurality of states in which the degree of facial expression is stepwise changed. No conventional technology is found, and if the direction of the facial image that describes the state of each degree of facial expression or the direction of the real face image to be recognized does not match, the processing is complicated to deal with it. There were issues such as becoming.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and is a facial expression recognition apparatus capable of recognizing the degree of facial expression such as the strength and intensity of a predetermined facial expression such as pain and vitality. The aim is to obtain a device.

It is another object of the present invention to provide a medication control system that determines the state of each degree of facial expression using a facial expression recognition device and evaluates the medication effect on a patient based on the result of the determination.

It is a further object of the present invention to obtain an awakening level evaluation system for determining the state of each degree of facial expression using a facial expression recognition device and evaluating the arousal level of the worker based on the result of the determination.

It is a further object of the present invention to provide a recovery evaluation system that determines the state of each degree of facial expression using a facial expression recognition device and evaluates a patient's recovery state or the like based on the determination result.

[0013]

According to the present invention, a facial expression recognition apparatus according to the present invention includes a facial image to be determined input by a facial image input unit and a facial expression transition map generated in advance by a facial expression transition map generating unit. The face image at each stage is compared with the matching unit, the most similar face image is selected, and the stage is output from the determining unit to determine the degree of the predetermined facial expression. .

The facial expression recognition apparatus according to the present invention uses time-series facial image data of facial expressions as face images at each stage of the facial expression transition map, and compares the time-series facial image data input from the facial image input means with the facial image data. It is something to do.

The facial expression recognition apparatus according to the present invention uses an average face generated by averaging the facial images of a plurality of persons for each facial expression in each stage of the facial expression transition map. is there.

The facial expression recognition apparatus according to the present invention uses an average face generated by averaging a specific individual's facial image for each facial expression at each stage of the facial expression transition map. is there.

The facial expression recognition device according to the present invention generates face images in intermediate states of each degree by interpolation or extrapolation processing from a plurality of facial images having different degrees of facial expression, thereby increasing the number of stages. An expression transition map having a corresponding face image is generated.

The facial expression recognition apparatus according to the present invention is characterized in that one or more of a subjective rating value of the degree of the emotion of the subject himself / herself, a rating value of a third party, and a physiological index value changing according to the emotion or state. Is used to determine the degree of facial expression of the face image used to generate the facial expression transition map.

The facial expression recognition apparatus according to the present invention uses a plurality of evaluation values such as the subjective evaluation value, the evaluation value by a third party, and the physiological index value to generate a facial image transition map of the evaluation value. The degree of facial expression is determined, and a facial image with a high degree of matching is selected to generate a facial expression transition map.

An expression recognition apparatus according to the present invention generates a face image in a specific direction from face images taken from a plurality of directions,
The expression transition map is generated by using this.

The facial expression recognition apparatus according to the present invention receives a facial image photographed from a plurality of directions, generates a facial image in the same direction as the facial expression transition map, and uses the facial image in the facial expression transition map as a judgment target image. It is intended to be collated.

A medication control system according to the present invention determines the degree of facial expression of a patient by using any of the facial expression recognition devices, and evaluates the effect of medication to the patient based on the determination result. It was done.

A wakefulness level evaluation system according to the present invention determines the degree of sleepiness of a working person using any one of the above facial expression recognition devices, and evaluates the wakefulness level based on the determination result. It was made.

The recovery evaluation system according to the present invention determines the degree of facial expression of a patient by using any of the facial expression recognition devices described above, and evaluates the health recovery state and rehabilitation effect of the patient based on the determination result. It is like that.

[0025]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below. Embodiment 1 FIG. FIG. 1 is a block diagram showing a configuration of an expression recognition device according to Embodiment 1 of the present invention, and FIG. 2 is a conceptual diagram for explaining recognition of a degree of a predetermined expression in the expression recognition device. In both of these figures, reference numeral 11 denotes a facial expression transition map in which the degree of the facial expression is classified into a plurality of stages with respect to a facial expression that changes in accordance with a predetermined emotion or a human state. As shown in FIG. It is composed of a plurality of face images whose degree of expression changes stepwise. Reference numeral 12 denotes an expression transition map generating means for generating the expression transition map 11 in advance. 1
Reference numeral 3 denotes a target person whose expression degree is to be determined. Reference numeral 14 denotes a face image input unit for inputting a face image of the target person 13. In the example of FIG. 2, a camera is used as the face image input unit 14.

Reference numeral 15 denotes a photographed face image of the target person 13 captured by the face image input means 14 into the facial expression recognition device. Reference numeral 16 denotes matching means for checking the actual photographed face image 15 with the face image of each stage of the expression transition map 11 generated in advance. Reference numeral 17 denotes a facial expression judging unit that selects the most similar face image from the collation result by the collating unit 16 and outputs the stage (state identification number i) to determine the degree of the predetermined facial expression. . 18
Implements the functions of the facial expression transition map generating means 12, the matching means 16 and the facial expression determining means 17, and stores the facial expression transition map 11 previously generated by the facial expression transition map generating means 12 in its memory. It is.

Next, the operation will be described. First, the facial expression of the target person 13 is photographed by the face image input means 14 using a camera. The photographed face image 15 of the facial expression of the target person 13
Is input to the computer 18. In the computer 18, the matching means 1
In step 6, the similarity between the input real photographed face image 15 and the facial expression transition map 11 in which the degree of facial expression is classified into several stages is calculated, and the two are compared. Next, facial expression determining means 1
7, the most similar face image in the facial expression transition map 11 is selected based on the result of the comparison by the comparing means 16, and the facial expression of that degree is determined as the facial expression level of the photographed face image. The degree of 13 facial expressions is grasped.

The similarity calculated by the matching means 16 between the real face image 15 and the expression transition map 11 is calculated by calculating the pixel value of each pixel of the real face image 15 or the average pixel value of a plurality of pixels by one element. Expression transition map exhibiting a minimum norm using a vector having the following expression and a norm of a vector having the pixel value of each pixel of the average face image indicating each degree of the expression transition map 11 or the average pixel value of a plurality of pixels as one element. It is determined that the state of 11 is the degree to which the photographed face image 15 presents. This is represented by the following equation.

[0029]

(Equation 1)

Here, as a method of creating the expression transition map 11 by the expression transition map generating means 12, there is, for example, the following method. FIG. 3 is a conceptual diagram showing an example of generation of the expression transition map 11. As shown in FIG. 3, face images 21 having the same degree of facial expression are input for a plurality of persons, and an averaging process 22 is performed. The average face 23 generated by the averaging process 22 is described as one state of the expression transition map 11. By performing such an averaging process 22, it is possible to emphasize only features of facial expressions having the same degree and cancel other differences. This averaging process 22 is performed for each expression of each stage.
To generate the expression transition map 11 based on the obtained average face 23 at each stage.

In the above description, the facial expression transition map 11 is generated using an average face obtained by averaging the facial images of a plurality of persons for each facial expression. A plurality of facial expressions may be input for each facial expression at each stage, an average face may be obtained by the averaging process, and the obtained facial expression transition map 11 at each stage may be generated.

When the facial expression transition map generating means 12 generates the facial expression transition map 11, the determination of the facial expression at the time of generating the average face is based on a value subjectively evaluated as to the degree of emotion of the target person 13 himself. This is performed using one or more of a value evaluated by observing a face image by a third party, a physiological index value that changes according to a target emotion or a state of a person. As a result, it is possible to classify the degree of facial expression in accordance with human senses or to classify objectively. Note that a plurality of rating values such as the subjective rating value, the rating value of a third party, and a physiological index value were used, and only a face image having a high degree of matching between the plurality of rating values was selected and used. A facial expression transition map may be generated, which makes it possible to more accurately capture characteristics of a predetermined facial expression degree.

Further, the face image input means 14 is provided with means for photographing a face image from a plurality of directions.
When comparing the facial expression transition map 11 with the facial expression transition map 11, a facial image in the same direction as the facial expression transition map 11 is generated from the facial images in a plurality of directions input from the facial image input means 14, and this is used as a determination target image. Determination target image and facial expression transition map 11
May be collated by the collation unit 16 with each of the face images. Thereby, the photographed face image 15 and the expression transition map 11
Can be simplified.

Embodiment 2 Next, a second embodiment of the present invention will be described. FIG. 4 shows Embodiment 2 of the present invention.
FIG. 6 is a conceptual diagram showing an example of generation of a facial expression transition map in a facial expression recognition device according to the present invention. In the figure, 31 is a face image representing the maximum degree as one of a plurality of face images having different degrees of predetermined facial expressions, and 32 is another one of a plurality of face images having different degrees of predetermined facial expressions. Is a face image that represents the minimum degree as. Reference numeral 33 denotes an interpolation process for obtaining an intermediate state from the face image 31 representing the maximum degree and the face image 32 representing the minimum degree. Reference numeral 34 denotes a face image corresponding to more stages generated by the interpolation process. It is a face image representing an intermediate degree.

Next, the operation will be described. The face image input means 14 such as a camera captures a face image 31 representing the maximum degree of the predetermined facial expression and a face image 32 representing the minimum degree, and inputs them to the facial expression transition map generating means 12. When generating each state of the facial expression transition map 11, the facial expression transition map generating means 12 outputs a face image 31 representing the maximum degree input.
By performing the interpolation process 33 on the face image 32 representing the minimum degree, the face image 34 representing the intermediate degree of the expression is generated. This makes it possible to efficiently generate the intermediate state.

In the above description, the face image 31 representing the maximum degree and the face image 32 representing the minimum degree are input as a plurality of face images having different degrees of predetermined facial expressions. If a plurality of face images other than the face image representing the maximum or minimum are input as a plurality of face images having different degrees, and an interpolation process or an extrapolation process is performed between them, an intermediate A face image representing the degree, a face image representing the maximum degree, and a face image representing the minimum degree can be generated, and the expression transition map 11 can be accurately generated.

As described above, according to the second embodiment,
Since the intermediate state is obtained by interpolation or extrapolation processing from the states of different degrees of the facial expressions for the states of each degree of the facial expression transition map 11, the facial expression transition map 11
It is no longer necessary to directly obtain the state of the intermediate degree from the face images of a plurality of persons exhibiting the same degree or a specific individual.

Embodiment 3 Next, a third embodiment of the present invention will be described. FIG. 5 shows Embodiment 3 of the present invention.
FIG. 3 is a conceptual diagram for explaining direction normalization of a face image in the facial expression recognition device according to the first embodiment. In the figure, reference numeral 41 denotes a face image in a different direction as a face image photographed from a plurality of directions, and reference numeral 42 denotes a face image in a specific direction obtained from the face image 41 in a different direction by direction normalization.

Next, the operation will be described. When each state of the facial expression transition map 11 is generated by the facial expression transition map generating means 12, face images 41 having different directions photographed from a plurality of directions are input from the facial image input means 14, and interpolation processing is performed on them. Alternatively, direction normalization is performed by performing extrapolation processing to obtain a face image 42 in a specific direction. The expression transition map 11 is generated using the face image 42 in the specific direction.
This makes it possible to match the direction of the input face image when describing the expression transition map 11.

Embodiment 4 FIG. Next, a fourth embodiment of the present invention will be described. FIG. 6 shows Embodiment 4 of the present invention.
FIG. 4 is a conceptual diagram for explaining a comparison between a real face image and a facial expression transition map in a facial expression recognition device according to the present invention. In the figure, reference numeral 51 denotes a time-series actual photographed face image photographed in a time-series, and 52 denotes time-series face image data describing the expression transition map 11. In the facial expression recognition apparatus of the fourth embodiment, the facial expression transition map 11 is described in the time-series face image data 52, and the facial expression transition map 11 is compared and determined with the facial image from the facial image input means 14 at the time. It differs from the above embodiments in that it is performed with series data.

Next, the operation will be described. The matching unit 16 compares the time-series actual photographed face image 51 captured and input in time series by the face image input unit 14 and the expression transition map 11 described using the time-series face image data 52. The most similar time-series face image data 52 in the facial expression transition map 11 is
Is selected by the facial expression judging means 17 and the degree of the facial expression of the target person 13 is grasped. As described above, by generating the expression transition map 11 by incorporating the time-series face image data 52, a subtle difference such as the degree of the expression can be expressed by a temporal change, and the ambiguity of the degree of the expression can be reduced. It is possible to reduce and express the state of the degree of the accurate expression. In addition, since the real face image and the facial expression transition map are compared using the time-series data, and the facial expression having the most similar degree is determined as the facial expression degree of the real facial face image, the facial expression degree of the target person 13 can be more improved. It becomes possible to recognize accurately.

The method for generating the expression transition map 11 by the expression transition map generating means 12 can be realized by using, for example, the method described in the first embodiment along the time direction. That is, data of a time-series average face is generated by averaging face images corresponding to each moment from a time-series face image for each predetermined expression level. At this time, normalization in the time axis direction is performed as necessary. A set of average faces obtained in time series for each degree of facial expression is generated and described as time-series face image data 52 of one state of the facial expression transition map 11. By performing such an averaging process, it is possible to emphasize only the features of facial expressions having the same degree, cancel out other differences, and generate the facial expression transition map 11 including time-series changes. Become.

As for the method of determining the facial expression when generating the average face, as in the case described in the first embodiment, each face image is subjectively evaluated by the person, and as a result, the facial image is evaluated by a third party. The determination may be made using a result or a physiological index value indicating a change in facial expression.

Further, when comparing the time-series real-shot face image 51 with the expression transition map 11, the direction of the time-series real-shot face image 51 is changed to each state of the expression transition map 11 by the method described in the first embodiment. And the direction of the time-series face image data 52 at the same time. Thereby, it is possible to simplify the comparison between the time-series actual photographed face image 51 and the expression transition map 11.

For the matching, for example, a learning / identification function of a hidden Markov model, which is one of the techniques for recognizing a time-series pattern in speech, can be used. Each state of the expression transition map 11 (predetermined degree of expression) is prepared in advance by a plurality of persons or their own time-series face images serving as teacher data representing the respective states, and time-series features are determined based on the teacher data. learn. The comparison between the time-series real-shot face image 51 and the facial expression transition map 11 finds the likelihood between the time-series feature describing each of the learned states and the facial expression indicated by the time-series real-shot face image 52 indicating the state in which the maximum value is obtained. And the degree of By the determination processing using such a likelihood, it is possible to appropriately acquire the state of the time-series actual photographed face image 51.

Embodiment 5 FIG. Next, a fifth embodiment of the present invention will be described. The fifth embodiment relates to a medication control system configured using the facial expression recognition device shown in the first to fourth embodiments, and the schematic configuration is shown in FIG. In the figure, reference numeral 61 denotes a patient receiving medication, and 62 denotes a face image input means such as a camera for photographing the expression of the patient 61. Reference numeral 63 denotes a computer-based expression recognition apparatus main body that evaluates the degree of the facial expression of the patient 61 input from the face image input unit 62 and controls the content and dosage of medication based on the evaluation result.

Next, the operation will be described. The facial expression of the patient 61 who is taking the medicine is photographed by the face image input means 62, and the actual photographed facial image is sequentially input to the facial expression recognition device main body 63.
The facial expression recognition device main body 63 sequentially evaluates the degree of the facial expression such as pain and sleepiness with respect to the facial expression of the patient 61 input from the facial image input means 62, and determines the content and dosage of the medication according to the evaluation result. Control. This makes it possible to understand the effect of the medication on the patient,
Appropriate medication can be performed by giving feedback on the content and dosage of medication.

Embodiment 6 FIG. Next, a sixth embodiment of the present invention will be described. The sixth embodiment relates to an arousal level evaluation system configured using the facial expression recognition device described in the first to fourth embodiments, and a schematic configuration thereof is shown in FIG. In the figure, reference numeral 71 denotes a worker who is performing a predetermined operation, and is intended for a driver of a vehicle. Reference numeral 72 denotes face image input means such as a camera for photographing the expression of the worker 71. Reference numeral 73 denotes a computer-based facial expression recognition device that evaluates the degree of the facial expression of the worker 71 input from the face image input means 72 and evaluates the awakening level of the worker 71 based on the evaluation result. is there.

Next, the operation will be described. The facial expression of the worker 71 driving the vehicle is photographed by the face image input means 72, and the photographed facial image is sequentially input to the facial expression recognition device main body 73.
The facial expression recognition device main body 73 sequentially determines the degree of sleepiness of the facial expression of the worker 71 input from the face image input means 72, and evaluates the awakening level of the worker 71 based on the determination result. . Sequential facial expression recognition device body 7
3 increases the awakening degree of the worker 71 by giving feedback such as playing music to stimulate the worker 71 in accordance with the evaluation result. This allows
Accidents can be prevented beforehand. In the above description, a case has been described in which the driver of the vehicle is targeted as the worker 71. However, the present invention is not limited to this case. It goes without saying that it is applicable.

Embodiment 7 FIG. Next, a seventh embodiment of the present invention will be described. The seventh embodiment relates to a recovery evaluation system constituted by using the facial expression recognition device shown in the first to fourth embodiments, and its schematic configuration is shown in FIG. In the figure, reference numeral 81 denotes a patient whose degree of recovery is evaluated, and reference numeral 82 denotes face image input means such as a camera for photographing the facial expression of the patient 81. Reference numeral 83 denotes a facial expression recognition device using a computer or the like, which evaluates the degree of liveliness of the facial expression of the patient 81 input from the face image input means 82 and evaluates the recovery state of the patient and the effect of rehabilitation based on the evaluation result. The main body.

Next, the operation will be described. The facial expression of the patient 81 for which the effect of the course of treatment such as health recovery or rehabilitation is evaluated is photographed by the facial image input means 82, and the real facial images are sequentially input to the facial expression recognition device main unit 83. The facial expression recognition device main body 83 sequentially evaluates the degree of vitality of the facial expression of the patient 81 input from the face image input means 82, and grasps the progress of the treatment of the patient based on the evaluation result. This is useful as a means for assisting a verbal progress report at the time of medical examination and as a means for grasping the treatment status when treating a large number of patients.

The expression transition map used for the evaluation of the liveliness is generated, for example, as follows. Here, FIG. 10 is a conceptual diagram showing an example of the creation method.
In the figure, reference numerals 91, 92, 93, and 94 denote face images of the patient at each stage of the treatment. For each of a plurality of patients, the patient's face image 91 at the start of treatment (A), the patient's face image 92 at an important time when diagnosing the therapeutic effect (B), and the important time at the time of diagnosing the therapeutic effect (C) , And the facial image 94 of the patient at the time of healing (D) are sequentially photographed in chronological order by the facial image input means 82, and are averaged for each period to generate an expression transition map. . The degree of vitality of the facial expression of the patient 81 input from the face image input means 82 is determined by comparison with the facial expression transition map.

[0053]

As described above, according to the present invention, a facial expression transition map composed of a plurality of face images whose facial expressions change stepwise with a specific emotion or a human state which changes according to the state of a human being. Facial expression transition map generating means for generating, a facial image inputting means for inputting a facial image for which the degree of facial expression determination is to be performed, and a face image of each stage of the input photographed facial image and a facial expression transition map generated in advance. And a facial expression judging means for selecting the most similar face image from the collation result and judging the degree of the predetermined facial expression, so that the degree of the predetermined facial expression of the target person is determined. There is an effect that a facial expression recognition device capable of measuring and recognizing is obtained.

According to the present invention, as the face image at each stage of the expression transition map, the time-series face image data of the expression is used, and the comparison with the time-series real face image is performed. There is an effect that the degree of facial expression can be accurately measured and recognized in consideration of the time-series change.

According to the present invention, since the facial expression transition map is generated by the average face obtained by averaging the facial images of a plurality of persons for each facial expression at each stage, the facial expression having a different degree of facial expression from the predetermined facial expression level is obtained. This has the effect of canceling out the effects and emphasizing only those features that exist in many human beings.

According to the present invention, since the facial expression transition map is generated by the average face obtained by averaging the facial image of the specific individual for each facial expression, the degree of the specific facial expression of the specific individual This has the effect of canceling out the effects of different facial expressions and emphasizing only the features of that degree.

According to the present invention, a face image corresponding to a greater number of stages is obtained from a face image having a plurality of different facial expressions generated by using a photographed face image by image interpolation processing or extrapolation processing. By generating the facial expression transition map, it is possible to eliminate the need to directly obtain the state of the intermediate degree in the facial expression transition map from the face images of a plurality of persons presenting the same degree. effective.

According to the present invention, as a method of determining the degree of facial expression of a face image used for generating a facial expression transition map, a subjective rating value of the degree of emotion of the subject himself / herself, and a third party observes and evaluates the facial image. Or a plurality of physiological index values that change according to the target emotion or human condition, so that the degree of facial expression can be determined in a method that matches the There is an effect that the index value can be used in an objective manner.

According to the present invention, as a method of determining the degree of facial expression of a face image used to generate a facial expression transition map, a subjective rating value of the degree of emotion of the subject himself / herself, and a third party observes and evaluates the facial image. Using multiple rating values, such as the estimated values, physiological index values that change according to the target emotion and the human state, and select a face image with a high degree of matching between the multiple rating values, and use that to perform facial expression transition Since the map is configured to be generated, not only can the degree of facial expression be classified in a manner that matches human senses or in an objective way by using physiological index values, but also the characteristics of a given facial expression can be more accurately determined. There are effects such as being able to catch.

According to the present invention, a facial expression transition map is generated by using a facial image in a specific direction from a facial image photographed from a plurality of directions. There are effects such as matching the directions of the images, facilitating the generation of the expression transition map, and accurately extracting the features.

According to the present invention, the face image input means is provided with a function of photographing a face image from a plurality of directions.
Since an image in which a face image in the same direction as the facial expression transition map is generated from the input facial images in a plurality of directions is used as a determination target image, and this is compared with each facial image in the facial expression transition map, the input facial image There is an effect that the collation can be performed accurately without depending on the direction of.

According to the present invention, the degree of facial expression of the patient is determined using the facial expression recognition device, and the effect of the medication on the patient is thereby evaluated. And it is possible to obtain a medication control system that can appropriately administer medication by giving feedback to the type and dosage of the medication.

According to the present invention, the degree of drowsiness of a worker such as a vehicle driver is determined by using the above-mentioned facial expression recognition device, and thereby the awakening level of the worker is evaluated. It is possible to sequentially grasp the degree of awakening of the worker, and obtain an arousal level evaluation system that can increase the degree of awakening of the worker by giving feedback that gives a stimulus to the worker based on the degree. There is.

According to the present invention, the facial expression recognition device is used to determine the degree of facial expression of the patient, thereby evaluating the health recovery state of the patient and the effect of rehabilitation. This has the effect of providing a recovery evaluation system that is useful as a means for assisting a verbal progress report or a means for grasping the treatment status when treating a large number of patients.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a facial expression recognition device according to Embodiments 1 to 4 of the present invention.

FIG. 2 is a conceptual diagram illustrating recognition of a degree of a predetermined facial expression in the first embodiment.

FIG. 3 is a conceptual diagram showing an example of generating an expression transition map according to the first embodiment.

FIG. 4 is a conceptual diagram showing an example of generating an expression transition map according to the second embodiment.

FIG. 5 is a conceptual diagram for describing direction normalization of a face image according to the third embodiment.

FIG. 6 is a conceptual diagram for explaining matching between a time-series real-shot face image and an expression transition map according to the fourth embodiment.

FIG. 7 is a schematic configuration diagram showing a medication control system according to Embodiment 5 of the present invention.

FIG. 8 is a schematic configuration diagram showing a driver awakening state recognition system according to Embodiment 6 of the present invention.

FIG. 9 is a schematic configuration diagram showing a recovery evaluation system according to Embodiment 7 of the present invention.

FIG. 10 is a conceptual diagram showing a method of generating an expression transition map according to the seventh embodiment.

FIG. 11 is a block diagram showing a configuration of a conventional expression recognition device.

[Explanation of symbols]

11 facial expression transition map, 12 facial expression transition map generating means, 13 target person, 14 face image input means, 15 real face image, 16 collating means, 17 facial expression determining means, 23
Average face, 31 maximum degree face images (face images with different degrees of predetermined expression), 32 minimum degree face images (face images with different degrees of predetermined expression), 34 intermediate degree face images (more Face images in different directions (face images taken from a plurality of directions), 42 face images in a specific direction, 51 time-series real face images, and 52 time-series face image data.

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Shotaro Miwa 2-3-2 Marunouchi, Chiyoda-ku, Tokyo Sanishi Electric Co., Ltd. (72) Inventor Hiromi Terashita 2-3-2 Marunouchi, Chiyoda-ku, Tokyo 3 Rishi Electric Co., Ltd. (72) Inventor Kazuo Hajima 2-3-2 Marunouchi, Chiyoda-ku, Tokyo F-term (reference) 5B057 AA07 AA19 CD11 DA12 DB02 DC33

Claims (12)

[Claims] 1. A facial expression transition map generating means for generating a facial expression transition map composed of a plurality of face images whose facial expressions change stepwise, the facial expression changing with a predetermined emotion or a human state, A face image input unit for inputting a face image of a target person whose degree of facial expression is to be determined, a real face image input from the face image input unit, and the facial expression previously generated by the facial expression transition map generating unit A matching unit that matches the face image at each stage of the transition map; and a face image that is the most similar is selected from the result of the matching by the matching unit, and the stage is output to determine the degree of the predetermined facial expression. A facial expression recognition device comprising: a facial expression determination unit. 2. The method according to claim 1, wherein the facial image at each stage of the facial expression transition map generated by the facial expression transition map generation unit comprises facial expression time-series facial image data, and the matching unit is input from the facial image input unit. 2. The method according to claim 1, wherein the time-series real-shot face image is compared with the time-series face image data of each stage of the expression transition map generated in advance by the expression transition map generating means. Facial expression recognition device. 3. A facial expression transition map generating means for generating a facial expression transition map using an average face obtained by averaging facial images of a plurality of persons for each facial expression at each stage. An expression recognition device according to claim 1 or 2. 4. A facial expression transition map generating means for generating a facial expression transition map using an average face obtained by averaging a specific individual's facial image for each facial expression at each stage. An expression recognition device according to claim 1 or 2. 5. A facial expression transition map generating means comprising: a plurality of face images having different degrees of a predetermined facial expression generated by using a photographed facial image by interpolation or extrapolation of the image in more stages; 3. The facial expression recognition device according to claim 1, wherein a facial expression transition map is generated by generating a corresponding facial image. 6. The facial expression transition map generating means determines the degree of facial expression of a facial image used for generating the facial expression transition map, and evaluates the subjective evaluation value of the degree of emotion of the subject, and observes the facial image by a third party. Either one of the following evaluation values, or the physiological index value that changes depending on the state of the target emotion or person
The facial expression recognition device according to claim 1 or 2, wherein one or a plurality of the facial expression recognition devices are used. 7. The facial expression transition map generating means determines a degree of facial expression of a facial image used for generating a facial expression transition map by a subjective rating value of a degree of emotion of a subject, and a third person observes the facial image. Using multiple rating values, such as a rating value performed by the user and a physiological index value that changes according to the state of the target emotion or person, select a face image with a high degree of matching between the rating values, and use that The facial expression recognition device according to claim 6, wherein the facial expression transition map is generated. 8. A facial expression transition map generating means for generating a facial image in a specific direction from facial images photographed from a plurality of directions, and using the facial image to generate a facial expression transition map. An expression recognition device according to claim 1 or 2. 9. The face image input means has a function of photographing a face image from a plurality of directions, and the collating means converts a face image in a plurality of directions input from the face image input means in the same direction as the facial expression transition map. 3. The facial expression recognition device according to claim 1, wherein an image in which the facial image is generated is used as a determination target image, and the image is compared with each facial image in the facial expression transition map. 10. The degree of facial expression of a patient is determined using the facial expression recognition device according to any one of claims 1 to 9, and the effect of medication to the patient is determined based on the determination result. A dosing control system characterized by evaluating and determining the kind and dosage of a drug. 11. The degree of sleepiness of a worker is determined using the facial expression recognition device according to any one of claims 1 to 9, and the awakening level of the worker is determined based on the determination result. A wakefulness level evaluation system characterized by evaluating 12. The degree of facial expression of a patient is determined using the facial expression recognition device according to any one of claims 1 to 9, and based on the determination result, the patient's state of health recovery or rehabilitation. A recovery evaluation system characterized by evaluating the effect of a recovery.