JP2023038870A - Impression evaluation method and impression evaluation system - Google Patents

Abstract

To provide an impression evaluation method and an impression evaluation system configured to evaluate multifaceted and complicated impressions of the face of a subject, objectively and more accurately.SOLUTION: An impression evaluation method includes a face information acquisition process, an expression motion evaluation process, an emotion index calculation process, and an information presentation process. The face information acquisition process acquires multiple pieces of different face information data. The expression motion evaluation process evaluates strength of motion per expression motion unit based on the motion of muscles of facial expression, for each of the face information data. The emotion index calculation process calculates an emotion index per expression motion unit, on the basis of a result evaluated by the expression motion evaluation process and an emotion associated with the expression motion unit. The information presentation process presents, using the emotion indices of multiple pieces of face information data, association information between the face information data and the emotion indices for evaluating impression of the face of a subject.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The present invention relates to an impression evaluation method and an impression evaluation system, and more particularly to an impression evaluation method and an impression evaluation system for evaluating an impression received from an evaluation subject's face based on the evaluation subject's face information.

In general, the impression of a person gives various impressions to others depending on factors such as appearance and speech and behavior. Various studies have been conducted on techniques for evaluating the impression that people feel regarding the appearance of this face. For example, Patent Literature 1 discloses a method of discriminating (evaluating) the visual impression of an evaluation subject's face based on the physical quantity in a portion that greatly affects the visual impression of the face. Such a method is as follows: (1) First, from a moving image of the subject's face captured by a camera, a large portion (for example, the cheek or the periphery of the eye) that gives a large visual impression to a person is determined, and (2) Secondly, the visual impression of the subject's face is evaluated based on the physical quantity of the skin change caused by the facial expression change at the determined site.

In addition, as research on human facial expressions, for example, Non-Patent Document 1 reports that when a person makes a fake smile, different facial expressions appear on the right and left faces, and left-right asymmetrical movements are observed. .

Development of software for evaluating human facial expressions is also underway. For example, in Non-Patent Document 2, the results of coding by a FACS-certified coder (those who have learned the "FACS" coding method based on the FACS manual and Investigator guide and passed the FACS coder certification test) have been turned into teacher data. Programs (eg, "Affdex" (Affectiva)) have been developed.

JP 2020-061194 A

Paul Ekman, Joseph C. Hager, Wallace V. Friesen, "The Symmetry of Emotional and Deliberate Facial Actions", Psychophysiology, Vol.18, No.2, March 1981 Affectiva Co., Ltd., [online], [searched July 6, 2021], Internet <https://www.affectiva.jp/>

It is common for a person's face to constantly change its expression according to changes in various emotions (inside). In addition, as in Non-Patent Document 1, different facial expressions are expressed on the left and right sides of the face, which can also be a factor that affects the impression of the face. However, the technology described in Patent Literature 1 is an impression evaluation method that focuses only on the parts that affect the age impression of the face, etc., and whether the appearance impression is younger than the actual age of the person to be evaluated. , or is old, and only relatively simple evaluation results can be obtained. Therefore, the technique described in Patent Literature 1 cannot accurately evaluate the impression received from the evaluation subject's face. In addition, the technique described in Non-Patent Document 2 is limited to the objective description of facial expressions and the estimation of basic emotions. There is room for improvement in terms of highly accurate evaluation of impressions.

SUMMARY OF THE INVENTION The present invention relates to providing an impression evaluation method and an impression evaluation system that can overcome the drawbacks of the prior art described above.

The present invention relates to an impression evaluation method for evaluating an impression received from the face of an evaluation subject based on face information data of the evaluation subject.
The impression evaluation method includes a facial information acquisition step of acquiring a plurality of different facial images, and a facial movement evaluation step of evaluating the intensity of movement for each facial movement unit based on the movement of facial muscles for each of the facial images. an emotion quotient calculation step of calculating an emotion quotient for each facial expression movement unit based on the evaluation result of the facial expression movement evaluation step and the emotion associated with the facial movement unit; and an information presenting step of presenting corresponding information between the face information data and the emotion quotient for evaluating the impression received from the face of the person to be evaluated, using the information presenting method.

The present invention also relates to an impression evaluation system for evaluating the impression received from the evaluation subject's face based on the evaluation subject's face information data.
The impression evaluation system includes a facial information acquisition unit that acquires a plurality of different facial images, and a facial movement evaluation unit that evaluates the strength of movement for each facial movement unit based on the movement of facial muscles for each of the facial images. and calculating the emotion quotient for each facial movement unit based on the evaluation result of the facial movement evaluation unit and the emotion associated with the facial movement unit, and outputting the emotion quotient of the plurality of facial information data. and an emotion quotient calculator.

According to the impression evaluation method and the impression evaluation system of the present invention, it is possible to objectively and accurately evaluate the multifaceted and complex impression received from the face of the person to be evaluated.

FIG. 1 is a schematic explanatory diagram showing one embodiment of the impression evaluation system of the present invention. FIG. 2 is a flow chart showing the contents of control processing executed in the impression evaluation system shown in FIG. FIG. 3 is a table showing facial images acquired by the system of FIG. 1 together with emotional quotient patterns based on the facial images. FIG. 4 is an example of correspondence information according to the present invention, and is a graph in which emotion indices are plotted on the vertical and horizontal axes. FIG. 5 shows facial images acquired over time and emotional quotient patterns based on the facial images. FIG. 6 is an example of correspondence information in the makeup impression evaluation method according to the present invention, and is a graph in which emotion indices are plotted on the vertical axis and the horizontal axis. FIG. 7 shows an emotion quotient pattern based on the face image along with the face image of a woman with a bare face and expressionless face according to the first embodiment. FIG. 8 shows a face image of a man with a bare face and expressionless face according to the first embodiment, and an emotion quotient pattern based on the face image. FIGS. 9(a) to 9(c) are graphs for evaluating the impression received from the evaluation subject's face created using each emotion quotient pattern of FIGS. 7 and 8. FIG. FIG. 10 shows a face image of a bare face and a face image with makeup 1 applied in Example 2, as well as emotional quotient patterns based on the face images. FIG. 11 shows a face image of a woman with makeup 2 of Example 2 and an emotion quotient pattern based on the face image. FIG. 12 shows a face image of a woman with makeup 3 of Example 2 and an emotion quotient pattern based on the face image. FIGS. 13(a) to 13(c) are graphs for evaluating the impression received from the evaluation subject's face, which are created using the emotion quotient patterns of FIGS. 10, 11 and 12. FIG. FIG. 14 shows images around the eyebrows before and after applying eyebrows in Example 3, as well as emotional quotient patterns based on the image data. FIG. 15 shows an image of a part of the face before and after applying makeup in Example 4, together with an emotion quotient pattern based on the image data. FIG. 16 shows emotion quotient patterns based on face images when a woman with a bare face expresses facial expressions in Example 5. FIG. FIG. 17 shows a facial image of a woman to whom FD1 was applied in Example 5 when expressing facial expressions, and an emotion quotient pattern based on the facial image. FIG. 18 shows a facial image of a woman to whom FD2 was applied in Example 5 when expressing facial expressions, and an emotion quotient pattern based on the facial image. FIGS. 19A and 19B are graphs for evaluating the impression received from a woman's face created using each emotion index pattern of FIGS. 16-18. FIG. 20 is a table showing emotion quotient patterns based on face images of persons to be evaluated at interview in Example 6. FIG.

BEST MODE FOR CARRYING OUT THE INVENTION An impression evaluation system of the present invention will be described below based on a preferred embodiment thereof with reference to the drawings. FIG. 1 shows how an impression evaluation system 1 (hereinafter simply referred to as "system 1") according to the present embodiment is used to evaluate the impression received from the face of an evaluation subject H. As shown in FIG. More specifically, the present system 1 and the impression evaluation method described later provide evaluation materials for evaluating the impression received from the face of the person H to be evaluated. Such evaluation materials can also be used to judge and explain the impression received from the evaluation subject H's face.

(Configuration of impression evaluation system 1)
The system 1 of this embodiment acquires a face image as face information data. The face information data is information that indicates the facial expression of the person to be evaluated. In addition to the face image, it includes a 3D model image using simulation software, and contour shapes of facial parts such as eyes, nose, mouth and ears. Information such as three-dimensional coordinate values or vector values is included.
As shown in FIG. 1, the impression evaluation system 1 of the present embodiment includes a plurality of (nine in this embodiment) imaging devices C1 to C9 for imaging the face of the person H to be evaluated, and the imaging devices C1 to C9. and a control device 2 that evaluates the impression of the person H to be evaluated using the photographed face image.

The plurality of imaging devices C1 to C9 are, for example, composed of cameras (for example, monocular cameras) capable of capturing still images, and surround the evaluation subject H in front at predetermined angular intervals (in this embodiment, , about 22.5 degrees apart). In addition, the imaging devices C1 to C9 according to the present embodiment are electrically connected (wired or wirelessly) to each other, and configured to simultaneously image the face of the subject H in synchronization with the imaging operation. be done. As a result, nine types of face images of the person H to be evaluated, including the front (imaging device C5), the left sideways (imaging device C1), and the right sideways (imaging device C9), with angular intervals shifted by about 22.5 degrees. can be obtained simultaneously.

In this embodiment, the number of imaging devices is nine in order to acquire a plurality of face images, but the number of imaging devices is not limited to this, and may be two or more and less than nine, or ten. It is good as above. In addition, it is preferable to arrange at least one imaging device at a position where the evaluation subject can be imaged almost from the front. Further, the imaging devices C1 to C9 are not limited to devices that capture two-dimensional images, and may be so-called 3D cameras that capture three-dimensional images. Furthermore, the imaging device is not limited to a device that captures a still image, and a device that captures a moving image (hereinafter referred to as a “moving image capturing device”) may be used. In this case, since a plurality of frame images (still images) and moving image clips can be extracted from moving image data, the number of moving image capturing apparatuses can be reduced to one (see "Embodiment 6" described later). ). Of course, a plurality of moving image pickup devices may be used like the image pickup devices C1 to C9. In this case, for example, the moving image pickup devices may be synchronized with each other, and frame images or the like at the same timing may be extracted from each moving image.
The face image may be a color image represented by color information such as RGB or HSV, and may be a grayscale image, a black and white binarized image, or information in a specific wavelength band as color information. It may be a reflected spectral image, or a multispectral image in which each spectral image is combined as color information.

Next, the control device 2 according to this embodiment will be described. The control device 2 of this embodiment can be configured using a known general-purpose computer. The general-purpose computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and the like. The CPU includes a graphics processor (Graphics Processing Unit (GPU)) for displaying images, a multimedia processor for encoding and decoding High-Definition (HD) video, a display controller for controlling the display, and power supply and charging. may include a power management Integrated Circuit (IC) or the like for controlling the The processing performed by the control device 2 is implemented by the CPU developing a program stored in a ROM, disk, or the like into a RAM and executing the program. Such processing may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or by a combination of an ASIC and an FPGA.

The control device 2 of this embodiment has a control section 3, a storage section 4, a face information acquisition section 5, and a display section 6, which are electrically connected to each other.

(Face information acquisition unit 5)
The face information acquisition unit 5 acquires a plurality of different face information data. The face information acquisition unit 5 of this embodiment acquires face information data from a plurality of different angles using the imaging devices C1 to C9. Specifically, the face information acquisition unit 5 is an interface capable of communicating with each of the imaging devices C1 to C9, and receives input of each face image data transmitted from the imaging devices C1 to C9. In this case, the system 1 uses face images viewed from a plurality of different angles to create correspondence information between face information data and emotion quotients for impression evaluation. The emotion quotient and corresponding information will be described later.
Further, the face information acquisition unit 5 may acquire face information data by means other than the imaging devices C1 to C9. That is, the face information acquisition unit 5 may acquire face information data such as a face image that is transmitted or input to the control device 2 . For example, the face information acquiring unit 5 machine-learns the face of the person H to be evaluated, and extracts a frame image of the face of the person H to be evaluated from the moving image data read by the control device 2. data may be obtained. In this case, the system 1 calculates data such as an emotion index and an emotion index for performing impression evaluation using face images acquired over time.

The face information acquisition section 5 stores the acquired face information data in the storage section 4 of the control section 3 . In addition, according to the operation (command) of the operator of the control device 2, the face information acquisition unit 5 reads the face information data stored in the storage unit 4 and the face information data acquired by the face information acquisition unit 5. , the face information data is displayed on the display unit 6 provided in the control device 2 or the display unit of an information terminal (for example, a smart phone) connected to the control device 2 via a network.
The operator of the control device 2 includes not only an administrator who manages the system 1 but also a user of the system 1, which will be described later.

(storage unit 4)
The storage unit 4 stores a basic operation program that governs the basic operations of the impression evaluation system 1, as well as a FACS estimation automatic program for calculating emotion quotients, calculation formulas for calculating emotion quotients, and the like. It is The storage unit 4 also stores face information data (for example, face image data). The storage unit 4 may store the face information data of the person H to be evaluated as a database. Such a database is created by associating the face information data of the person H to be evaluated, the identification information of the person H to be evaluated, and the analysis conditions for impression evaluation.

(Display unit 6)
The display unit 6 includes, for example, a liquid crystal display (LCD), and displays a predetermined image based on a command from the control unit 3. An emotion index pattern (for example, Table 1) in which the emotion index calculated by the index calculator 3C is described is displayed. The emotion quotient pattern will be described later.

(control unit 3)
The control unit 3 of the present embodiment reads various programs stored in the storage unit 4, performs predetermined arithmetic processing, and performs general control for displaying a predetermined image on the display unit 6. Control is performed to evaluate the impression received from the evaluation subject H's face based on the nine types of face images transmitted from each of C9.

In the present embodiment, as a facial expression analysis method for evaluating such an impression, the "Facial Action Coding System" (Facial Action Coding System, hereinafter referred to as "FACS") developed by Paul Ekman, Wallace Friesen, et al. ) is adopted.

Here, in order to facilitate understanding of this embodiment, the above-described "FACS" will be described. "FACS" is a method developed to comprehensively measure facial movements based on the movements of facial muscles (e.g., corrugator and buccal muscles) from an anatomical point of view. Of the basic motions, 32 motions with clear anatomical foundations are defined as "Action Units" (hereinafter referred to as "AU"). The "AU" defined in this way can be coded by evaluating the strength of the action in five stages from "A" to "E", thereby enabling the expression of facial expressions. can be evaluated objectively. Such "AU" corresponds to the facial action unit according to the present invention.
In addition, in the FACS evaluation, if no movement is observed in the facial expression unit, none of "A" to "E" is applicable.

Also, for "AU", emotional valence can be evaluated by its defined action. That is, it is possible to evaluate whether the motion of the AU is a "positive" facial expression or a "negative" facial expression. For example, based on previous research (Kohler et al., (2004) Psychiatry Research) that investigated the relationship between the emotional impression received by looking at the face (expression) and each facial expression, "AU6" (movement to lift the cheek) and " AU12" (movement of pulling up both ends of the lips) can be regarded as a facial expression classified as "Positive". Also, "AU2" (movement to raise the outer side of the eyebrow) and "AU4" (movement to lower the eyebrow) can be regarded as facial expressions classified as "Negative". "AU" that does not correspond to either "Positive" or "Negative", for example, "AU1" (movement to raise the inner side of the eyebrow) and "AU5" (movement to raise the upper eyelid) are classified as "Neutral". It can be regarded as a facial expression that is done. The "emotional valence" used here is a concept that defines a qualitative difference in emotions including two polarities of "Positive" and "Negative" and "Neutral" that is intermediate between them.

"AU" can be classified into "Positive", "Negative" and "Neutral" as described above, as well as so-called basic emotions ("Happy", "Sad", "Angry" and "Fear"). It is possible to classify For example, "AU6" (movement to lift the cheek) and "AU12" (movement to lift both lips) are classified as "Happy" as facial expressions, and "AU17" (raise the chin) is classified as "Sad". “AU9” (wrinkle the nose) is an expression classified as “Angry,” and “AU2” (raise the outside of the eyebrows) is an expression classified as “Fear.” As an expression, it is possible to evaluate each.

In this embodiment, a program (FACS estimation automatic program) consisting of an algorithm for coding the above-described "AU" motion intensity is stored in the storage unit 4, and the control unit 3 is configured to evaluate the facial expression of the person to be evaluated H (movement intensity for each AU).

The control unit 3 includes a facial movement evaluation unit 3A, an emotion quotient calculation unit 3B, an emotion index calculation unit 3C, and an information presentation unit 3D. It is configured to derive correspondence information for performing impression evaluation received from.

The facial expression movement evaluation unit 3A evaluates the intensity of movement for each facial expression movement unit based on the movement of facial muscles for each face information data. The facial expression action evaluation unit 3A of the present embodiment evaluates the intensity of movement ( For example, it has a function of evaluating five grades from "A" to "E").

The emotion quotient calculation unit 3B calculates an emotion quotient for each facial expression movement unit based on the evaluation result of the facial expression movement evaluation unit 3A and the emotion associated with the facial movement unit. The emotion quotient calculation unit 3B of the present embodiment has a function of calculating an emotion quotient for each "AU" based on the evaluation result by the facial movement evaluation unit 3A and the emotion associated with the "AU". ing. The emotional quotient calculation unit 3B of the present embodiment expresses emotions associated with facial movement units (AU) by emotional valences such as "Positive", "Negative" and "Neutral", and calculates the emotional quotients of the emotional vales. do.
Here, the above motion strength evaluations “A” to “E” can be quantified as 1 to 5. For example, AU12 is associated with "Positive", and if its movement intensity is B, this is used in the calculation in the form of "2" of "Positive". In the calculation (numericalization) of the emotion quotient, if no movement is observed in the facial expression unit, it is assumed that none of "A" to "E" is applicable, and thus "0" is given.

The processing performed by the facial expression action evaluation section 3A and the emotion quotient calculation section 3B in this embodiment is based on the FACS estimation automatic program. The FACS estimation automatic program extracts and analyzes the facial expression action for each "AU" from the face image, evaluates and encodes the action intensity of the "AU", and evaluates the emotion associated with the "AU" A program for deriving valence and its emotional quotient. Specifically, the facial expression action evaluation unit 3A evaluates the action intensity of each AU in the face image data as A to E. The emotional quotient calculator 3B derives the emotional valence and its emotional quotient for each AU from the evaluation result of the action intensity of each AU. The FACS estimation automatic program can be realized, for example, by using a program (for example, "Affdex" (Affectiva)) in which the results of coding by the above-described FACS-certified coder are turned into teacher data.

The emotion index calculator 3C calculates the emotion index of the face information data based on a plurality of emotion indexes. The emotion index calculator 3C of the present embodiment has a function of calculating an emotion index for each face image or for a total of a plurality of face images based on the emotion index of the emotional valence for each "AU". Specifically, the emotion index calculator 3C performs control to calculate the emotion index using a calculation formula to be described later.

Arithmetic processing for calculating an emotion index by the emotion index calculator 3C is performed using a calculation formula based on a plurality of emotion indexes. These calculation formulas are stored in the storage unit 4, and the emotion index calculation unit 3C uses the calculation formulas to obtain an emotion index such as a face image. The following calculation formulas (1) to (13) for obtaining emotion indices are examples.
(1) Mean Positive: ΣX _{AU positive} /n _{AU positive}
(2) Mean Negative: ΣX _{AU negative} /n _{AU negative}
(3) Mean Neutral: ΣX _{AU neutral} /n _{AU neutral}
(4) Max Positive: Max ( _{XAU positive} )
(5) Max Negative: Max ( _{XAU negative} )
(6) Max Neutral: Max (X _{AU neutral} )
(7) Emotion: Max (X _{AU positive} )-Max (X _{AU negative} )
(8) Emotion Intensity: || (Max (X _{AU positive} ), Max (X _{AU negative} )) ||
(9) Total positivity: Max (X _{AU positive} ) / {Max (X _{AU positive} ) + Max (X _{AU negative} ) + Max (X _{AU neutral} )}
(10) Complexity: Mean Positive x Mean Negative
(11) Max Complexity: Max Positive x Max Negative
(12) Mean Intensity: Σ _{(image1→imageN) XAUx image} /N
(13) Image DependencyAUx: STDEV(X _{AUx, image1} , X _{AUx, image2} , ... X _{AUx, imageN} )

In the above formula, X _AUx is the coding result of AUx (x indicates positive, negative, neutral, or action unit number), n is the number of data, STDEV is the function for obtaining the standard deviation, and N is the image each number.

Each of the calculation formulas (1) to (13) for calculating the emotion index described above may be used alone or in combination. For example, other calculation formulas may be added to the above formulas (1) to (13) depending on the purpose of analysis. Other calculation formulas include the following.
・Total positive (ΣX _{AU positive} )
・Total Negative (ΣX _{AU negative} )
· Total Negativity [ Max (X _{AU negative} ) / {Max (X _{AU positive} ) + Max (X _{AU negative} ) + Max (X _{AU neutral} )}]

In the following description, each calculation formula will be referred to as follows for convenience.
Positivity: (4) Max Positive
Negative degree: (5) Max Negative
Neutral degree: (6) Max Neutral
Emotional impression: (7) Emotion
Emotion Intensity: (8) Emotion Intensity
Complexity or Complexity: (11) Max Complexity
Mean Intensity: (12) Mean Intensity
Multifacetedness or multifacetedness: (13) Image DependencyAUx for still images of faces
Frame dependency: (13) Image DependencyAUx for facial moving images (time-series images)

The positivity, negativity, and neutrality are indices indicating the maximum positive value, the maximum negative value, and the maximum neutral value among AUs in one or more facial images. ), it is easy to grasp each emotional valence that contributes to
Emotional impression is an index that indicates the difference between positive and negative degrees in one or more facial images, and it is possible to grasp the emotional valence of the facial expression as a whole.
Emotional intensity is an index that indicates the overall intensity of emotional valence regardless of whether one or more facial images are positive or negative, and it is possible to grasp the emotional intensity of the entire facial expression.

"Complexity" means the degree of complexity of the facial expressions of the person H to be evaluated. "Facial complexity" means that the face has facial expressions that express multiple emotions. For example, the evaluation subject H, who can see the action of "AU4" (movement to lower the eyebrows) derived from "Negative" and the action of "AU12" (movement to raise both ends of the lips) derived from "Positive", Increased “complexity”.

“Multifacetedness” means the degree of multifacetedness in which the impression of a face differs depending on the viewing angle. By observing the face from various angles, we can get a multifaceted impression of the other person, including their emotions. mean different things. For example, when the face at a certain angle and the face at another angle have different emotional indices such as "AU4" and "AU12", the "multifacetedness" is high.

In the emotion index calculation step, a plurality of types of emotion indexes relating to the single face information data may be substituted into the respective calculation formulas described above to calculate the emotion index relating to the single face information data. That is, the emotion index may be calculated based on a plurality of types of emotion index regarding single face information data. For example, in Table 1 described later, for a face image (single face image) captured by the imaging device C1, a plurality of emotion quotients such as AU6 and AU4 are calculated using any of the above-described calculation formulas (1) to (13). By substituting in , an emotional index such as a degree of positivity or a degree of negativity may be calculated.
In this case, an emotion index is calculated for each face image, and the plurality of emotion indexes are compared between the face images, or the plurality of emotion indexes are integrated, thereby obtaining an impression received from the face of the person to be evaluated. Can be used for evaluation. As for "combining a plurality of emotion indices", there is an aspect using "another calculation formula" to be described later.

In the emotion index calculation step, one type of emotion index relating to the plurality of face information data may be substituted into each of the above-described calculation formulas to calculate the emotion index relating to the plurality of face information data. That is, the emotion index may be calculated based on one type of emotion index related to multiple pieces of face information data. For example, in Table 1 to be described later, by substituting one type of emotion quotient AU6 for a plurality of face images captured by imaging devices C1 to C9 into any of the above-described calculation formulas (1) to (13), , an emotional index such as average intensity or multifacetedness may be calculated.
In this case, it is possible to use the emotion index reflecting the facial expressions in a plurality of face images to evaluate the impression received from the evaluation subject's face.

The information presentation unit 3D presents corresponding information for evaluating the impression received from the face of the person H to be evaluated, using the emotion quotients of the plurality of face information data calculated by the emotion quotient calculation unit 3B. Corresponding information is information in which face information data and emotion quotients are associated with each other, and is created by associating the emotion quotient obtained by the emotion quotient calculator 3B with the corresponding face information data. The correspondence information can be used to evaluate the impression received from the evaluation target person H's face.

Further, the information presenting unit 3D of the present embodiment presents correspondence information including emotion indices of a plurality of face information data. When the corresponding information includes an emotional index, the corresponding information is created by associating the emotional index calculated by the emotional index calculator 3B and the emotional index calculated by the emotional index calculator 3C with the corresponding face information data. It is what was done. Since such correspondence information includes an emotion index, the evaluation accuracy of the impression received from the face of the person H to be evaluated can be improved by using the correspondence information.

Table 1 shows an example of correspondence information presented by the information presentation unit 3D in this embodiment. The emotional quotient of emotional valence is represented by a score of 0 to 5, for example. A table in which emotion quotients and emotion indexes as shown in Table 1 are described is hereinafter also referred to as an "emotional quotient pattern". That is, the emotion quotient pattern is a type of correspondence information.
'C1', 'C2', . indicates
As shown in Table 1, the emotion index can be obtained from one face image or from a plurality of face images (for example, a plurality of face images obtained from imaging devices C1 to C9).

The information presentation unit 3D of this embodiment has a function of graphing a plurality of types of emotion indices. Such a graph is also a type of correspondence information presented by the information presentation unit 3D, and the correspondence information includes emotion indices. For example, a graph may be created in which the vertical axis is the degree of negativity and the horizontal axis is the degree of positivity, and each emotion index of face information data obtained under different conditions may be plotted on the graph. As shown in FIGS. 4 and 6, which will be described later, the face information data of different evaluation subjects and the face information data of the same subject with different makeup methods are plotted on a graph, and the facial attributes of the face information data are plotted. can grasp.
In the graph presented by the information presentation unit 3D, the emotion index of a single face information data may be plotted, or the emotion indexes of a plurality of face information data may be plotted as shown in FIGS. good.

The corresponding information such as the above-mentioned emotional quotient patterns and graphs is based on the emotional quotient or the emotional index of the face information data obtained from multiple angles, which emotional value ("Positive", " In other words, the attribute of the impression received from the face of the person to be evaluated is grasped from the emotion quotient pattern and scatter diagram (graph) based on the emotion quotient. In this case, the attribute of the impression received from the face may be classified for each different imaging angle, thereby making it possible to evaluate the impression by comparing the attributes of the face images captured from each angle. A plurality of angles may be combined to classify face attributes, in which case the emotion quotients of a plurality of AUs may be used, or the emotion quotients of a selected portion of the AUs may be used.

As described above, the information presentation unit 3D of the present embodiment presents the emotion quotients of a plurality of face information data and corresponding information including the emotion indexes calculated by the emotion quotient calculation unit 3B and the emotion index calculation unit 3C. The correspondence information may be an emotion quotient pattern that summarizes the emotion quotients and emotion indexes of a plurality of face information data, or may be a graph reflecting these.
The correspondence information is presented in a mode e1 in which the correspondence information is displayed on the display unit 6 provided in the system 1, a mode e2 in which the data of the correspondence information is transmitted to an information terminal separate from the system 1 via the network, example e3 in which the correspondence information is displayed on a general-purpose web browser accessed by the information terminal. Here, the “separate information terminal” is an information terminal owned by the user of the system 1, and includes a tablet terminal such as a smart phone, a PC, and the like. Such users are generally those who use the correspondence information presented by the system 1, and those who use the correspondence information to grasp the impression received from the face of the person to be evaluated, analyze or evaluate the impression, and the like. mentioned. More specifically, at a store where face-to-face sales of cosmetics are carried out, there are advisors who give makeup advice to customers, general users who receive support information provision services, and the like.
For example, the mode e2 includes a mode in which correspondence information is distributed through an application installed on a smartphone owned by a general user.
Further, as the mode e3, an advisor or a general user may use a tablet terminal to access a general-purpose web browser and display corresponding information on a page of the web browser.

In this embodiment, since the emotion quotient and emotion index are calculated based on a plurality of different face information data (face images), the corresponding information using the emotion quotient and the emotion index is used to receive from the face of the person to be evaluated H. Multifaceted and complex impressions can be evaluated with higher accuracy.
More specifically, in the system 1 of the present embodiment, the emotion quotient calculation unit 3B calculates the emotion quotient using face information data acquired from a plurality of different angles. can be grasped. For example, based on the emotion quotient, a face image viewed from the front has more “gentleness” associated with Positive, and a face image viewed from another angle (e.g., right side) has more “severity” associated with Negative. Impression evaluation such as strong feeling is possible. As described above, in the present embodiment, it is possible to objectively evaluate the multifacetedness of the same face, in which various expressions can be felt depending on the viewing angle.
In addition, since the complexity of a facial expression that expresses multiple emotions can be quantified based on the emotion index of each of a plurality of different face information data (face images), a mixed positive and negative facial expression such as a wry smile can be obtained. However, the complexity can be evaluated objectively. As a result, the degree of contribution of emotional valence such as Positive and Negative can be evaluated with respect to the mixed facial expressions.
Furthermore, when correspondence information based on face information data obtained by extracting a frame image of the face of person H to be evaluated from moving image data is used [impression diagnosis (ii), which will be described later], transitions in emotional quotients over time can be grasped. Therefore, it is possible to accurately grasp the impression received from the face that accompanies various changes in facial expression.

The system 1 of the present embodiment includes the emotional index calculator 3B as well as the emotional index calculator 3C. Facial expressions can be evaluated with higher accuracy. That is, the impression received from the evaluation target person H's face is evaluated using the emotion indices of a plurality of face images. For example, multifacetedness and complexity can be comprehensively evaluated using the above-described calculation formulas (1) to (13) for a plurality of face images acquired using the imaging devices C1 to C9.
As described above, the system 1 of the present embodiment uses emotion quotients corresponding to the same or a plurality of facial movement units for each plurality of face information data or for a plurality of face information data as analysis materials, and analyzes the analysis results. Comprehensive evaluation (e.g., emotional index, etc.) makes it possible to objectively and accurately evaluate the multifaceted and complex impression received from the subject's face. Such effects will be specifically described in the examples described later.

Next, the impression evaluation method of the present invention will be described with reference to the drawings, taking as an example control processing (hereinafter referred to as "impression evaluation processing") executed in the impression evaluation system 1 described above. For convenience of explanation, the following description assumes that the imaging devices C1 to C9 and the control device 2 are in a state of being able to communicate with each other.

As shown in FIG. 2, the impression evaluation process according to the present embodiment is mainly executed in the control section 3, and starts from performing the process in step S100. The CPU of the control unit 3 is the subject of execution of each step shown below.
First, in step S100, the face information acquisition unit 5 executes a face information acquisition step of acquiring a plurality of different face information data. In this embodiment, the face information acquisition unit 5 receives a plurality of face image data of the person H to be evaluated input from the imaging devices C1 to C9, stores the face image data in the storage unit 4, and stores the face image data in the storage unit 4. The image data is specified as a processing target after step S200. The photographing of the face image by the imaging devices C1 to C9 is executed according to the user's operation of the system 1. FIG.

In the subsequent step S200, the facial expression action evaluation section 3A executes the facial expression action evaluation process for the facial information data (face image data) acquired in step S100. That is, the facial expression motion evaluation step evaluates the strength of the movement of each facial expression motion unit based on the movement of facial muscles for each piece of face information data (face image data). In this embodiment, the facial expression action evaluation section 3A evaluates the action intensity (movement intensity) for each “AU” with respect to the face image data. Specifically, the facial expression action evaluation unit 3A, for each face image data transmitted from the imaging devices C1 to C9, according to the FACS estimation automatic program, facial parts corresponding to each "AU" (for example, around the eyebrows, After extracting the eyes), control is performed to evaluate the intensity of movement for each "AU".

For example, as shown in FIG. 3, the facial expression action evaluation unit 3A assigns "AU6:0", "AU12:0", "AU2:0", . . . AU26:0", and for image data captured by the imaging device C2, "AU6:0", "AU12:1", "AU2:0" . . . Control is performed to evaluate the intensity of motion for each “AU”. “C1” to “C9” in the emotion quotient patterns in FIG. 3 indicate image data captured by “imaging device C1” to “imaging device C9” (see FIG. 1), respectively. FIG. 3 shows facial images of imaging devices C1 to C9 together with emotion quotient patterns.
After performing the facial expression action evaluation step of step S200, the process proceeds to step S300.

At step S300, the emotion quotient calculation unit 3B executes an emotion quotient calculation step. That is, based on the evaluation result of the facial movement evaluation process and the emotion associated with the facial movement unit, the emotion quotient of the facial movement unit is calculated. In the present embodiment, the emotion quotient calculation unit 3B calculates the emotion quotient for each "AU" of the face image data based on the evaluation result by the facial action evaluation unit 3A and the emotion associated with "AU". . Emotions associated with "AU" in this embodiment are indicated by emotional valences such as "Positive", "Negative" and "Neutral".

For example, as shown in FIG. 3, the emotional quotient calculation unit 3B, according to the FACS estimation automatic program, for "AU6", "emotional valence: Positive", "C1:0", "C2:0", "C3: 0" ... "C9: 0", and for "AU12", "Emotional valence: Positive", "C1: 0", "C2: 1", "C3: 0" ... "C9: 0" ... For "AU26", the emotional quotient is set as "emotional value: Neutral", "C1: 0", "C2: 0", "C3: 0" ... "C9: 0". Control the calculation.

In the present embodiment, when calculating the emotional quotient, emotional valences (“Positive”, “Negative” and “Neutral”) associated with “AU” are used. ' (eg, 'Happy', 'Sad', 'Angry' and 'Fear') may be used.
After performing the emotion quotient calculation step of step S300, the process proceeds to step S400.

At step S400, the emotion index calculator 3C executes an emotion index calculation process. That is, the emotion index of the face information data is calculated based on the plurality of emotion indexes. In this embodiment, the emotion index calculator 3C calculates the emotion index of the face image based on the emotion index for each “AU”. For example, for single face information data, an emotion index is calculated based on multiple types of emotion indexes. Alternatively, an emotion index is calculated based on one kind of emotion index related to a plurality of face information data.

The emotion index calculation unit 3C of the present embodiment performs control to calculate the emotion index of the face image according to the calculation formulas [for example, equations (1) to (13)] for obtaining the emotion index stored in the storage unit 4.
For example, as shown in FIG. 3, the emotion index calculation unit 3C provides "positivity: 0", "negativeness: 0", "neutrality: 0", "emotional impression: 0", ""complexity:0" . First, the emotion index is calculated based on the emotion index of the face information data obtained for each of the imaging devices C1 to C9. In addition, the emotion index calculation unit 3C calculates "average strength: 0.00" and "multifacetedness: 0.00" for "AU6", . : 0.00", an emotion index is also calculated by integrating the emotion index of the face information data obtained from the plurality of imaging devices C1 to C9 for each "AU".
After performing the emotion index calculation step of step S400, the process proceeds to step S500.

In step S500, the information presenting section 3D executes an information presenting process. That is, the correspondence information between the face information data and the emotion quotient is presented. Alternatively, corresponding information including emotion indices of a plurality of face information data is presented. In this embodiment, the information presentation unit 3D presents the correspondence information between the emotional quotient and the facial image obtained by the emotional quotient calculator 3B, or the correspondence information between the emotional quotient and the facial image obtained by the emotional quotient and the emotional index calculator 3C. presents. The correspondence information is an emotion quotient pattern, a graph, or the like, and the type of correspondence information to be presented is set in advance by the user of the system 1 . Also, the correspondence information is presented in one of the modes e1 to e3 described above.
After performing the information presentation step of step S500, the control unit 3 terminates this impression evaluation process.

When the impression evaluation process is finished, the evaluation result of the information presenting section 3D is stored in the storage section 4. FIG. Alternatively, the evaluation result is displayed on the display unit 6 or a display unit of a separate information terminal (for example, a smart phone) connected via a network.

In the present embodiment, the processing from step S100 to step S500 is executed under the control of the control device 2, but instead of the control device 2, it is also possible to perform it manually. That is, in the impression evaluation method of the present embodiment, any one or two or more of the facial information acquisition process, the facial expression action evaluation process, the emotion quotient calculation process, the emotion quotient calculation process, and the information presentation process are performed manually. may be executed.
For example, when a person performs the facial expression motion evaluation step, the person visually confirms the facial expression of the person H to be evaluated, and evaluates the strength of the motion of each facial expression motion unit. In this case, a FACS-certified coder simultaneously executes the facial information acquisition process and the facial expression action evaluation process by manual coding.

A usage example of the correspondence information presented in the information presentation step of step S500, that is, evaluation of the impression received from the face of the person to be evaluated using the correspondence information will be described in detail. Since the correspondence information of this embodiment includes the emotion quotient and the emotion index for each of a plurality of different face information data (face images), in evaluating the impression using the correspondence information, the evaluation subject H Multifacetedness and complexity of faces can be evaluated with high accuracy.
Impression evaluation using correspondence information may be based on a plurality of emotional quotients, or may be based on a single emotional quotient. For example, among the AUs shown in FIG. 3, the impression may be evaluated using the emotional quotients of some AUs. More specifically, the impression of the nine face image data in FIG. 3 may be evaluated based on the emotion quotient of AU6 and the emotion quotient of AU1, and the multifaceted degree based on the emotion quotient. In this case, it is possible to evaluate the impression by paying attention to a part of the face such as the eyes and mouth. It is effective to use the emotion indices of AU1 to 7 for the evaluation of the peripheral portions of the eyes such as the eyes and eyebrows of the face. It is effective to use the emotional index of AU 10-28 to evaluate the impression around the mouth.
The impression diagnosis and makeup impression evaluation methods (i) and (ii) described later may be performed based on the emotion index and the emotion index of a part of the face such as the eyes and mouth in the corresponding information.

As an example of impression evaluation using correspondence information, there is a mode of performing impression evaluation of the evaluation target person H's face using emotion indexes or emotion indexes of a plurality of different face information data. For example, an impression diagnosis of the person H to be evaluated may be performed using an emotional index or an emotional index.
Further, the impression evaluation method may include a step of showing a plurality of types of emotion indices in the form of graphs (corresponding information) based on face information data acquired under different conditions. For example, as shown in FIG. 4, a graph is created in which the vertical axis and the horizontal axis are different emotional indexes, and the numerical values of the emotional indexes of the person H to be evaluated are plotted on the graph to make an impression diagnosis. may be performed. In FIG. 4, positivity and negativity are used as emotional indices on the vertical and horizontal axes. In this case, it is possible to acquire facial image data from a plurality of different angles, calculate an emotion index from the facial image data, and comprehend the attribute of the impression of the person to be evaluated. The attribute of the impression can be grasped based on, for example, one or both of the emotional quotient and the emotional index, depending on whether the impression is classified into emotional valence or basic emotion such as Positive or Negative. Also included in such a category is "expressionlessness," which does not correspond to any emotional valence or basic emotion.

In the impression diagnosis, (i) face image data of different persons are acquired from a plurality of different angles, emotion indices are calculated from the face image data, and corresponding information obtained by plotting the values of the emotion indices may be used. . In the case of FIG. 4, the difference in impression between the person a1 and the person b can be grasped, and it can be used as a study material for studying the expression and makeup method that are close to the impression of either the person a1 or the person b.

In the impression diagnosis, (ii) face image data of the same person is acquired from a plurality of different angles at different timings, an emotion index is calculated from the face image data, and the numerical value of the emotion index is plotted. Information may be used. “Obtaining facial image data at different timings” means acquiring facial image data of the same person over time, or obtaining facial image data of the same person by changing the environment and situation in which the subject of evaluation is placed. Including getting. In the case of FIG. 4, the emotion index values a1 and a2 of the same person whose face image data was acquired are distributed, and the situation when the face image data was acquired indicates the impression of the same person (points a1 and a2 in FIG. 4). ) can be understood.

As for the impression diagnosis of (ii), face image data of the same person is acquired over time in the face information acquisition step, an emotional index and an emotional index are calculated based on this, and impression evaluation is performed using corresponding information including this. conduct. As a result, the impression (expression) of the person to be evaluated H that changes over time can be analyzed. In this case, the face information data (face image data) of the person to be evaluated H may be obtained from different angles or may be obtained from one angle.
For example, as shown in FIG. 5, face image data acquired from one angle (the front in FIG. 5) is acquired over time, and the emotion quotient and emotion index are calculated from the face image data. In the information presentation step (impression diagnosis), by plotting the calculated emotional quotients and emotional indexes on graphs such as FIG. 4, it is possible to grasp impressions that change over time (points a1 and a2 in FIG. 4).

Moreover, when acquiring face image data over time, it is also possible to image with a plurality of moving image capturing devices. In this case, from the viewpoint of evaluating multifacetedness with higher accuracy, it is preferable to image the face of the person to be evaluated from a plurality of angles, as with imaging devices C1 to C9 (see FIG. 1). Alternatively, a plurality of frame images may be extracted from the moving images acquired by each moving image capturing device, and impression evaluation may be performed based on these frame images. Since impression evaluation can be performed using a plurality of frame images captured from a plurality of angles in this way, it is possible to obtain more accurate evaluation results.
In Example 6, which will be described later, impression evaluation was performed using a plurality of frame images extracted from a moving image. Impression evaluation may be performed using a plurality of still images captured over time.

From the viewpoint of further improving the accuracy of this impression diagnosis (ii), the face image data of the same person acquired over time in the face information acquisition step should be acquired at the timing when the facial expression of the person to be evaluated changes. is preferred. This makes it possible to analyze with high accuracy the impression received from the subject's face, including the factors that contributed to the change in facial expression. Expressions of such persons to be evaluated include expressionlessness. For example, when extracting multiple frame images from a video, by selecting facial images that show changes in the facial expressions of the person being evaluated, we can determine the temporal transition of emotional indicators such as positivity, negativity, or complexity. It can be grasped with high accuracy. The facial image to be subjected to impression diagnosis (ii) may be artificially selected from a plurality of facial images (including moving images) acquired over time. You may extract (select) by well-known techniques, such as.

Furthermore, the impression diagnosis can be used in (iii) a makeup impression evaluation method that compares different makeup impressions. In the makeup impression evaluation method, the impression evaluation method is performed for each of the bare face before applying makeup and/or the makeup face with makeup, and the impression of the facial expression of the bare face and the makeup face is compared, or if the makeup is different. The impressions of the facial expressions of the put-up faces are compared.
For example, as shown in FIG. 6, an impression diagnosis may be performed by plotting numerical values of emotion indices of the same person's natural face b1 and makeup face b2 on a graph. In addition, impression diagnosis may be performed by plotting the numerical values of the emotion indices of different makeup faces b2 and b3 of the same person on a graph. In the graph shown in FIG. 6, similarly to FIG. 4, the vertical axis and horizontal axis use the emotion indices of positivity and negativity. With such a makeup impression evaluation method, it is possible to easily grasp the impression caused by makeup, and to easily propose a makeup method that gives a desired impression. In addition, the evaluation result of the makeup impression evaluation method can be utilized for makeup counseling. In particular, evaluation of the impression based on the emotional index or the emotional index of a part of the face such as the eyes and mouth can be used for point makeup counseling.

Although the impression diagnosis shown in FIGS. 4 to 6 uses positivity and negativity as emotion indexes, other emotion indexes may be used for impression diagnosis. The emotion index used for impression diagnosis can be appropriately used according to the purpose of the diagnosis. An arbitrary emotion index (for example, neutrality) may be used according to the attributes (for example, sex and age) of the person to be evaluated H, requests, and the like. Also, when using a graph showing the distribution of attributes, etc., the horizontal axis (x-axis) and the vertical axis (y-axis) may have different emotion indexes or different combinations of emotion indexes.

Although the present invention has been described above based on its preferred embodiments, the present invention is not limited to the above-described embodiments.
Although the system 1 in the above-described embodiment includes the emotion index calculator 3C, it does not have to include the emotion index calculator 3C. Similarly, the impression evaluation method of the present invention may not include the emotion index calculation step.
Moreover, although the system 1 in the above-described embodiment includes the information presentation section 3D, the information presentation section 3D may not be provided. In this case, the emotional quotient calculated by the emotional quotient calculator 3B or the emotional index calculated by the emotional quotient calculator 3C can be output to another information terminal. In this case, the corresponding information using the emotion quotient or the emotion index may be created by a human using another information terminal, for example.

In addition, when presenting the corresponding information according to the aspect e3, the control device 2 is not provided with an OS (Operating System) such as dedicated software or hardware for the system 1, an on-premise server configuration, etc., but a cloud server SaaS (Software as a Service), Paas (Platform as a Service), and IaaS (Infrastructure as a Service) may be used. With such a configuration, the correspondence information derived by the information presentation unit 3D of the control device 2 can be provided to the user of the system 1 via a general-purpose web browser. That is, the emotional quotient pattern (see FIG. 3) and the emotional quotient graph (see FIGS. 4 and 6) output from the system 1 are displayed on the display unit of the separate information terminal owned by the user via the web browser. It is possible to display.

In the above-described embodiment, an automatic FACS estimation program (e.g., "Affdex" (Affectiva)) is used for generating emotion quotient patterns, etc., but for example, a so-called 3D image capturing analysis device (VECTRA (Canfield)) is used. It is also possible to use In this case, the face of the person to be evaluated may be subjected to three-dimensional shape analysis by a 3D image capturing and analysis device, and an emotion quotient pattern may be generated based on the analysis result. It is also possible to extract the position and intensity (e.g., depth) of facial wrinkles using known image recognition technology, and generate an emotion quotient pattern based on the extracted wrinkle information. .

EXAMPLES Hereinafter, the present invention will be described in more detail with reference to Examples, but the present invention is not limited to such Examples.

(Example 1)
In Example 1, the system 1 shown in FIG. 1 was used to compare and evaluate the impression received from each face of the subject W and the subject M who are the subject H of evaluation. Each face image of the subject W and the subject M was obtained by photographing in the state of "bare face (without makeup)" and "expressionless". Such facial images were taken using the imaging devices C1 to C9 shown in FIG. FIG. 7 shows the emotional quotient pattern of the subject W, and FIG. 8 shows the emotional quotient pattern of the subject M. As shown in FIG.

As shown in FIG. 7, the facial expression of the subject W in the state of "real face (no makeup)" and "expressionless" is "average intensity of positivity: 0.44" and "average intensity of negativity: 0.44". 0.11", "Average intensity of emotional impression: 0.33", "Multifacetedness of emotional impression: 0.50", and "Average intensity of complexity: 0.11".

As shown in FIG. 8, the facial expressions of the subject M in the state of "real face (no makeup)" and "expressionless" are "average intensity of positivity: 0.11" and "average intensity of negativity: 0.11". 0.33", "Average intensity of emotional impression: -0.22", "Multifacetedness of emotional impression: 0.83", and "Average intensity of complexity: 0".

FIGS. 9(a) to 9(c) show graphs for evaluating facial impressions of subjects W and M, which are created using the emotion quotient patterns of FIGS. 7 and 8. FIG. be. In the figure, the point W is the emotional index value of the subject W derived from the emotional index pattern of FIG. 7, and the point M is the emotional index of the subject M derived from the emotional index pattern of FIG. , respectively.

As shown in FIG. 9(a), according to the graph in which the x-axis is "average intensity of positivity" and the y-axis is "average intensity of negativity", the distance from the origin 0 to point W and point M It can be read that the distance is approximately the same as the distance of Therefore, it is possible to derive the result that the intensity (emotional intensity) indicating emotion (emotional valence) is about the same for the subject W and the subject M. In addition, when comparing the positions of point W and point M, point W is plotted at a position closer to positive emotions than point M, so the subject W's face has a relatively "soft impression". Impression evaluation is possible. On the other hand, since the point M is plotted at a position closer to negative emotions than the point W, the face of the subject M can be evaluated as having a relatively "clear impression".

Based on these evaluation results, from the graph in FIG. 9A, it can be seen that the emotional intensity of each face of the subject W and the subject M is about the same, and the face of the subject W is relatively "soft." It is possible to obtain an objective evaluation result such as "impression" of the face and "clear impression" of the subject M's face.

As shown in FIG. 9B, according to the graph in which the x-axis is "average intensity of emotional impression" and the y-axis is "average intensity of complexity", when the positions of points W and M are compared, emotional The point W is plotted at a position where the value is higher than the point M in both the impression (inner impression) and the complexity. For this reason, the subject W's face has a higher "complex inner impression" than the subject M's face, for example, a face with a higher degree of inner impression depending on the viewing angle. can be obtained.

As shown in FIG. 9C, according to the graph in which the x-axis is "average intensity of emotional impression" and the y-axis is "multifaceted degree of emotional impression", comparing the positions of point W and point M shows that emotional The point M is plotted at a position where the value of the multifaceted impression is higher than that of the point W. Therefore, an objective evaluation result is obtained such that the face of the subject M has a higher ``multifaceted inner impression'' than the face of the subject W, for example, a face with an ``impression that looks rich in expression''. Is possible.

(Example 2)
In Example 2, a makeup impression evaluation method was performed. FIG. 10 shows emotion quotient patterns based on the face image of the subject W with a bare face and the face image of the subject W with makeup. Each face image of a real face before applying makeup and a face with makeup applied was photographed with a natural expression. The face images in FIG. 10 correspond to the imaging devices C1 to C9 (see FIG. 1). The makeup of the makeup face in FIG. 10 (hereinafter referred to as “makeup 1”) is dark overall, “foundation: relatively matte and bright,” “eyebrow: relatively dark and thick,” and “eye makeup: The shadows around the eyes are strong" and "lip makeup: relatively dark red or pink".

As shown in FIG. 10, the facial expressions of the subject W with the “real face” are, for example, “average intensity of positivity: 0.44”, “average intensity of negativity: 0.11”, “complexity The facial expression of the subject W who applied “Makeup 1” was “average intensity of positivity: 0.22”. ', 'average intensity of negativity: 0.44', 'average intensity of complexity: 0', and 'multifaceted degree of emotional impression: 1.2'.
As shown in FIG. 10, comparing the emotion quotient pattern when the "bare face" is applied with the emotion quotient pattern when "makeup 1" is applied, it is found that when "makeup 1" is applied to the "bare face", "complex "Average intensity of positivity" from "0.11" to "0" (difference: -0.11), and "Average intensity of positivity" from "0.44" to "0.22" (difference: - 0.22)”, while the “multifaceted degree of emotional impression” increased from “0.50” to “1.2” (difference: +0.7).

From these results, in the makeup method of "Makeup 1", the "average intensity of complexity" and the "average intensity of positivity" decreased, resulting in a "clear impression" and a "multifaceted emotional impression". It is possible to make an objective evaluation, for example, that the "multifaceted impression" is emphasized by the amount that the "degree" is increased.

Next, a comparative example in which makeup different from "makeup 1" is applied to the subject W described above will be described with reference to FIGS. 11 and 12. FIG.
FIG. 11 shows a face image of a subject W who has makeup different from "makeup 1" (hereinafter referred to as "makeup 2") and emotion quotient patterns based on the face image. FIG. 12 shows emotion quotient patterns based on the face image (not shown) of the subject W who has applied makeup different from "makeup 1" and "makeup 2" (hereinafter referred to as "makeup 3"). .
“Makeup 2” and “makeup 3” according to the second embodiment will be described. “Makeup 2” according to the second embodiment is generally thin, “foundation: thin”, “eyebrow: relatively light color and thin”, “eye makeup: weak shadow around the eyes”, “lip Make-up: It is based on a make-up method that results in a relatively light red or pink color. In addition, "Makeup 3" is based on a makeup method such that all of "Foundation", "Eyebrow", "Eye Makeup" and "Lip Makeup" have a makeup density intermediate between "Makeup 1" and "Makeup 3". It is.

The facial images of each of the makeup faces with "Makeup 2" and "Makeup 3" were both captured with a natural expression in the same manner as the face images with "Normal face" and "Makeup 1" described above. These face images correspond to the imaging devices C1 to C9, respectively, and were captured using the imaging devices C1 to C9 (see FIG. 1).

As shown in FIG. 11, the facial expressions of the subject W who applied the “makeup 2” are, for example, “average intensity of positivity: 0.333 . . . ”, “average intensity of negativity: 0”, The results were "average intensity of complexity: 0" and "multifaceted degree of emotional impression: 0.707...".

As shown in FIG. 12 , the facial expressions of the subject W who applied “makeup 3” are, for example, “average intensity of positivity: 0.67”, “average intensity of negativity: 0”, “complexity 0” and “multifaceted degree of emotional impression: 0.87”.

FIGS. 13A to 13C show evaluation results by the makeup impression evaluation method, which are created using the emotion quotient patterns of FIGS. 11 and 12. FIG. Specifically, it is a graph for evaluating the impression received from the face of the subject W with a different makeup face.
As shown in FIG. 13A, according to a graph in which the x-axis is "average intensity of positivity" and the y-axis is "average intensity of negativity", when "makeup 1" is applied to "real face" , while the “average intensity of positivity” decreased (“0.44” → “0.22”), the “average intensity of negativity” increased (“0.11” → “0.44 ”).

In addition, when "makeup 2" was applied to the "real face", both the "average intensity of positivity" and the "average intensity of negativity" decreased ("average intensity of positivity: 0.44→0. 333 .
Furthermore, when “makeup 3” was applied to the “real face”, the “average intensity of positivity” increased (“0.44” → “0.67”), whereas the “average It can be read that the strength” has decreased (“0.11”→“0”).

From these evaluation results, for example, "Makeup 1" is used to create a "clear impression", "Makeup 2" is used to create a "neutral impression" (calm impression), and "Soft impression" is used. It is possible to make a proposal such as "Make 3".

Further, as shown in FIG. 13B, according to a graph in which the x-axis is "average intensity of positivity" and the y-axis is "average intensity of complexity", "makeup 1 ” to “makeup 3”, the “average intensity of complexity” decreases from “0.11” to “0”. That is, from such an evaluation result, applying any of “Makeup 1” to “Makeup 3” to the “bare face” reduces the “complexity” (complexity) that influences the inner impression of the subject W. It is possible to make an evaluation such as

Analyzing each emotion quotient pattern in FIGS. 10 to 12, among the “real face” face images, only the face image corresponding to the imaging device C6 has “complexity: 1”, and the other face images (“ It is presumed that the cause is that all face images of "real face" and "makeup 1" to "makeup 3" have "complexity: 0". In other words, it can be seen that the direct cause is that the "negative degree" of the face image corresponding to the imaging device C6 is lowered to "0" by applying makeup to the "real face" (see above). See "Max Negative," "Max Complexity," "Mean Intensity," etc. in the calculation formula).

Based on the above evaluation results, it is also possible to propose a makeup method that does not reduce the "complexity" (complexity). For example, out of the face images corresponding to the image pickup device C6, an image that emphasizes "AU2" (movement to raise the outside of the eyebrows) or "AU4" (movement to lower the eyebrows) derived from "Negative" (emotional valence). You can suggest make-up. As a result, it is possible to effectively prevent the value of the "negative degree" from decreasing, and as a result, it is possible to enhance the subject's W internal impression.

In this case, it is also possible to additionally propose makeup that emphasizes "AU12" (movement to pull up both ends of the lips) derived from "Positive" (emotional valence). That is, in addition to the above-described "Mean Negative", the value of "Mean Positive" also increases, and as a result, the value of "Complexity" can be further increased. Therefore, by applying makeup that emphasizes “AU12” in addition to “AU2” and “AU4”, it is possible to further enhance the inner impression of the subject W (“Max Negative” in the above calculation formula). intensity), Max Complexity and Mean Intensity, etc.).

Next, the graph of FIG.13(c) is demonstrated. As shown in FIG. 13(c), in a graph in which the x-axis is "average intensity of positivity" and the y-axis is "multifaceted degree of emotional impression", "makeup 1" to "makeup 1" to "makeup 3”, the “multifaceted degree of emotional impression” increased from “0.5” (“makeup 1: 1.2”, “makeup 2: 0.707 . . . ” and “makeup 3:0.37").

From these evaluation results, it is possible to secure a multifaceted impression even if any of “Makeup 1” to “Makeup 3” is applied. "Makeup 1" and "Makeup 2", which are lower than the "bare face", and "Makeup 3", where the "average strength of positivity" is higher than the "bare face", if you want to make a "glamorous impression" ” can be proposed.

(Example 3)
Next, Example 3 will be described with reference to FIG. In Example 3, the effect of eyebrow applied to the peripheral portion of the eyes of the subject W was evaluated.
FIG. 14 shows an image of the area around the eyebrows of the subject W to whom eyebrows have been applied, together with the emotion quotient pattern based on the image data. The images around the eyebrows are cut out from the natural facial images (9 sheets) and the makeup 1 facial images (9 sheets) used in the second embodiment.

As shown in FIG. 14, in Example 3, the impression received from the face of the subject W (evaluation subject H) was evaluated based only on "AU" related to the eyebrow circumference. Specifically, in the third embodiment, "AU1" (movement to raise the inside of the eyebrow), "AU2" (movement to raise the inside of the eyebrow), "AU4" (movement to raise the inside of the eyebrow), and "AU4" (movement to raise the inside of the eyebrow) are used as motion units around the subject W's eyebrow. The effect of the eyebrow applied to the subject W was evaluated based on "AU6" (movement to lift the cheek) and "AU7" (movement to tighten the eyelid).
Comparing the emotional quotient pattern of the "real face" with the emotional quotient pattern of applying eyebrows, when the "real face" was given eyebrows, the "multifaceted degree of emotional impression" increased from "0.33" to "0.88" (difference: +0.55).
From these results, it is possible to make an evaluation such as emphasizing the "multifaceted impression". As described above, according to the third embodiment, it is possible to evaluate the impression of the subject W without any problem even with an image of a part of the face.

(Example 4)
Next, Example 4 will be described with reference to FIG. Example 4 is a modified example of Example 3, and evaluates the effect of eye makeup applied around the eyes of the subject W. FIG.
FIG. 15 shows an image around the eyes of the subject W who has applied eye makeup and the emotion quotient pattern based on the image data. The images around the eyes are cut out from the natural face images (9 sheets) and the makeup 1 face images (9 sheets) used in the second embodiment.

As shown in FIG. 15 , in Example 4, similar to Example 3, based on only the “AU” related to the eye periphery, which is a part of the face, the subject W (evaluation subject H) It is designed to evaluate the impression received from the face. Specifically, in the fourth embodiment, "AU5" (motion to raise the upper eyelid), "AU6" (motion to lift the cheek), and " The effect of the eye makeup applied to the subject W was evaluated based on AU7" (movement to tighten the eyelids).

As shown in FIG. 15, according to the emotion quotient pattern when eye makeup is applied, the numerical values based on the face images of the angles captured by the imaging devices C1 to C6 uniformly increased in “AU5”. From the results shown in FIG. 15, it was found that the "AU5" value generally increased from the front to the right side. As a result, it is possible to propose to the subject W to apply the same eye makeup on the left side as on the right side so that the value of "AU5" increases uniformly. According to the fourth embodiment, similarly to the third embodiment, it is possible to perform the impression evaluation without any problem even if the image of a part of the face is used.

(Example 5)
Next, Example 5 will be described with reference to FIGS. 16 to 19. FIG. Example 5 evaluates the impression received from the face of a subject W (evaluation subject H) who applied a foundation (hereinafter referred to as "FD").
FIG. 16 shows the emotion quotient pattern based on the facial image when the subject with the bare face expresses his/her facial expression, FIG. show the emotion quotient patterns based on the facial images when the subjects to whom "FD2" different from "FD1" was applied showed facial expressions. FIG. 19 shows graphs for evaluating the impression received from the subject's face created using each emotion quotient pattern of FIGS. 16 to 18, respectively. The face images described above were taken with natural facial expressions, and were taken by the image pickup devices C1 to C9 (see FIG. 1) in the same manner as in Examples 1 to 4 above. In addition, “FD1” is an FD that covers unevenness such as fine wrinkles and pores when applied to the bare skin to create a finish with a reduced gloss (so-called “matte skin”), and “FD2” is , and FD for achieving a natural finish with a glossy feeling like bare skin (so-called “glossy skin”), respectively.

As shown in FIG. 16, the facial expression of the target person when the facial expression is expressed is, for example, “average intensity of positivity: 0.44”, “average intensity of negativity: 1.00”, “emotional impression Average strength: -0.56" and "average strength of complexity: 0.22".
As shown in FIG. 17, the facial expressions of the subject who applied “FD1” when expressing expressions are, for example, “average intensity of positivity: 0.67”, “average intensity of negativity: 1.89”, The results were "average intensity of emotional impression: -1.22" and "average intensity of complexity: 1.22".
As shown in FIG. 18, the facial expressions of the subject who applied “FD2” when expressing expressions are, for example, “average intensity of positivity: 1.00”, “average intensity of negativity: 1.56”, The results were "average intensity of emotional impression: -0.56" and "average intensity of complexity: 1.56".

As shown in FIG. 19(a), according to the graph in which the x-axis is "average intensity of positivity" and the y-axis is "average intensity of negativity", the distance from the origin 0 to the point "real face", Comparing the distance from the origin 0 to the point "FD1" and the distance from the origin 0 to the point "FD2", it can be read that the distance (length) from the origin 0 is clearly longer in the latter. Since this length represents the "strength of emotion", it is possible to evaluate from this graph that the emotion is emphasized when "FD1" and "FD2" are applied.

Further, according to the graph of FIG. 19(a), when "FD1" is applied to the "real face", the "average intensity of positivity" changes from "0.44" to "0.67" (difference: +0. 23), it can also be seen that the "average intensity of negativity" has increased from "1.0" to "1.89" (difference: +0.89). On the other hand, when "FD2" is applied, the "average intensity of positivity" changes from "0.44" to "0.1" (difference: +0.56), and the "average intensity of negativity" increases from "1.0" to "1.56" (difference: +0.56). From these results, when "FD1" is applied, it can be evaluated that "negative degree" is mainly increased, while when "FD2" is applied, both "negative degree" and "positive degree" It is possible to make an objective evaluation such as whether the Note that the point "Neutral face" in the graph is plotted based on the emotional quotient pattern based on the face image of the subject W captured in the "bare face (without makeup)" and "expressionless" state in FIG. This is shown in the graph as a reference. The same applies to the point "Neutral Face" in the graph of FIG. 19(b).

As shown in FIG. 19B, according to the graph in which the x-axis is "average intensity of emotional impression" and the y-axis is "average intensity of complexity", when "FD1" is applied to "real face", , while the “average intensity of emotional impression” decreased from “−0.56” to “−1.22” (difference: −0.66), the “average intensity of complexity” was “0.22”. to "1.22" (difference: +1.0). On the other hand, when "FD2" is applied, the "average intensity of emotional impression" does not change from "-0.56" to "-0.56" (difference: 0), and the "average intensity of complexity ' rises from '0.22' to '1.56' (difference: +1.0). From these results, when "FD1" was applied, the "average intensity of emotional impression" (comprehensive emotion) shifted toward the negative side ("emotional impression" = Max (X _{AU positive} ) - Max (X _{AU Negative} ), and the "average intensity of complexity" (complexity) increases. It is possible to make an objective evaluation such as an increase in the average intensity of "(complexity)".
From such evaluation results, for example, when "FD1" and "FD2" are applied, both can increase the "complexity" (complexity) that affects the inner impression of the subject, but further A suggestion can be made to the subject that it is 'FD2' that increases 'complexity'.

(Example 6)
In Example 6, the impression received from the evaluation subject's face when talking about an interesting story (favorite story) and the evaluation subject's face when talking about an uninteresting story (dislike story). It was evaluated by comparing the impression received from.
In Example 6, emotion quotient patterns were created based on facial images of persons to be evaluated at the time of interviews (subjects in this embodiment). In FIG. 20, (A1) shows the emotional quotient pattern when the subject of evaluation is telling a story that the subject is interested in (favorite story), and (B1) is the emotion index pattern when the subject of evaluation is telling a story that the subject is not interested in (dislikes). The emotional quotient patterns for each case are shown.

In the present embodiment, unlike the first to fifth embodiments, the impression received from the face of the person to be evaluated is evaluated based on the face image acquired by one moving image pickup device. Specifically, in this embodiment, the moving image data is classified according to the content of the utterance (a story that one likes, one that one dislikes, etc.), and a plurality of facial images (frames) are extracted at the timing when the utterance ends and the mouth is closed. That is, impression evaluation was performed based on a plurality of frame images (still images) over time.

As shown in FIG. 20, in the case (A1) where the subject of evaluation is telling an interesting story (favorite story), the "average intensity of complexity" was a value of "0.66667". On the other hand, in the case (B1) in which the person to be evaluated talks about a story he or she is not interested in (dislikes), the “average intensity of complexity” is a high value of “5”.
From these results, when the subject of evaluation is talking about an interesting story, the facial expression with "complexity" (complexity) does not (almost) occur, and the subject of evaluation is not interested. When talking, it is possible to evaluate that the face tends to have a "complicated" expression.

(Example 7)
In Example 7, the impression received from the face of an evaluation subject H (subject W in this embodiment) during stress monitoring was evaluated.
A frame image extracted from the moving image of the subject W's face during stress monitoring was used. The moving images were classified according to the contents of the utterance (likes, dislikes, etc.), and a plurality of facial images (frames) were extracted at the timing when the utterance ended and the mouth was closed. Based on this frame image, an emotion quotient pattern based on the face image of the subject W during stress monitoring was created. (A2) shows the emotion quotient pattern when the subject W, who is the subject of evaluation H, is talking about an interesting story (favorite story), and (B2) is the emotion quotient pattern when the subject W is talking about a stressful situation. The emotional quotient patterns for each case are shown.

In this embodiment, the impression received from the subject W's face is evaluated based on moving images captured by a plurality of (two in this embodiment) moving image capturing devices C11 and C12 (for example, video cameras). gone. Specifically, in the present embodiment, two moving image capturing devices synchronized with each other are used to scan the face of the subject W in two directions (in this embodiment, the oblique front left side of the face and the front right side of the face). The image is captured from the oblique side), and the impression received from the subject W's face is evaluated based on a plurality of frame images (each frame image at predetermined timings T1 to T3) extracted from the moving image.

As shown in Table 2, when the subject W is telling an interesting story (favorite story) (A2), the emotion quotient pattern based on the frame image of C11 and the emotion quotient pattern based on the frame image of C12 For example, "difference in average intensity of positivity: 0.67", "difference in frame dependency of positivity: 0.58", "difference in average intensity of negativity: 0.00", The results were "difference in frame dependence of negativity: 0.00", "difference in mean intensity of complexity: 0.33", and "difference in frame dependence of complexity: 0.58".

On the other hand, when talking about a stressful situation (B2), comparing the emotion quotient pattern based on the frame image of C11 and the emotion quotient pattern based on the frame image of C12, the "average of positivity "difference in intensity: 1.33", "difference in frame dependence of positivity: 0.84", "difference in average intensity of negativity: 1.00", "difference in frame dependence of negativity: 1.00". 00”, “difference in average intensity of complexity: 2.67”, and “difference in frame dependence of complexity: 3.06”.

From these results, when comparing the case where the subject W is talking about an interesting story and the case where the subject W is talking about a stressful situation, the “positivity” and “negativeness” between C11 and C12 ” and “complexity” are significantly higher in the latter than in the former.

From the above evaluation results, depending on the viewing angle, the impression of subject W's face was higher when subject W was talking about a stressful situation than when subject W was talking about something interesting. , “Positiveness”, “Negativeness” and “Complexity” all tend to give different impressions.

1 impression evaluation system 2 control device 3 control unit 3A facial movement evaluation unit 3B emotion quotient calculation unit 3C emotion index calculation unit 3D information presentation unit 4 storage unit 5 face information acquisition unit 6 display units C1 to C9 imaging device H evaluation subject

Claims (6)

An impression evaluation method for evaluating the impression received from the evaluation subject's face based on the evaluation subject's face information data,
a face information acquisition step of acquiring a plurality of different face information data of the person to be evaluated;
a facial expression movement evaluation step of evaluating the intensity of movement for each facial movement unit based on the movement of facial muscles for each face information data;
an emotion quotient calculation step of calculating an emotion quotient for each facial movement unit based on the evaluation result of the facial movement evaluation step and the emotion associated with the facial movement unit;
an information presenting step of presenting correspondence information between the face information data and the emotion quotients for evaluating the impression received from the face of the person to be evaluated using the emotion quotients of a plurality of the face information data. , the impression evaluation method. an emotion index calculation step of calculating an emotion index of the face information data based on a plurality of the emotion indexes;
2. The impression evaluation method according to claim 1, wherein, in said information presenting step, said correspondence information including said emotion indicators of a plurality of said face information data is presented. 3. The impression evaluation method according to claim 2, wherein in said emotion index calculation step, said emotion index is calculated based on a plurality of types of said emotion index relating to said single face information data. 3. The impression evaluation method according to claim 2, wherein in said emotion index calculation step, said emotion index is calculated based on one type of said emotion index relating to a plurality of said face information data. 5. The impression evaluation method according to any one of claims 1 to 4, wherein said emotion in said emotion quotient calculating step is an emotional valence. An impression evaluation system for evaluating the impression received from the evaluation subject's face based on the evaluation subject's face information data,
a face information acquisition unit that acquires a plurality of different face information data;
a facial movement evaluation unit that evaluates the strength of movement for each facial movement unit based on the movement of facial muscles for each of the facial information data;
an emotion quotient calculation unit that calculates an emotion quotient for each facial movement unit based on the evaluation result of the facial movement evaluation unit and the emotion associated with the facial movement unit;
an information presenting unit for presenting correspondence information between the face information data and the emotion quotients for evaluating the impression received from the face of the person to be evaluated using the emotion quotients of a plurality of the face information data. , impression evaluation system.

Images