JP2008046691A - Face image processor and program for computer - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily estimate the feeling of a person changing due to the secretion of a specific neurotransmitter. <P>SOLUTION: A face image obtained by picking up the face of a person is successively obtained, and a predetermined region whose shape is changing in the face of a person due to the secretion of a specific neurotransmitter is set as a detection object region in the face image, and the change of the set detection object region is detected based on the successively acquired face image, and the feeling of a person to be generated based on the specific neurotransmitter is estimated based on the detected change. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing technique, and more particularly to a technique for estimating a person's mood projected on a face image.

  With the rapid spread and development of computers, there are increasing opportunities for computers to provide services while communicating with people. For computers to communicate with people, it is desirable to be able to understand people's feelings. If computers can understand people's feelings, it will be possible to provide more flexible and diverse services in various fields such as medical fields such as counseling and entertainment fields such as games. .

  Therefore, conventionally, a plurality of sample face images such as a face when the person is tired, a face when the person feels anxiety, a face when the person is impatient, etc. are captured by a camera. A technique has been proposed in which a newly captured face image is stored in a database and matching is performed with reference to the database to determine human emotions (see Patent Document 1).

  However, when judging emotions, for example, it identifies the change in facial shape corresponding to emotions, such as raising the corner of a mouth when laughing. Or with an angry face, it may be insufficient to determine the state of the person with the device.

On the other hand, when a person suddenly feels stress, or when he / she relaxes suddenly, a mood change that is difficult to control by human will occurs. Based on the clinical fact that this change in mood is caused by the secretion of neurotransmitters in the human body, humans are stressed by detecting neurotransmitters such as dopamine, serotonin, and noradrenaline in human urine and blood. Some have proposed a technique for determining whether or not the user feels the problem (see Patent Document 2).
JP 2000-76421 A JP 2000-131318 A

  However, the prior art of Patent Document 1 cannot discriminate a state in which it is difficult to control by human intention, that is, mood, because discrimination based on facial images is limited to emotion discrimination. Moreover, in the prior art described in Patent Document 2, a specialized device is required to detect the secretion of a neurotransmitter directly from the human body, which has been expensive and troublesome. For this reason, it has not been easy to judge a person's mood.

  The present invention has been made in view of the above problems, and one of its purposes is a face image processing apparatus and a computer that can easily estimate a person's mood that changes due to the secretion of a specific neurotransmitter. Is to provide a program.

  In order to solve the above problems, a face image processing apparatus according to the present invention includes an image acquisition unit that sequentially acquires a face image obtained by imaging a human face, and a shape change in the human face due to secretion of a specific neurotransmitter. Based on area setting means for setting a predetermined area to be detected as a detection target area in the face image, and face images sequentially acquired by the image acquisition means, a change in the detection target area set by the area setting means is detected. It includes a change detection means, and a mood estimation means for estimating a person's mood generated based on a specific neurotransmitter based on a change detected by the change detection means.

  Further, a program for causing a computer according to the present invention (including a personal computer, a server computer, a printer, a personal digital assistant (PDA), a mobile phone, a game device, etc., and so on) to function is a face obtained by imaging a human face. Image acquisition means for sequentially acquiring images, area setting means for setting a predetermined area whose shape changes in a human face due to secretion of a specific neurotransmitter as a detection target area in the face image, and sequentially acquired by the image acquisition means A change detection means for detecting a change in the detection target area set by the area setting means based on the face image, and a person generated based on a specific neurotransmitter based on the change detected by the change detection means This is a program for causing a computer to function as a mood estimation means for estimating the mood of a person.

  The program may be stored in a CD-ROM (Compact Disk-Read Only Memory), a DVD-ROM (Digital Versatile Disk-Read Only Memory), a memory card, or any other computer-readable information storage medium.

  In the present invention, a human face image captured by a camera is input to a computer. Then, with respect to the input face image, a predetermined region whose shape changes in the human face due to the secretion of a specific neurotransmitter (for example, a shape change in a predetermined part such as the eye, nose, mouth, eyebrows) is set as a detection target region To do. Then, based on the face images sequentially input to the computer, a change in the detection target area in the face image is detected. Based on the detected change, the human mood generated based on the neurotransmitter presumed to be secreted is estimated.

  According to the present invention, a specific neurotransmitter is secreted by image processing based on a human face image by finding that there is a predetermined region whose shape changes in the human face due to the secretion of the specific neurotransmitter. This makes it possible to easily estimate the mood of a person who changes.

  In the aspect of the invention, the specific neurotransmitter includes at least dopamine, the region setting unit sets the eye as a detection target region, and the change detection unit is set as the detection target region. Detecting a change in curvature of at least one of the upper and lower contours of the eye, wherein the mood estimation means estimates a person's mood based on dopamine secretion based on the change in curvature detected by the change detection means. Features. By doing so, it is possible to easily estimate a person's mood that changes based on the dopamine secretion based on the curvature change of the eye contour.

  In one aspect of the present invention, the specific neurotransmitter includes at least dopamine, the region setting unit sets a plurality of parts included in a face as a detection target region, and the change detection unit A change in the positional relationship of each detection target region set by the region setting unit is detected, and the mood estimation unit is configured to detect a human mood based on dopamine secretion based on the change in the positional relationship detected by the change detection unit. It is characterized by estimating. By doing so, it is possible to easily estimate a person's mood that changes based on the secretion of dopamine based on the change in the positional relationship between the plurality of detection target regions.

  In one aspect of the present invention, the specific neurotransmitter includes at least dopamine, and the change detection unit detects a change in the concentration of an edge included in the detection target region set by the region setting unit, The mood estimation means estimates a person's mood based on dopamine secretion based on a change in edge concentration detected by the change detection means. By doing so, it is possible to easily estimate a person's mood based on the secretion of dopamine based on a change in the concentration of the edge included in the detection target region.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention (hereinafter referred to as embodiments) will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram showing an entire face image processing system including a computer according to the present embodiment.

  As shown in FIG. 1, the face image processing system 1 includes a computer 10 and a camera 100, and the computer 10 and the camera 100 are connected. The camera 100 has a structure in which a lens is combined with a CCD element or a CMOS element, and takes an image projected on the lens at predetermined time intervals (for example, every 1/60 seconds). In the present invention, the face of the person 200 is imaged by the camera 100. The face images of the person 200 captured by the camera 100 are sequentially input to the computer 10. The computer 10 performs image processing on the input face image to acquire information, and estimates the mood of the person 200 projected on the face image based on the acquired information. The configuration of the computer 10 for estimating the mood of the person 200 projected on the face image and an example of processing performed using the configuration will be described in detail below.

  FIG. 2 shows an example of the hardware configuration of the computer 10 according to the embodiment of the present invention. As shown in FIG. 1, the computer 10 includes a processor 12, a memory 14, a graphic interface 16, and an I / O interface 18 as physical configurations, and each unit is capable of data communication with each other via a bus 20. . A display 30 is connected to the graphic interface 16, and an image (including a face image captured by the camera 100) is output based on drawing data stored in a VRAM built in the graphic interface 16. A camera 100 is connected to the I / O interface 18, and image data captured by the camera 100 is input to the computer 10. The I / O interface 18 may be an interface provided in a general personal computer such as a USB interface.

  Details of the face image processing performed using the hardware of the computer 10 will be described below with reference to the flowchart shown in FIG. The face image processing is realized by the operation of each unit of the computer 10 in accordance with a computer program. The computer program may be stored in the memory 14 or a hard disk device (not shown), may be read from an information storage medium such as a DVD-ROM, or downloaded to the computer 10 via a network. It is good as well.

  A human face image is captured by the camera 100, and the captured face image is input to the computer 10 via the I / O interface 18 (S101). The input face image is composed of, for example, M × N two-dimensional lattice pixels, and each pixel is converted into digital data (hereinafter referred to as face image data) having gradation values of R, G, and B components. It is converted and stored in the memory 14.

  In the computer 10, image processing is performed on the face image data stored in the memory 14. For the face image indicated by the face image data, the top left vertex of the face image is set to the origin (0, 0), the X coordinate is set to the right from the origin, and the Y coordinate is set to the bottom.

  In the present invention, a predetermined region whose shape changes in a human face due to secretion of a specific neurotransmitter is set as a detection target region, and the specific neurotransmitter is based on secretion based on a change detected from the detection target region. Estimate a person's mood.

  In this embodiment, the specific neurotransmitter is dopamine. And based on the knowledge of the following experiment, when dopamine was secreted, the area | region where a shape changes in a human face was specified.

  In the experiment, 18 people (healthy adult males) shown in FIG. 5 were allowed to smoke based on the clinical fact that dopamine is secreted when a person smokes, and facial images before and after smoking were respectively taken. In addition, the other three people who did not smoke (two healthy adult men and one woman) performed the same operation (sucking operation) as the smoking operation, and captured the front and back face images respectively. The experiment was conducted after quitting 6 hours or more. 6 to 8 show the results of the above experiment.

  FIG. 6 shows a relative increase in the absolute value of the quadratic coefficient of the quadratic curve approximating the eye contour before and after smoking. It shows that the curvature of the outline of an eye has changed greatly, so that the increase amount is large. In FIG. 6, the results for smokers are shown as white bar graphs, and the results for non-smokers are shown as black bar graphs (the same applies to FIGS. 7 and 8). In the upper right contour shown in FIG. 6, the numbers shown in the bar graph are numbers indicating subjects, 1 to 18 are smokers, and 91 to 93 are non-smokers. In addition, the numbers of other parts are omitted, but the positional relationship of each subject's bar graph is the same as that shown above. In the bar graph of FIG. 6, the vertical axis indicates the amount of change in the absolute value of the secondary coefficient. The graph shows the amount of change in the quadratic coefficient of the approximate curve for the upper and lower contours for each eye. And as FIG. 6 shows, the tendency for the curvature of the outline of an eye to become large in a smoker (1-18) is seen, and the tendency is remarkable especially in a lower outline.

  FIG. 7 shows changes between before and after smoking with respect to the distance between the eyes and the mouth (the distance between the right eye and the right mouth corner, the distance between the left eye and the left mouth) and the distance between the nose and the mouth (the distance between the nose and the lower lip). Indicates the amount. Each distance is normalized by dividing by the distance between both eyes. As shown by the bar graph in FIG. 7, in the smokers (1 to 18) shown by the white bar graph, the distance between the facial parts tends to increase, that is, the face tends to change to a relaxed state. Can be seen.

  FIG. 8 shows the amount of change of the edge component before and after smoking for each part of the face (left and right under the orbit, left and right nostrils, left and right mouth corners, left and right mouth corners, left and right cheeks, and between eyebrows). In the nasal cleft, the diagonal direction is the edge search direction, and between the eyebrows, both the left and right sides are the edge search direction, and in other parts, the top and bottom are the edge search direction. As shown in FIG. 8, when dopamine is secreted, the wrinkles tend to become shallow, and this tendency is particularly remarkable in the eyebrows, under the orbit, and cheeks.

  Based on the above experimental results, when dopamine is secreted, in the human face, the curvature of the upper or lower contour of the eye increases, or the entire surface of the face relaxes. It can be said that the shape change of the resulting cocoon appears. In the present embodiment, dopamine secretion is estimated by detecting the above change based on the face image.

  With reference to FIG. 4, specific criteria for estimating dopamine secretion in the present embodiment will be described. First, in order to detect a change in curvature of the eye contour, the eye 312 is set as a detection target region from the face region 310 included in the face image 300 as shown in FIG. Then, as shown in FIG. 4B, the change that the face surface relaxes as a whole is the distance (d1) between the eye 312 and the mouth 316 and the distance (d2) between the nose 314 and the mouth 316. It is detected based on each change. Accordingly, the nose 314 and the mouth 316 are also set as detection target areas. Moreover, the change that the surface of the face relaxes as a whole is also detected based on the change in the darkness of the eyebrows 318 as shown in FIG. 4C. Therefore, the eyebrows 318 is also set as the detection target area. The change in the darkness of the eyelids may be detected from a part other than the eyebrows, for example, the lower eyelid, the cheek, the nasal lip, the side of the mouth corner, and the mouth corner. In this embodiment, the eye, nose, mouth, and eyebrows are set as detection target regions, and dopamine secretion is estimated based on information obtained from each detection target region.

  Hereinafter, details of the process of estimating the secretion of dopamine according to the amount of information calculated based on the detection target region specified from the face image will be described.

[Identify face area]
First, a process of specifying a face area where a face is projected from the input face image is performed (S102). In the present embodiment, the eye and mouth candidate areas are identified based on the fact that the brightness of the eyes and mouth is lower than that of other parts of the face, and the entire face is determined based on the positions of the candidate areas. This area is specified.

  The eye and mouth candidate area is detected as an area surrounded by a luminance inflection point including a luminance minimum point at which the luminance is minimized and increasing in luminance. The luminance minimum point and the luminance inflection point are determined as follows.

  Based on the face image data stored in the memory 14, a luminance gradient in the Y-axis direction and a Laplacian are calculated. Then, based on the calculated gradient and Laplacian, a luminance minimum point at which the luminance is minimized and a luminance inflection point at which the luminance increases are determined. The brightness minimum point is determined as a point having a gradient of 0 and a positive Laplacian. The luminance inflection point is determined as a point having a positive gradient and a Laplacian of 0.

  Based on the brightness minimum point and the brightness inflection point thus determined, eye and mouth candidate regions are selected from the face image. For the selected eye and mouth candidate regions, an appropriate combination is determined as a positional relationship in each face, and an appropriate combination is specified as the eye and mouth region. A candidate area where a face exists is cut out based on the position of the center of gravity of both eyes thus identified. The extracted candidate area is matched with the face standard pattern, and when it is confirmed that the face is normally projected, the candidate area is set as the face area.

  Note that the face area may be specified by using another known method. For example, the face area included in the face image data may be specified based on the skin color area extracted from the face image data.

  Next, the process (S103) for identifying each part of the detection target area (between eyes, nose, mouth, and eyebrows) from the identified face area will be described.

[Identification of eyes]
Eyes are specified from the face areas specified by the face area specifying process (S102). The identification of the eyes is performed by applying a filter process using an eye contour extraction filter for extracting the eye contour to the face region. The filtering process may be performed on an area including the eye candidate area selected in the face area specifying process. In the eye contour filter, parameters of a filter for separating an image are set so that a dark curved contour of the eye is appropriately extracted with respect to a skin color image around the eye. The details of the filter function and parameters for eye identification may be those described in known documents (for example, Japanese Patent Application Laid-Open No. 2004-30006).

[Specification of nose]
A nose is specified from the face areas specified by the face area specifying process (S102). The nose is specified by detecting an edge in both the X-axis direction and the Y-axis direction of the face area and specifying the nostril. The nostril is specified as a black and round region in the face region. Edges are detected in two directions, the X-axis direction and the Y-axis direction, because the shape of the nostrils can be either vertically or horizontally depending on the orientation of the face. This is because the detection accuracy of the nostril is improved. Here, the search for the nostrils may be performed by narrowing the search area based on the eye positions that have already been specified. The details of the process for specifying the nose may be referred to the description of a known document (for example, Japanese Patent Application Laid-Open No. 2006-38689).

[Mouth identification]
The mouth is specified from the face areas specified by the face area specifying process (S102). To specify the mouth, a pixel group including the color component value in the range of the color component value that defines the color space of the lips is extracted from the pixel group in the face region, and the left and right widths of the region occupied by the extracted pixel group This is done by specifying the mouth according to the aspect ratio with the vertical width. The details of the process for specifying the mouth may be referred to the description of a known document (for example, Japanese Patent Application Laid-Open No. 2003-187247).

[Specification between eyebrows]
An eyebrow space is specified from the face areas specified by the face area specifying process (S102). In specifying the space between the eyebrows, first, the position of the eyebrows is specified. The eyebrow position is searched based on edge detection above the already detected eye position. And based on the searched position of both eyebrows, the intermediate position is specified as the space between eyebrows.

  Next, a process (S104) for calculating a feature amount for detecting a change that occurs in the face due to secretion of dopamine based on each detection target area of the face specified from the face area will be described.

[Calculation of feature values based on changes in curvature of eye contour]
First, based on the eyes detected from the face image, the curvatures of the contours of the upper and lower eyes are calculated. The curvatures of the upper and lower eye contours are calculated based on the coordinate values of a plurality of pixels on the eye contour. In calculating the curvature change amount of the eye contour, the curvatures of the upper and lower eye contours in the standard state are calculated and stored. The stored curvature value may be the curvature of the contour of the eye calculated based on the face image captured at the initial setting, or the eye calculated based on the face image captured for a certain period of time. It is good also as an average value of curvature of the outline of. Then, the difference between the calculated curvature of each of the upper and lower contours and the curvature stored in the memory 14 is calculated, and this value is used as the curvature change amount for each of the upper and lower contours of the eye. In addition, as a feature quantity of the curvature change of the eye contour, a quadratic curve of the contour is obtained by the least square method based on the coordinate value of the pixel on the eye contour, and the value of the quadratic coefficient in the quadratic curve The amount of change may be used.

[Calculation of features based on changes in the relaxed state of the face]
Next, as described above, in the present embodiment, two distances (the distance between the eye and the mouth and the nose and the mouth) are calculated as feature amounts for determining the relaxed state of the face. First, the distance between the eyes and the mouth is calculated from the distance between the position of the center of the straight line connecting both eyes and the position of the center of the straight line connecting both ends (mouth angles) of the mouth. As the distance between the nose and the mouth, the distance between the nose and the lower end of the lower lip of the mouth is calculated. These distances may be calculated by affine transformation in a state where the face image is corrected to face forward, and the influence of the distance change may be suppressed by the face inclination. In addition, in order to suppress the influence of the distance change due to the inclination of the face, for example, the distance and inclination of both eyes may be calculated, and each distance may be corrected based on the value.

  Further, when calculating the distance change amounts of the eyes and mouth, and the nose and mouth, each distance in the standard state is calculated and stored. The stored distance may be a distance calculated based on the face image captured at the initial setting, or an average of the distances calculated based on the face image captured for a certain period of time. It may be a value. Then, a difference between each calculated distance and each distance stored in the memory 14 is calculated, and this value is set as a distance change amount of each of the eyes and mouth and the nose and mouth.

[Calculation of features based on wrinkle depth change]
Next, for the region between the eyebrows identified from the face image, the edge component due to the vertical eyelid of the region between the eyebrows is calculated with the X-axis direction as the edge search direction. Then, a value obtained by integrating the edge components calculated for all the pixels in the interbrow area is set as the vertical edge density. Also, the edge component due to the lying down of the interbrow region is calculated with the Y-axis direction as the edge search direction. A value obtained by integrating the edge components calculated for all the pixels in the interbrow region is defined as the edge density in the horizontal direction.

  In calculating the edge density change amount of each edge density in the vertical direction and the horizontal direction, each edge density in the standard state is calculated and stored. The stored edge density may be each edge density calculated based on the face image captured at the initial setting, or each edge calculated based on the face image captured for a certain period of time. It is good also as an average value of density. Then, a difference between the calculated edge density in the vertical direction and the horizontal direction and each edge density stored in the memory 14 is calculated, and this value is set as each edge density change amount.

[Estimation of dopamine secretion]
Then, based on the calculated curvature change amount, distance change amount, and edge density change amount, it is estimated whether or not dopamine is secreted in the person imaged in the captured face image (S105). Dopamine is presumed to be secreted when the curvature of the eye contour is large, the face is relaxed, and the eyelids are stretched. Therefore, when the curvature change amount is positive, the distance change amounts are both positive, and the edge concentration change amounts are both negative, it is estimated that dopamine is secreted. Note that this estimation criterion is an example, and how to estimate the calculated curvature change amount, distance change amount, and edge density change amount is not limited to the above example.

[Estimation of mood]
Based on the neurotransmitter estimated to be secreted in step S105, the person's mood projected on the captured face image is estimated (S106). In the present embodiment, when it is estimated that dopamine is secreted in step S105, it is estimated that the person feels relaxed.

  According to the computer (face image processing apparatus) according to the above-described embodiment of the present invention, a region where a change can appear in a human face when dopamine is secreted is set as a detection target region, and the set detection target region Based on this change, it is possible to easily estimate a person's mood that changes due to the secretion of dopamine.

  Further, the present invention is not limited to the above embodiment.

  In the present embodiment, description has been made focusing on dopamine as a neurotransmitter, but the technique of the present invention may be applied to other neurotransmitters. In that case, it is good also as estimating the imaged person's mood based on the combination of the neurotransmitter estimated to be secreted.

1 is a system configuration diagram illustrating an entire face image processing system including a computer according to an embodiment. It is a figure which shows an example of the hardware constitutions of a computer. It is a flowchart of face image processing. It is a figure explaining the standard at the time of estimating the secretion of dopamine. It is a table | surface which shows a test subject's information. It is a graph which shows one of an experimental result. It is a graph which shows one of an experimental result. It is a graph which shows one of an experimental result.

Explanation of symbols

  1 face image processing system, 10 computer, 12 processor, 14 memory, 16 graphic interface, 18 I / O interface, 20 bus, 30 display, 100 camera, 200 people, 300 face image, 310 face area, 312 eyes, 314 nose 316 mouths, 318 eyebrows.

Claims (5)

Image acquisition means for sequentially acquiring face images obtained by imaging human faces;
A region setting means for setting, as a detection target region in the face image, a predetermined region whose shape changes in a human face due to secretion of a specific neurotransmitter;
A change detection unit that detects a change in a detection target region set by the region setting unit based on the face images sequentially acquired by the image acquisition unit;
Mood estimation means for estimating a person's mood generated based on a specific neurotransmitter based on the change detected by the change detection means,
A face image processing apparatus characterized by that. The face image processing apparatus according to claim 1,
The specific neurotransmitter comprises at least dopamine;
The region setting means sets the eye as a detection target region,
The change detecting means detects a curvature change of at least one of the upper and lower contours of the eye set as the detection target region,
The mood estimation means estimates a person's mood based on dopamine secretion based on the change in curvature detected by the change detection means.
A face image processing apparatus characterized by that. The face image processing apparatus according to claim 1 or 2,
The specific neurotransmitter comprises at least dopamine;
The region setting means sets a plurality of parts included in the face as detection target regions,
The change detection means detects a change in the positional relationship of each detection target area set by the area setting means,
The mood estimation means estimates a person's mood based on dopamine secretion based on the change in the positional relationship detected by the change detection means.
A face image processing apparatus characterized by that. The face image processing apparatus according to any one of claims 1 to 3,
The specific neurotransmitter comprises at least dopamine;
The change detecting means detects a change in density of an edge included in the detection target area set by the area setting means,
The mood estimation means estimates a person's mood based on dopamine secretion based on the edge concentration change detected by the change detection means.
A face image processing apparatus characterized by that. Image acquisition means for sequentially acquiring face images obtained by imaging human faces;
A region setting means for setting a predetermined region whose shape changes in a human face due to secretion of a specific neurotransmitter as a detection target region in the face image;
A change detection unit that detects a change in a detection target region set by the region setting unit based on the face images sequentially acquired by the image acquisition unit;
Mood estimation means for estimating a person's mood that occurs based on a specific neurotransmitter based on the change detected by the change detection means;
As a program to make the computer function.

Images