CN102906786A - Face feature point position correction device, face feature point position correction method, and face feature point position correction program - Google Patents

Abstract

The facial feature point reliability generation means generates a reliability map of each facial feature point from the facial image. The initial facial feature point position calculation means calculates the position of each facial feature point in the face image based on the reliability map. The off-position facial feature point judgment means judges whether or not each facial feature point is an off-position facial feature point that does not satisfy a prescribed condition. The facial feature point difference calculation means calculates the difference between the position of each facial feature point other than those determined as off-position facial feature points and the position of the corresponding point of the facial feature point. The facial feature point position correction means corrects the determined facial feature point positions based on the judgment result of the off-position facial feature point judgment means and the calculation result of the facial feature point difference calculation means.

Description

Face feature point position correction device, face feature point position correction method, and face feature point position correction program

technical field

The present invention relates to a face feature point position correction device, a face feature point position correction method, and a face feature point position correction program for correcting facial organs (eyes, nose, etc.) detected from an image The location of the feature points.

Background technique

For performing face orientation estimation, face authentication, facial expressions, etc. with high precision, for detecting the positions of feature points (facial feature points) of facial organs such as eyes and nose from images in which faces are captured (face feature point position) and a technique of outputting the detected face feature point position is important. Various methods have been proposed regarding detection of facial feature point positions.

For example, Non-Patent Document 1, which describes a technique for detecting facial feature points and outputting facial feature point positions, discloses a method for correcting detected facial feature point positions based on a statistical face shape model. method. In this method, facial landmarks are first detected by applying a facial landmark detector (each detector has been constructed for each facial landmark) to the face region, and for each detected face The landmarks generate a reliability map indicating the suitability of the facial landmarks.

Then, for each face feature point, a face feature point position that has high reliability and minimizes the difference from the position indicated by the statistical face shape model is searched based on a prescribed evaluation function. In this process, penalties (specifically, no weights) are assigned to facial landmark locations that are far from the corresponding locations indicated by the statistical face shape model. Therefore, even when part or all of the facial feature points are hidden (blocked) by something (occlusion) in the face image, plausible facial feature point positions can be acquired.

Patent Document 1, which describes a technique for detecting facial feature points and outputting facial feature point positions, discloses a method for correcting detected facial feature point positions based on a geometric arrangement (positional relationship). In this method, firstly, the face feature points in the face image are searched in a preset search area based on the input reliability map, and as a result of the search, the initial face feature point positions are obtained through the input reliability map.

The positions of the initial face feature points obtained by searching are corrected based on the positional relationship of the initial face feature points by performing a statistical geometric constraint process using the eigenspace. Subsequently, a search area is preset based on the corrected initial facial feature point positions, and a search for the facial feature point positions is performed again within the preset search area. Finally, by judging the position reliability of the initial facial feature point position and the position reliability of the facial feature point position as the second search result, the plausible facial feature point position is obtained.

prior art literature

patent documents

Patent Document 1: JP-A-2009-211177

Non-Patent Document 1: D.Cristinacce and T.F.Cootes, "A Comparison of Shape Constrained Facial Feature Detectors", in 6th International Conference on Automatic Face and Gesture Recognition 2004, Korea, pp.357-380.

Contents of the invention

technical problem

In the method described in Non-Patent Document 1, when an erroneous reliability map is generated for one or some facial feature points due to some factors (change in lighting conditions, occlusion, etc.) The evaluation function assigns a penalty to the positions of such face feature points, and corrects such face feature point positions to positions close to the average positions calculated from the face shape model. However, since calculation using a prescribed evaluation function (also including the reliability of such facial feature points) is performed. Therefore, there is a problem in that moving the positions of other face feature points is affected by the wrong reliability map.

Meanwhile, in the method described in Patent Document 1, when information with low reliability is input about one or more facial feature points, the positions of such facial feature points are appropriately corrected based on the geometric arrangement (positional relationship), As long as the positions of other facial feature points have been correctly determined. However, if another failure occurs in the normalization to the position, size, and on-screen rotation of the face region due to errors such as face detection, the facial feature point positions deviate as a whole from the standard face represented by the eigenspace shape. In this case, it cannot be judged which facial feature point positions deviate from the standard face shape. Therefore, the geometric arrangement (positional relationship) cannot be properly calculated, and the facial feature point positions cannot be correctly determined.

Therefore, the main object of the present invention is to provide a face feature point position correction device, a face feature point position correction method, and a face feature point position correction program, enabling Even when the feature point input has low reliability information, it can output the position of the facial feature point with high accuracy.

means of solving problems

The face feature point position correction device according to the present invention includes: face feature point reliability generating means for each face feature point from the input image where the face is photographed, generating a feature point corresponding to the face feature point The reliability map indicated by the suitability of the facial feature points, the facial feature points indicate the feature points of the facial organs; the initial facial feature point position calculation means is based on the reliability generated by the facial feature point reliability generation means feature map, calculating the position of the facial feature points in the image of the face being photographed; the out-of-position facial feature point judging device, judging whether each of the facial feature points is an out-of-position facial feature point, as the dissatisfaction based on the set The position of the facial feature point calculated by the initial facial feature point position calculation device and the facial feature point based on the specified condition specified by the position of the corresponding point corresponding to the facial feature point in the statistical face shape model; The feature point difference calculation means calculates the position of each facial feature point except those facial feature points judged as the off-position facial feature points by the out-of-position facial feature point judging means according to a prescribed evaluation function and the difference between the positions of the corresponding points corresponding to the facial feature points; and the facial feature point position correction device, based on the judgment result of the out-of-position facial feature point judging device and the facial feature point difference The calculation result of the calculation device is used to correct the determined facial feature point positions.

The face feature point position correction method according to the present invention includes, for each face feature point from an input image in which the face is photographed, generating a reliability map indicating the suitability of the feature points of the face feature point , the facial feature points indicate the feature points of the organs on the face;

Based on the generated reliability map, calculate the position of the facial feature points in the captured image of the face; judge whether each facial feature point is an out-of-position facial feature point, as the dissatisfaction based on The position of the facial feature point calculated and the facial feature point of the specified condition specified by the position of the corresponding point corresponding to the facial feature point in the statistical face shape model; The difference between the position of each facial feature point other than those facial feature points of the out-of-position facial feature point and the position of the corresponding point corresponding to the facial feature point; and based on each a result of judging whether the facial feature point is the out-of-position facial feature point and a calculation result of the difference between the position of each of the facial feature points and the position of the corresponding point corresponding to the facial feature point , to correct the determined facial feature point positions.

The facial feature point position correction program according to the present invention causes the computer to perform the following operations: a facial feature point reliability generation process, for each facial feature point from an input image where the face is photographed, to generate a reference to the facial feature point The reliability map indicated by the suitability of the feature points of the points, the feature points of the facial feature points indicating the feature points of the face organs; the initial facial feature point position calculation process, based on the facial feature point reliability generation process. The reliability map of the face is calculated to calculate the position of the facial feature point in the image taken by the face; the out-of-position facial feature point judging process is to judge whether each of the facial feature points is an out-of-position facial feature point , as a face that does not satisfy the specified condition specified based on the position of the facial feature point calculated by the initial facial feature point position calculation process and the position of the corresponding point corresponding to the facial feature point in the statistical face shape model Feature point: the facial feature point difference calculation process, according to the prescribed evaluation function, calculates the position of each facial feature point and the described and The difference between the positions of the corresponding points corresponding to the facial feature points; and the facial feature point position correction process, based on the judgment result of the out-of-position facial feature point judgment process and the calculation of the facial feature point difference calculation process As a result, the determined facial feature point positions are corrected.

Advantageous effect of the present invention

According to the present invention, even when information with low reliability is input about one or more facial feature points, it is possible to output high-precision facial feature point positions by correcting the positions of the facial feature points.

Description of drawings

[ Fig. 1 ] A block diagram showing a configuration example of a facial feature point position correction device according to an exemplary embodiment of the present invention.

[ Fig. 2 ] is a flowchart showing the operation of the face feature point position correction device of the exemplary embodiment.

[ Fig. 3 ] is an explanatory diagram showing an example of a face image (an image in which a face is captured) input by a face image input device.

[ Fig. 4 ] is an explanatory diagram showing that facial feature points are expected to be detected in a facial image.

[ Fig. 5 ] is an explanatory diagram showing an example of a reliability map for the right-eye pupil center.

[ Fig. 6 ] is an explanatory diagram indicating a maximum reliability position (position at which reliability is maximized), in which an X mark in the reliability diagram surrounds the pupil center of the right eye.

[ Fig. 7 ] is an explanatory diagram showing positions of initial facial feature points calculated by the initial facial feature point position calculation means.

[ Fig. 8 ] is an explanatory diagram showing an example of two face feature points selected at random.

[ Fig. 9 ] is an explanatory diagram showing an example of a judgment result by the out-of-position facial feature point judging means.

[ Fig. 10 ] is an explanatory diagram showing an example of a result of position correction of facial feature points deviating from a face shape model performed by the face feature point position correcting means.

[ Fig. 11 ] is a block diagram showing an outline of the present invention.

Detailed ways

Referring now to the drawings, a description will be given in detail of an exemplary embodiment of the face feature point position correction device according to the present invention. FIG. 1 is a block diagram showing a configuration example of a face feature point position correction device according to an exemplary embodiment of the present invention.

As shown in FIG. 1 , the face feature point position correction device 1 according to the present invention includes a data processing device 100 and a storage device 200 . The data processing device 100 includes a facial image input device 110, a facial feature point reliability generating device 120, an initial facial feature point position calculating device 130, an out-of-position facial feature point judging device 140, a facial feature point difference calculating device 150 and Facial feature point position correction device 160. The storage device 200 includes a face shape model storage means 210 .

The face image input device 110 inputs an image (face image) in which a face is captured. Based on the facial image input by the facial image input device 110, the facial feature point reliability generating device 120 generates, for each facial feature point, the suitability of the feature points of the facial feature points (eyes, nose, etc.). Reliability graph. The initial facial feature point position calculating means 130 calculates the facial feature point position (the position of the facial feature point) based on the reliability map generated by the facial feature point reliability generating means 120 . Hereinafter, the facial feature point positions calculated by the initial facial feature point position calculating means 130 are referred to as "initial facial feature point positions".

Based on the initial facial feature point positions calculated by the initial facial feature point position calculation means 130, the out-of-position facial feature point judging means 140 judges whether each facial feature point at each initial facial feature point position is located at a deviation (departure) facial feature points away from the position of the statistical facial shape model stored in the facial shape model storage device 210 . The facial feature point difference calculation means 150 calculates the position and sum of each facial feature point except those facial feature points judged as out-of-position facial feature points by the out-of-position facial feature point judging means 140 according to a prescribed evaluation function. The difference between the positions of the corresponding face feature points (corresponding points) in the face shape model is counted. Based on the judgment result of the out-of-position facial feature point judging means 140 and the calculation result of the facial feature point difference calculating means 150, the facial feature point position correcting means 160 corrects and (the statistical face shape stored in the face shape model storage means 210 The position of the facial feature point corresponding to the position of the facial feature point in the model) has a larger position error.

Next, the operation of the face feature point position correction device 1 is described below with reference to the drawings. FIG. 2 is a flowchart showing the operation of the face feature point position correction device of the exemplary embodiment.

First, the face image input device 110 inputs an image (face image) in which a face is captured (step S111). Subsequently, based on the face image input in step S111, the face feature point reliability generating means 120 indicates a reliability map indicating suitability of feature points of face feature points (eyes, nose, etc.) (step S112). The initial facial feature point position calculation means 130 calculates an initial facial feature point position based on the reliability map generated in step S112 (step S113).

Subsequently, based on the initial facial feature point positions calculated in step S113, the out-of-position facial feature point judging means 140 judges whether each facial feature point at each initial facial feature point position is an out-of-position facial feature point ( It is located at a position deviated from (out of) the statistical face shape model stored in the face shape model storage device 210) (step S114). The facial feature point difference calculation means 150 calculates the position and sum of each facial feature point except those facial feature points judged as out-of-position facial feature points by the out-of-position facial feature point judging means 140 according to a prescribed evaluation function. The difference between the positions of the corresponding face feature points (corresponding points) in the face shape model is counted (step S115). Subsequently, based on the judgment result in step S114 and the calculation result in step S115, the face feature point position correction means 160 corrects the face feature point corresponding to (in the statistical face shape model stored in the face shape model storage means 210) The position of the facial feature point whose position error is larger (step S116).

According to this exemplary embodiment, according to a prescribed evaluation function, the positions of facial feature points are corrected by excluding facial feature points located away from (out of) the statistical face shape model from facial feature points detected from a face image. The facial feature point at the position corresponding to the facial feature point position; and according to the prescribed evaluation function, calculate the difference between each (remaining) facial feature point and each corresponding facial feature point in the statistical face shape model The position difference (error). Therefore, high-precision facial feature point positions can be output.

<example>

Next, the configuration and operation according to the exemplary embodiment of the present invention are described in detail using some specific examples. In the facial feature point position correction device 1 according to the present invention of FIG. 1 , the storage device 200 is realized by, for example, a semiconductor memory or a hard disk drive. The facial image input device 110 is realized by, for example, a digital camera. Facial image input device 110, facial feature point reliability generating device 120, initial facial feature point position calculation device 130, out-of-position facial feature point judging device 140, facial feature point difference calculation device 150 and facial feature point position The correcting means 160 is realized, for example, by a CPU (Central Processing Unit) that controls the execution process according to a program. The facial shape model storage device 210 is realized by, for example, a semiconductor memory or a hard disk drive.

The face image input device 110 inputs an image in which a face is captured. FIG. 3 is an explanatory diagram showing an example of a face image (an image in which a face is captured) input by the face image input device 110 . The image input by the face image input device 110 may include not only a face but also a background. The face image input means 110 may also be configured to perform face detection in advance, extract a face area (area occupied by the face) from the face image, and input the extracted face area to the facial feature point position correction device 1 .

According to the facial image input by the facial image input device 110, the facial feature point reliability generation device 120 generates a reliability map (indicating the suitability of the feature points of the facial feature points (eyes, nose, etc.) sex). FIG. 4 is an explanatory diagram showing that facial feature points are expected to be detected in a facial image. In the example of FIG. 4 , facial feature points expected to be detected are indicated by X marks. In this example, a total of 14 X markers are output as shown in Figure 4 (each (left, right) eyebrow ends, each (left, right) eye center and both ends, the lower part of the nose, the center of the mouth and both ends). In this case, the facial feature point reliability generating means 120 generates 14 reliability maps corresponding to these points. Various techniques proposed so far can be used to generate a reliability map indicating the suitability of feature points of face feature points. For example, similar to the technique of Non-Patent Document 1, the reliability map can be generated by applying a detector for each facial feature point to the entire image area input by the data processing device 100. Constructed by AdaBoost with similar characteristics to Haar proposed by Jones.

FIG. 5 is an explanatory diagram showing an example of a reliability map for the pupil center of the right eye. In the example shown in FIG. 5 , each point (position) is indicated by being darker with greater reliability indicating the suitability of the face feature point. The example of FIG. 5 indicates that the reliability is higher not only around the center of the pupil of the right eye, but also around the center of the pupil of the left eye, the right eyebrow, and the area under the nose.

The initial face feature point position calculating means 130 calculates the position of the face feature point (the position of the face feature point) based on the reliability map generated by the face feature point reliability generating means 120 . In the case where it is desired to detect the positions of the 14 face feature points shown in FIG. 4 , the positions of the 14 initial face feature points are calculated. For example, the initial facial feature point position calculating means 130 determines the position (the position with the highest reliability) that maximizes the reliability in each reliability map generated by the facial feature point reliability generating means 120 as each initial feature point s position. In addition to the method of using the maximum reliability position in the reliability map as the position of the initial facial feature point, the initial facial feature point position calculation device 130 can also use the relationship between the reliability of the facial feature point and the position of the facial feature point The position where the product of the prior distribution is maximized is used as the position of the initial facial feature point.

FIG. 6 is an explanatory diagram indicating a position of maximum reliability (position at which reliability is maximized), in which an X mark in the reliability diagram surrounds the pupil center of the right eye. Figure 6 indicates that the reliability is high not only around the center of the pupil of the right eye, but also around the center of the pupil of the left eye, the right eyebrow, and the area under the nose. However, FIG. 6 also indicates that the center of the pupil of the right eye (the position indicated by the X mark) has been selected as the position of the initial facial feature point, since the reliability is highest at the center of the pupil of the right eye.

Based on the initial facial feature point positions calculated by the initial facial feature point position calculation means 130, the out-of-position facial feature point judging means 140 judges whether each facial feature point at each initial facial feature point position is located at a deviation (departure) facial feature points away from the position of the statistical facial shape model stored in the facial shape model storage device 210 . In the face shape model storage means 210, the coordinate values of the 14 face feature points shown in FIG. 4 have been pre-stored as a statistical face shape model.

Incidentally, the coordinate values of the 14 facial feature points may also be determined as an average of the facial feature point coordinate values in a large number of facial images (including individual differences and changes in facial expression and facial orientation). The facial shape model storage device 210 may also pre-store a set of coordinate values obtained by applying the k-means method to the coordinate values of facial feature points in a large number of facial images (including various variations) as a facial shape model. For example, a robust estimation technique can be used to judge whether each facial feature point is an out-of-position facial feature point (located at a position deviated (away) from the statistical face shape model).

A method of out-of-position face feature point judgment (one type of robust estimation technique) using LMedS (least median square) is described here with reference to the drawings. FIG. 7 is an explanatory diagram showing the positions of the initial facial feature points calculated by the initial facial feature point position calculation means 130 . As shown in FIG. 7 , it is assumed here that the initial facial feature point position calculating means 130 uses the calculated initial facial feature point positions of the center of the right eye pupil, the inner corner of the right eye, and the right end of the mouth as some factors (such as , a change in lighting conditions) greatly deviates (leaves) the position of the corresponding position in the face image (the image in which the face is captured).

The out-of-position facial feature point judging means 140 first randomly selects two facial feature points from the 14 facial feature points shown in FIG. 4 . FIG. 8 is an explanatory diagram showing an example of two facial feature points selected at random. The example of FIG. 8( a ) indicates that a facial feature point at the outer corner of the right eye and a facial feature point at the left end of the mouth are selected.

The following detailed description is given by assigning the character “a” or “b” to each facial feature point selected by the out-of-position facial feature point judging means 140 . In the input image, a set of coordinate values of initial facial feature points corresponding to two randomly selected facial feature points is defined as (t _a , t _b ). Each symbol t _a , t _b represents a two-dimensional vector representing coordinate values in the input image. In addition, in the statistical face shape model stored in the face shape model storage means 210, the two facial feature points corresponding to the facial feature point selected by the out-of-position facial feature point judging means 140 (in this example In , the coordinate value set of the point at the outer corner of the right eye and the point at the left end of the mouth) (that is, the set of coordinate values of the point at the outer corner of the right eye and the point at the left end of the mouth) is defined as (k _a , k _b ), such as Figure 8(b) shows. Each symbol k _a , k _b represents a two-dimensional vector representing coordinate values in the face shape model.

Since the coordinate system for the facial feature points in the face shape model and the coordinate system for the initial facial feature points are different from each other, it is necessary to correlate the coordinate values in the former coordinate system and the latter coordinate system with each other . Therefore, the out-of-position facial feature point judging means 140 determines the coordinate transformation p from the coordinate value set (t _a , t _b ) of the initial facial feature point to the coordinate value set (ka _, k _b ) in the face shape model. The coordinate transformation p is a four-dimensional vector, including a translation component along the x-axis direction, a translation component along the y-axis direction, a rotation component along the screen direction, and a scaling component. Subsequently, the out-of-position facial feature point judging device 140 calculates the coordinate values of the remaining 12 initial facial feature points after transformation using the coordinate transformation p and the 12 facial features corresponding to the 12 initial facial feature points in the face shape model. The square error between the local feature points, and then according to the following expression (1) to determine the median value of the 12 square errors:

$\underset{j\&NotEqual;a,b}{\mathrm{med}}\left|\right|p\left(t_j\right)-k_j\left|\right|$ …Formula 1)

The symbol "med" in the above expression (1) denotes a function for calculating the median. The out-of-position face feature point judging means 140 executes the above sequence (random selection of two face feature points, determination of coordinate transformation p, and calculation of square error of coordinate values) multiple times. Then, the out-of-position facial feature point judging means 140 holds the coordinate transformation p that minimizes the median value of the square error as the coordinate transformation p'. Finally, the out-of-place facial feature point judging means 140 judges whether each facial feature point is out-of-place facial feature point. Specifically, if the square error between the coordinate value of the initial facial feature point after transformation by coordinate p' and the coordinate value in the face shape model is greater than a prescribed threshold, then it is judged that the facial feature point is an out-of-position facial feature point .

FIG. 9 is an explanatory diagram showing an example of a determination result by the out-of-position face feature point determination means 140 . In FIG. 9 , each facial feature point judged as an out-of-position facial feature point by the out-of-position facial feature point judging means 140 is indicated by an X mark surrounded by a circle. FIG. 9 indicates that the initial facial feature points for the center of the pupil of the right eye, the inner corner of the right eye, and the right end of the mouth have been judged as out-of-position facial feature points.

Incidentally, the number of facial feature points randomly selected by the out-of-position facial feature point judging means 140 for determining the coordinate transformation p may also be three or more. For example, the out-of-position facial features shown in FIG. A case where the point judging means 140 randomly selects 3 facial feature points from 14 facial feature points. In the input image, a set of coordinate values of 3 initial facial feature points corresponding to 3 randomly selected facial feature points is defined as (t _a , t _b , t _c ). In addition, in the statistical facial shape model stored in the facial shape model storage device 210, the coordinate values of the three facial feature points corresponding to the facial feature point selected by the out-of-position facial feature point judging device 140 A set is defined as (k _a , k _b , k _c ).

The out-of-position facial feature point judging means 140 determines the coordinate transformation p that minimizes the coordinate value from the initial facial feature point set (t _a , t _b , t _c ) to the coordinate value in the face shape model The coordinate value of the initial face feature point corresponding to the previous coordinate value in the coordinate transformation of the set (k _a , k _b , k _c ) is different from other coordinate values in the face shape model (the coordinate values of the remaining face feature points ) between squared errors. Thereafter, similarly to the case where the number of randomly selected facial feature points is 2, the out-of-position facial feature point determining means 140 determines whether each facial feature point is an out-of-position facial feature point.

The facial feature point difference calculation means 150 calculates the position and sum of each facial feature point except those facial feature points judged as out-of-position facial feature points by the out-of-position facial feature point judging means 140 according to a prescribed evaluation function. Differences between positions of corresponding facial feature points in the face shape models are counted.

For example, the facial feature point difference calculation means 150 determines the coordinates of the initial facial feature points other than those initial facial feature points judged as out-of-position facial feature points by the out-of-position facial feature point judging means 140 by the least square method Coordinate transformation p of the value to the coordinate value of the facial feature point corresponding to the initial facial feature point in the face shape model. Thereafter, the facial feature point difference calculation device 150 can use the square error between the coordinate value of each initial facial feature point after coordinate transformation p transformation and the coordinate value in the face shape model as each facial feature point (except The difference between the positions of the facial feature points other than those judged to be out-of-position facial feature points) and the positions of the facial feature points in the statistical face shape model. The facial feature point difference calculation device 150 can also use the coordinate transformation p' (determined by the out-of-position facial feature point judging device 140) to replace the above coordinate transformation p. In addition to the square error between the transformed coordinate value of each initial facial feature point and the coordinate value in the face shape model, the facial feature point difference calculation device 150 can also calculate the coordinates of each initial facial feature point Reliability of facial feature points.

In the case of using the facial feature point reliability, for example, the facial feature point difference calculating means 150 can calculate the reciprocal number (reciprocal number) of the facial feature point reliability and the transformed coordinate value of the initial facial feature point and the face shape The product of the square error between the coordinate values in the model is output as the error between the transformed coordinate value of the initial face feature point and the coordinate value in the face shape model.

Based on the judgment result of the out-of-position facial feature point judging device 140 and the calculation result of the facial feature point difference calculation device 150, the facial feature point position correction device 160 corrects the facial feature point with a large error from the statistical face shape model. Location. The facial feature points corrected by the facial feature point position correction device 160 are those facial feature points judged as out-of-position facial feature points by the out-of-position facial feature point judging device 140, and the facial feature point difference calculation device 150 The calculated position errors (errors from positions in the face shape model) are those face feature points for which a prescribed threshold value or more is greater.

For example, the facial feature point position correcting means 160 may use a face shape model stored in the face shape model storage means 210 for correcting facial feature points. Specifically, the facial feature point position correction device 160 can transform the corresponding facial feature points in the face model (corresponding to the corrected target face feature points) to determine the corrected coordinate values of facial feature points (facial feature points serving as correction targets) that deviate from the face shape model.

FIG. 10 is an explanatory diagram showing an example of a result of position correction of facial feature points deviating from the face shape model performed by the face feature point position correcting means 160 . FIG. 10 indicates that the positions of the right eye pupil center, the inner corner of the right eye, and the facial feature point at the right end of the mouth (which have been judged as out-of-position facial feature points in FIG. 9 ) have been corrected to appropriate positions.

According to this exemplary embodiment, by correcting the position of a facial feature point judged to be located at a position deviated from the position of the corresponding facial feature point in the statistical face shape model, the position of the high-precision facial feature point output to the appropriate location.

Next, the outline of the present invention will be described. FIG. 11 is a block diagram showing an outline of the present invention. As shown in FIG. 11 , the facial feature point position correction device 300 according to the present invention includes a facial feature point reliability generation device 301, an initial facial feature point position calculation device 302, an out-of-position facial feature point judging device 303, a facial feature point Facial feature point difference calculation means 304 and facial feature point position correction means 305.

The facial feature point reliability generation means 301 generates a reliability map indicating the suitability of the feature points of the facial feature points for each facial feature point from the input image where the face is photographed, the facial feature point Feature points indicating the organs on the face. The initial facial feature point position calculating means 302 calculates the position of the facial feature point in the image where the face is captured based on the reliability map generated by the facial feature point reliability generating means 301 . Out-of-position facial feature point judging means 303 judges whether each facial feature point is an out-of-position facial feature point, as a position that does not satisfy the facial feature points calculated based on the initial facial feature point position calculating means 302 and the statistical face A face feature point of a predetermined condition specified by a position of a corresponding point corresponding to the face feature point in the face shape model.

The facial feature point difference calculation means 304 calculates the position sum of each facial feature point except those facial feature points judged as out-of-position facial feature points by the out-of-position facial feature point judging means 303 according to a prescribed evaluation function. The difference between the positions of the corresponding points corresponding to the facial feature points. The facial feature point position correction unit 305 corrects the determined facial feature point positions based on the judgment result of the out-of-position facial feature point judging unit 303 and the calculation result of the facial feature point difference calculation unit 304 .

Such a configuration makes it possible to input a less reliable image (for example, an unclear image) of one or more facial feature points due to some factors (change in lighting conditions, occlusion, etc.) It is also possible to output high-precision facial feature point positions.

The following facial feature point position correction devices (1) to (4) are also disclosed in the above exemplary embodiments:

(1) Facial feature point position correction device, wherein the out-of-position facial feature point judging means 303 uses a robust estimation technique to judge whether each facial feature point is an out-of-position facial feature point.

(2) Facial feature point position correction equipment, wherein, the out-of-position facial feature point judging device 303 repeats the following operations:

Randomly select some facial feature points from the facial feature points;

Transforming the coordinate values representing the positions of the facial features not selected by the selection operation based on the vectors used to transform the selected facial features into coordinate values representing the positions of the corresponding points of the facial feature points the coordinate value at which the position of the point is represented; and

calculating the median of the squared differences between the transformed coordinate values and the coordinate values representing the positions of the corresponding points of the face feature points, and

If the squared error between the coordinate value representing the position of the corresponding point of the facial feature point and the coordinate value of the facial feature point transformed based on a specific vector is greater than a prescribed threshold, the facial feature point is judged to be an out-of-position facial feature point, the specific vector is a vector that minimizes the median value among the vectors used for coordinate value transformation in the repeated operation.

(3) The device for correcting facial feature point positions includes: a facial shape model storage device for storing information related to statistical facial shape models.

(4) Facial feature point position correcting device, wherein, the facial feature point position correcting means 305 determines the position of the facial feature point and the facial feature point as a face feature point determined by the off-position facial feature point judging means 303. Correction is performed at positions of facial feature points where the difference between the calculation results of the facial feature point difference calculation means 304 is a preset threshold or greater.

Although the present invention has been described above with reference to exemplary embodiments and examples, the present invention is not limited to the specific illustrative exemplary embodiments and examples. Various modifications understandable to those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority from Japanese Patent Application No. 2010-121017 filed on May 26, 2010, the entire contents of which are hereby incorporated by reference.

Industrial applicability

The invention is widely applicable to face direction estimation, face authentication system, facial expression recognition and other accuracy improvements.

List of reference signs

1,300 facial feature point position correction equipment

100 data processing equipment

110 facial image input device

120, 301 Facial feature point reliability generating device

130, 302 Initial facial feature point position calculation device

140, 303 Out-of-position facial feature point judgment device

150, 304 Facial feature point difference calculation device

160, 305 facial feature point position correction device

200 storage devices

210 face shape model storage device

Claims (7)

1. A facial feature point position correction device, comprising: facial feature point reliability generating means for each facial feature point from an input image in which a face is photographed, generating a reliability map indicating suitability of feature points of said facial feature point, said face feature point The facial feature points indicate the feature points of the organs on the face; The initial facial feature point position calculation means, based on the reliability map generated by the facial feature point reliability generation means, calculates the position of the facial feature point in the captured image of the face; Out-of-position facial feature point judging means, judging whether each of the facial feature points is an out-of-position facial feature point, as a position that does not satisfy the facial feature point calculated based on the initial facial feature point position calculation means and facial feature points based on prescribed conditions specified by positions of corresponding points corresponding to said facial feature points in the statistical face shape model; The facial feature point difference calculation means calculates each facial feature point except those facial feature points judged as the off-position facial feature points by the out-of-position facial feature point judging means according to a prescribed evaluation function The difference between the position of and the position of the corresponding point corresponding to the facial feature point; and The facial feature point position correcting means corrects the determined facial feature point positions based on the judgment result of the out-of-position facial feature point judging means and the calculation result of the facial feature point difference calculation means. 2. The face feature point position correction device according to claim 1, Wherein, the out-of-position facial feature point determining means uses a robust estimation technique to determine whether each facial feature point is the out-of-position facial feature point. 3. facial feature point position correcting device according to claim 2, Wherein, the out-of-position facial feature point judging device repeats the following operations: A selection operation of randomly selecting some facial feature points from the facial feature points; A transformation operation for transforming coordinate values representing positions of facial features not selected by the selection operation based on vectors for transforming coordinate values representing positions of selected facial feature points into pairs Coordinate values represented by positions of corresponding points of facial feature points; and a calculation operation of calculating the median of the squared difference between the transformed coordinate value and the coordinate value representing the position of the corresponding point of the facial feature point, and If the error between the coordinate value representing the position of the corresponding point of the facial feature point and the coordinate value of the facial feature point transformed based on a specific vector is larger than a predetermined threshold value, it is judged that the facial feature point is the The out-of-position facial feature point, the specific vector is a vector that minimizes the median value among the vectors used for coordinate value transformation in the repeated operation. 4. The face feature point position correction device according to any one of claims 1 to 3, comprising: The facial shape model storage device stores information related to the statistical facial shape model. 5. The face feature point position correction device according to any one of claims 1 to 4, Wherein, the facial feature point position correction device determines the position of the facial feature point judged as the out-of-position facial feature point by the out-of-position facial feature point judging device and the facial feature point difference calculation device Correction is performed at the positions of facial feature points whose calculation result difference is equal to or greater than a preset threshold. 6. A method for correcting facial feature point positions, comprising: For each facial landmark from an input image of which the face is captured, a reliability map is generated indicating the suitability of the facial landmarks for the facial landmarks indicating the characteristics of the organs on the face point; calculating the positions of the facial feature points in the captured image of the face based on the generated reliability map; Judging whether each facial feature point is an out-of-position facial feature point, as not satisfying the position of the facial feature point based on calculation and the position corresponding to the facial feature point based on the statistical facial shape model The facial feature points of the specified condition specified by the position of the corresponding point; According to a prescribed evaluation function, the position of each facial feature point other than those judged as the out-of-position facial feature point and the position of the corresponding point corresponding to the facial feature point are calculated the difference between; and Based on the judgment result of whether each facial feature point is the out-of-position facial feature point and the position of each facial feature point and the position of the corresponding point corresponding to the facial feature point The calculation results of the difference are used to correct the determined facial feature point positions. 7. A facial feature point position correction program that causes the computer to perform the following operations: A face feature point reliability generation process for generating, for each face feature point from an input image in which a face is photographed, a reliability map indicating suitability of feature points of the face feature point, the face feature point The facial feature points indicate the feature points of the organs on the face; The initial facial feature point position calculation process, based on the reliability map generated by the facial feature point reliability generation process, calculates the position of the facial feature point in the image where the face is captured; An out-of-position facial feature point judging process, judging whether each of the facial feature points is an out-of-position facial feature point, as a position that does not satisfy the facial feature points calculated based on the initial facial feature point position calculation process and facial feature points based on prescribed conditions specified by positions of corresponding points corresponding to said facial feature points in the statistical face shape model; The facial feature point difference calculation process, according to the prescribed evaluation function, calculates the position of each facial feature point and the position of each facial feature point judged as an out-of-position facial feature point by the out-of-position facial feature point judging process. The difference between the positions of the corresponding points to which the points correspond; and The facial feature point position correction process corrects the determined facial feature point positions based on the judgment result of the out-of-position facial feature point judgment process and the calculation result of the facial feature point difference calculation process.

Images