JP7044504B2 - Image processing device, image processing method and image processing program - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing program, and more particularly to a technique that can be used to estimate a line of sight, which is a feature of an eye, based on an image obtained by capturing a human face or the like.

Conventionally, there has been known an image processing device that processes image data obtained by photographing a person's face with a camera to specify eye features (for example, Patent Document 1). The technique of Patent Document 1 aims to accurately detect the region of the pupil.

Further, in the image processing apparatus shown in Patent Document 1, the second differential unit 41 for obtaining the brightness gradient by differentiating at least the eye region of the face image in the vertical direction in the vertical direction and the eye region have brightness. It is a curve expressed by the binarization unit 42 that binarizes the gradient and extracts the edge points, the inner and outer corner points of the eyes as both end points, and the both end points and the control points, and is compatible with the above edge points. It is provided with an eyelid contour specifying portion 44 for specifying the curve to be formed as a curve representing the contour of the upper eyelid or the lower eyelid.

Japanese Unexamined Patent Publication No. 2012-190351

In a device that estimates the person's line of sight by image processing from a moving image of a person's face, it is desirable to be able to detect the center coordinates of the black eye, that is, the iris area or the pupil area from the image with high accuracy.

For example, in the black eye detection method in image processing shown in Patent Document 1, an edge in an image is used as a detection feature. Specifically, a differential filter typified by a Sobel filter is used, and the absolute value of the result of this filter processing is used as a feature quantity.

However, with such a method, edges are uniformly generated even for objects other than the detection target, such as the eyelids, eyebrows, and reflection of light reflected on the eyeballs, which causes a decrease in detection accuracy and false detection. In some cases. Then, if the detection position of the black eye shifts, an error occurs when estimating the line of sight.

Further, since the Hough transform is used in the technique of Patent Document 1, an increase in the processing amount is unavoidable when improving the accuracy, and the hardware of the computer is required to have a high processing capacity. Further, when the image is binarized or binarized, the threshold value becomes a problem and is easily affected by the ambient light at the time of shooting. Further, when the ellipse detection algorithm is used, for example, when the eyes are turned sideways and the shape of the black eyes on the two-dimensional image is greatly crushed, the detection itself becomes difficult.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing device, an image processing method, and an image processing program capable of detecting a pupil or an iris with high accuracy. ..

In order to achieve the above-mentioned object, the image processing apparatus according to the present invention is characterized by the following (1) to ( 5 ).
(1) An image processing device including a data processing unit that processes data on a face image taken so as to include a face.
The data processing unit
A filter process for detecting an edge in one scanning direction is applied to the face image to generate an edge image in which the magnitude of the gradient and the positive / negative are retained as information.
In a state where a sampling curve virtually located on the surface of the three-dimensional model is projected on the edge image, the magnitude of the gradient is large for each position in the edge image corresponding to each of the plurality of points constituting the sampling curve. Extract the positive and negative as the sampling value,
In the first point group located on the front side in the scanning direction among the plurality of points constituting the sampling curve, those having a positive gradient are targeted for likelihood evaluation, and the plurality of points constituting the sampling curve. In the second point group located on the rear side of the scanning direction among the points, the likelihood with respect to the sampling curve is calculated by using the point having a negative gradient as the target of the likelihood evaluation.
Of the plurality of sampling curves, the sampling curve having the maximum likelihood is detected as a pupil or an iris.
( 2 )
The sampling curve consists of a semicircular arc curve that bulges downward.
The image processing apparatus according to (1) above.
( 3 ) The image processing apparatus according to (1) or (2) above, wherein the scanning direction in which the data processing unit detects an edge by the filter processing is the direction in which the eyes are lined up in the face image.
( 4 ) The image processing apparatus according to ( 3 ) above, wherein the sampling curve has an arc shape, and the bottom of the most bulging arc is projected onto the edge image in a state of facing downward in the three-dimensional model.
( 5 ) The data processing unit adds the magnitude of the gradient at the point where the gradient is positive in the first point group, and the point where the gradient is negative in the second point group. The image processing apparatus according to (1) above, which calculates the likelihood with respect to the sampling curve by adding the magnitude of the gradient.

According to the image processing apparatus having the above configurations (1) and (2) , even in an environment with a lot of disturbance, it is possible to avoid the adverse effects of unnecessary features that cause noise with a small amount of calculation, and to increase the pupil or iris. It becomes possible to detect with accuracy. When edge information is used in image recognition, it is common to remove the sign using an absolute value. However, if the detection target is only the pupil or iris, the magnitude of the gradient at each edge in the image is to separate the peripheral noise (light reflection, etc.) from the detection target pupil or iris. Not only that, the detection accuracy can be improved by using the difference between positive and negative. For example, when scanning the image area of the eyeball sequentially from left to right, the right contour position of the iris at the boundary between the white eye and the iris, or the right contour position of the pupil at the boundary between the iris and the pupil is the above-mentioned first. In the point group of 1, there are many points where the gradient is positive, and in the left contour position of the iris or the left contour position of the pupil, there are many points where the gradient is negative in the second point group. Therefore, the likelihood is maximized when the position of the sampling curve coincides with the contour position of the pupil or the iris. In this way, it becomes possible to detect the pupil or the iris with high accuracy.
According to the image processing apparatus having the above configuration ( 3 ), the information necessary for recognizing the position of the iris with high accuracy can be efficiently acquired. Assuming that the scanning direction for detecting an edge is the vertical direction of the eye, the boundary adjacent to the contour of the iris may be the upper eyelid or the lower eyelid. In addition, since the eyelids are easily changed by the influence of ambient light and makeup, it becomes difficult to detect the boundary with the iris on the edge image. However, by setting the scanning direction for detecting the edge to the direction in which the eyes are lined up in the face image, for example, the horizontal direction, the influence of the upper eyelid and the lower eyelid is almost eliminated. Therefore, the detection accuracy of the iris is improved.
According to the image processing apparatus having the above configuration ( 4 ), the information necessary for recognizing the position of the iris with high accuracy can be efficiently acquired. In a typical facial image, the contour of the upper edge of the iris is often the area of the eye where the upper edge of the iris is adjacent to the upper eyelid or the upper eyelid covers part of the iris. Hard to detect. On the other hand, on the left and right sides and the lower end of the iris, the boundary with the white of the eye often appears clearly, so that the outline of the iris can be easily detected. Therefore, by adopting an arc shape in which the bottom of the most bulging arc faces downward, it becomes easy to detect the contour portion of the iris.
According to the image processing apparatus having the above configuration ( 5 ), the feature of the edge of the boundary where the iris is switched to the white eye or the boundary where the pupil is switched to the iris in the scanning direction is detected as the magnitude of the gradient at the point where the gradient is positive. Then, the characteristics of the edge of the boundary where the white of the eye switches to the iris or the boundary where the iris switches to the pupil in the scanning direction can be detected as the magnitude of the gradient at the point where the gradient is negative, and the likelihood can be calculated based on these characteristics. ..

In order to achieve the above-mentioned object, the image processing method according to the present invention is characterized by the following ( 6 ).
( 6 ) An image processing method including a data processing unit that performs data processing on a face image taken so as to include a face.
A step of generating an edge image in which the magnitude of the gradient and positive / negative are retained as information by a filter that detects an edge in one scanning direction with respect to the face image.
In a state where a sampling curve virtually located on the surface of the three-dimensional model is projected on the edge image, the magnitude of the gradient is large for each position in the edge image corresponding to each of the plurality of points constituting the sampling curve. Steps to extract positive and negative as sampling values,
In the first point group located on the front side in the scanning direction among the plurality of points constituting the sampling curve, those having a positive gradient are targeted for likelihood evaluation, and the plurality of points constituting the sampling curve. In the second point group located on the rear side of the scanning direction among the points, the step of calculating the likelihood with respect to the sampling curve with the negative gradient as the target of the likelihood evaluation.
A step of detecting the sampling curve having the maximum likelihood among the plurality of sampling curves as a pupil or an iris, and a step of detecting the sampling curve.
Have.

According to the image processing method having the configuration of ( 6 ) above, as in the case of the image processing apparatus of (1) above, even in an environment with a lot of disturbance, an unnecessary feature of noise is generated with a small amount of calculation. It is possible to avoid adverse effects and detect the pupil or iris with high accuracy.

In order to achieve the above-mentioned object, the image program according to the present invention is characterized by the following ( 7 ).
( 7 ) An image processing program for causing a computer to execute each step of the image processing method described in ( 6 ) above.

According to the image processing program having the configuration of ( 7 ) above, by executing this image processing program on a predetermined computer, the environment is subject to a lot of disturbance, as in the case of the image processing apparatus of (1) above. However, with a small amount of calculation, it is possible to avoid the adverse effects of unnecessary features that become noise and detect the pupil or iris with high accuracy.

According to the image processing apparatus, the image processing method and the image processing program of the present invention, it becomes possible to detect the pupil or the iris with high accuracy. The present invention is useful because it is possible to avoid the adverse effects of unnecessary features that cause noise with a small amount of calculation even in an environment with many disturbances that cause detection errors.

The present invention has been briefly described above. Further, the details of the present invention will be further clarified by reading through the embodiments described below (hereinafter referred to as "embodiments") with reference to the accompanying drawings. ..

FIG. 1 is a perspective view showing a specific example of an arrangement state of an image processing device mounted on a vehicle. FIG. 2 is a flowchart showing an outline of a line-of-sight detection algorithm implemented by the image processing apparatus according to the embodiment of the present invention. FIG. 3 is a schematic diagram showing the relationship between a three-dimensional model of an eyeball and a two-dimensional image. FIG. 4 is a specific example of the process flow and data for searching an iris from an image in a conventional line-of-sight detection algorithm. FIG. 5 is a flowchart showing the details of step S18 shown in FIG. FIG. 6 is a schematic diagram showing a specific example of processing data when the processing shown in FIG. 5 is executed. FIG. 7 is a diagram illustrating a process in which the detection position shift of the iris is generated by the conventional line-of-sight detection algorithm. FIG. 8A is a diagram showing an example of the scanning direction of the sobel filter for detecting the edge of the image and the state of each data before and after the processing, and FIG. 8B is the data of the edge image after the processing. It is a figure which shows the example of. FIG. 9 is a flowchart showing detailed contents regarding step S18d shown in FIG. FIG. 10 is a schematic diagram showing a specific example of the pattern shape of the particle filter used for detecting the iris. 11 (a) and 11 (b) are schematic views showing specific examples of processing data when the processing shown in FIG. 9 is executed. FIG. 12 is a schematic diagram showing the pattern shape of the particle filter in the second embodiment. 13 (a) and 13 (b) are schematic views showing specific examples of processing data when the pattern shape of the particle filter of FIG. 12 is applied. FIG. 14 is a graph showing an example of the relationship between the presence or absence of application of the present invention and the detection error. FIG. 15 is a schematic diagram showing the pattern shape of the particle filter in the third embodiment. 16 (a) and 16 (b) are schematic views showing specific examples of processing data when the pattern shape of the particle filter of FIG. 15 is applied.

Specific embodiments of the image processing apparatus, image processing method, and image processing program of the present invention will be described below with reference to each figure.

<Specific example of environment to which the present invention is applied>
FIG. 1 shows a specific example of the arrangement state of the image processing device mounted on the vehicle.
In the example shown in FIG. 1, the camera 12 and the processing unit 13 are installed in the vehicle interior 10 of the vehicle. Specifically, in the meter unit in front of the driver's seat or at the column cover so that the person 11 sitting in the driver's seat, that is, the face including the driver's eye area 11a can be photographed by the camera 12. A camera 12 is installed.

The processing unit 13 processes the two-dimensional image data obtained by photographing the face of the person 11 by the camera 12, and estimates the information indicating the direction of the line of sight 11b from the data in the eye region 11a. The information of the line of sight 11b estimated by the processing unit 13 can be used, as a typical example, for the in-vehicle device to recognize whether or not the driver who drives the vehicle is moving the line of sight for safety confirmation. Of course, such line-of-sight information can be used for various purposes. Therefore, it is assumed that the image processing apparatus of the present invention is configured as, for example, the processing unit 13 in FIG. 1 or a part thereof.

<Overview of line-of-sight detection algorithm>
The outline of the line-of-sight detection algorithm implemented by the image processing apparatus according to the embodiment of the present invention will be described. FIG. 2 is a flowchart showing an outline of a line-of-sight detection algorithm implemented by the image processing apparatus according to the embodiment of the present invention. When the computer of the image processing apparatus executes a predetermined program, the operations according to the line-of-sight detection algorithm shown in FIG. 2 are sequentially performed.

The camera 12 constantly captures an image of a region including the face of the person 11 at a fixed cycle, and outputs an image signal. The computer of the image processing device captures one frame of the video signal from the camera 12 in step S12.

Then, in the next step S13, conversion of the data format of the image including grayscale conversion, that is, "preparation" is performed. For example, for each pixel position in one frame, 8-bit data representing the brightness in gradations in the range of "0 to 255" are arranged vertically and horizontally according to the scanning direction in the frame at the time of shooting. Generate image data of a dimensional (2D) array.

In the next step S14, face detection is performed using, for example, the "Viola-Jones method", and a region including a face is extracted as a rectangular region from the two-dimensional image data of one frame. That is, the area of the face is extracted using the detector created by learning using "Boosting", which is characterized by the difference in the shading of the face. The technique of "Viola-Jones method" is shown in the following literature, for example.
"P. viola and MJ Jones," Rapid Object Detection Using a Boosted Cascade of Simple Features, "IEEE CVPR (2001)."

In the next step S15, the eye region is detected from the data of the rectangular region of the face extracted in S14 by using, for example, the "Viola-Jones method".

Further, in the present embodiment, since it is assumed that the line of sight is detected by a model-based method using an eyeball 3D model described later, it is necessary to specify the center of the eyeball. The center of the eyeball is estimated in step S16. Here, it is assumed that the center coordinates of the rectangular region of the eye detected in step S15 are the center of the eyeball. It is also conceivable to use, for example, a method of determining the center of the eyeball based on the positions of the outer and inner corners of the eye, or a method of estimating the skeleton from the feature points of the face and simultaneously calculating the center of the eyeball.

In step S17, a template matching method is applied to the data in the rectangular region of the eye detected in S15 to perform a rough search for the pupil or iris. Specifically, the coordinates of the image center (center of the black circle) of the black circle image with the highest likelihood are matched by matching the black circle image as a template to the binarized image of the eye image cut out around the eye. The center position of the pupil or iris in the image, and the radius of the black circle image with the highest likelihood is the radius of the pupil or iris in the eye image. The process of step S17 is performed in order to obtain an approximate target for the center position and radius of the pupil or iris in the eye image.

In step S18, the center position and radius of the pupil or iris searched in S17 are used, and the technique of the particle filter is applied to detect the center position and radius of the pupil or iris with higher accuracy. Since step S18 includes a characteristic process of the present invention, the contents thereof will be described in detail later.

By the processing of steps S13 to S18, data of the coordinates of the center of the eyeball and the coordinates of the center position of the pupil or the iris can be obtained for the image of one frame, and the numerical data is output in step S19. The direction of the line of sight can be specified by the coordinates of the center of the eyeball and the coordinates of the center position of the pupil or iris. Further, by repeating the loop-shaped processing of steps S11 to S20 until the image output by the camera 12 is interrupted, the line-of-sight detection in real time is realized.

<Explanation of 3D eyeball model>
The relationship between the 3D (three-dimensional) model of the eyeball and the 2D (two-dimensional) image is shown in FIG.
In the eyeball 3D model 30 shown in FIG. 3, this eyeball is a sphere represented by an eyeball center 31 and a radius R. Further, there is a circular iris 32 on the surface of the eyeball, and a circular pupil 33 in the center of the iris 32.

Therefore, the direction of the line of sight can be specified as the direction from the center of the eyeball 31 toward the center of the iris 32 or the pupil 33, and the rotation angle yaw with respect to the reference direction in the horizontal plane and the rotation angle pitch with respect to the reference direction in the vertical direction (pitch). ) Can be expressed. Further, the center coordinates of the iris 32 or the pupil 33 can be represented by the eyeball radius R, yaw, and pitch when the eyeball center 31 is used as a reference.

On the other hand, the image captured by the camera 12 represents a specific two-dimensional plane. Therefore, when the two-dimensional image 34 obtained by the image taken by the camera 12 is applied to the eyeball 3D model 30, it is necessary to perform two-dimensional / three-dimensional mutual conversion. Therefore, for example, the conversion is performed using the following formula.

X = -R x cos (pitch) x sin (yaw) ... (1)
Y = R × sin (pitch) ・ ・ ・ (2)
X: Distance in the x direction from the center of the eyeball 31 on the two-dimensional image plane Y: Distance in the y direction from the center of the eyeball 31 on the two-dimensional image plane

<Specific example of the process of searching for an iris from an image>
Hereinafter, first, a specific example of the process of searching for an iris from an image will be described.

<Explanation of processing of the conventional line-of-sight detection algorithm>
<Processing flow>
Prior to explaining the details of the line-of-sight detection algorithm implemented by the image processing apparatus according to the present invention, the details of the conventional line-of-sight detection algorithm will be described first. FIG. 4 is a specific example of the flow of processing and data for searching an iris from an image in a conventional line-of-sight detection algorithm.
In step S41 of FIG. 4, the result of step S15 in FIG. 2 is used to cut out a rectangular area of the eye and its periphery from the two-dimensional image data of the entire frame, and acquire the data D41.

In step S42 of FIG. 4, after generating data D42 obtained by binarizing the data D41 acquired in S41 so that the gradation of each pixel is only a binary value of black / white, the figure is shown with respect to this data D42. The template matching in step S17 of 2 is performed. That is, an image D4t having a black circle shape similar to the shape of the iris is used as a template, and while scanning this template on the image of the data D42, an approximate position P41 of the iris having similar features is searched for, and the position is searched for. And identify the radius or diameter of the iris.

Further, in step S43 of FIG. 4, the sobel filter process is applied to the eye data D41 acquired in S41. Specifically, the eye data D41 is scanned sequentially in the horizontal direction from left to right, black (gradation value: 0) is output in the portion where the luminance does not change, and the larger the gradient of the luminance change, the whiter. The edge is detected by approaching (gradient value: 255). As a result, the data D43 of the eye image from which the edges are extracted can be obtained.

In step S44 of FIG. 4, a particle filter process is executed on the data D43 of the eye image obtained as a result of S43, starting from the approximate position of the iris obtained as a result of S42.

<Processing of iris detection by particle filter>
The details of step S44 shown in FIG. 4 are shown in FIG. That is, the process for detecting the iris position using the particle filter is shown in FIG. Further, FIG. 6 shows a specific example of the processing data when the processing shown in FIG. 5 is executed.

By executing the rough search of step S42 on the binarized data D91 in FIG. 6, the region 91 of the circular iris is detected. In step S18a of FIG. 5, the center position of the iris obtained by the rough search in step S17 or S42 is determined by the coordinate system (R, Pp, Py) of the eyeball 3D model 30 based on the equations (1) and (2). ), And the angle of rotation (Pp, Py) representing the center position of the iris is obtained.
R: Eye radius Pp: Variable representing the angle of rotation punch Py: Variable representing the angle of rotation yaw

Therefore, the position of the circular iris region 92 obtained by the rough search can be reflected in the eyeball surface position on the eyeball 3D model 30A as shown in FIG.

In step S18b of FIG. 5, a plurality of particles necessary for finding a candidate for an actual iris position are scattered in the vicinity of the center position (Pp, Py) of the iris obtained by the rough search. Specifically, for example, 100 sets of sampling circles 93 are created using random numbers. These sampling circles 93 can be reflected in the eyeball surface position on the eyeball 3D model 30B as shown in FIG. Each sampling circle 93 is a circular region centered on a position obtained by adding different random number components (Ppd, Pyd) to the center position (Pp, Py) of the iris.
Ppd: Random number component added to the rotation angle punch Pyd: Random number component added to the rotation angle yaw

When creating 100 sets of sampling circles 93, each random number component (Ppd, Pyd) is generated so as to add random noise according to a normal distribution. Therefore, as in the eyeball 3D model 30B shown in FIG. 6, sampling circles 93 are randomly formed at various positions in the vicinity of the iris region 92.

In step S18c of FIG. 5, each of 100 sets of sampling circles 93 is projected onto the plane of the edge image D92 (corresponding to the data D43 in FIG. 4) shown in FIG. The shape of the iris region 92 and each sampling circle 93 is circular, but unless the position of each circle faces the front of the camera 12, each circle on the eyeball 3D model 30 is a plane of a two-dimensional image. When projected onto, it becomes an elliptical shape on a two-dimensional plane because it is projected in an inclined state. Therefore, for convenience, the sampling circle 93 projected on the two-dimensional plane is referred to as a sampling ellipse 94. The sampling ellipse 94 is composed of a plurality of points evenly assigned to each position on the arc.

In step S18d of FIG. 5, the “sampling process” described below is performed. That is, for each sampling ellipse 94, the edge intensities (gradient magnitude) of the edge images of each point (the total number is 120 points in this example) constituting the ellipse are added, and the result is the likelihood of one sampling ellipse 94. And.

The processing of steps S18c and S18d is repeated for all the sampling ellipses 94 of 100 sets, and when all the processing is completed, the process proceeds from steps S18e to S18f. In step S18f, one sampling ellipse 94 having the highest likelihood detected in step S18d is specified from among 100 sets of sampling ellipses 94, and its position and shape are detected as the actual position and shape of the iris. The above is the process of iris detection by the particle filter in the conventional line-of-sight detection algorithm.

<Explanation of detection position shift by conventional line-of-sight detection algorithm>
FIG. 7 is a diagram illustrating a process in which the detection position shift of the iris is generated by the conventional line-of-sight detection algorithm. Assuming that an image processing device that performs the above-mentioned processing is used, for example, in an outdoor environment, the detection accuracy is lowered due to the influence of ambient light.

For example, when the binarization search S51 (corresponding to S17 and S42) is executed for the binarized data D52 generated based on the data D51 of the eye image shown in FIG. 7, the position P51 of the iris is incorrect. It may be detected at the position. As a result, when the above-mentioned particle filter processing S52 is applied to the edge image D53, the scattering base points of the particles are shifted, and it becomes easy to pick up edge information other than the iris at the time of likelihood evaluation, and finally. The detection accuracy will be lowered.

This is because the upper eyelid is illuminated by sunlight, resulting in a black portion on the upper eyelid in the binarized image. When binarization template matching is performed on this image, it is determined that the iris in the eye image is located at the upper eyelid. Then, since each sampling circle is created based on that position, 100 sets of sampling circles are formed at positions closer to the upper eyelid side as a whole. Then, a large displacement occurs in the result of the processing of the particle filter.

In the image data D54 shown in FIG. 7, the actual position of the iris is P53, but the position P52 greatly deviated from this position is detected as a result of the particle filter processing S52. In this way, when the iris is detected by the conventional line-of-sight detection algorithm, the iris detection position may shift as described above.

<Processing of the first embodiment>
In order to suppress the detection position shift of the iris as shown in FIG. 7, the image processing apparatus of the present embodiment includes the following <configuration A> and <configuration B>.

<Structure A> A filter process for detecting an edge in a certain scanning direction is applied to a face image to generate an edge image in which the magnitude of the gradient and the positive and negative signs are retained as information. For example, when the edge is extracted in step S43 shown in FIG. 4 and the data D43 is generated, a special edge image in which the magnitude of the gradient and the positive / negative sign are retained as information is generated.

<Structure B> In the first point group located on the front side in the scanning direction among the plurality of points constituting the sampling curve, the one having a positive gradient is targeted for the likelihood evaluation, and the sampling curve is used. In the second point group located on the rear side in the scanning direction among the plurality of constituent points, the one having a negative gradient is set as the target of the likelihood evaluation, and the likelihood with respect to the sampling curve is calculated.

<Specific example of "Structure A">
FIG. 8A shows an example of the scanning direction of the sobel filter for detecting the edge of the image and the state of each data before and after the processing, and FIG. 8B shows an example of the data of the edge image after the processing. In the example shown in FIG. 8, the sobel is directed from the left to the right of the image in the horizontal direction or in the direction in which the eyes are lined up for the purpose of accurately detecting the characteristics of the left-right boundary of the eye region. It is assumed that the position of the filter is scanned on the image.

In the data D71 before processing shown in FIG. 8, the luminance between the horizontal boundary D71a and the boundary D71b is at the black level like an iris, and the luminance in the other regions is at the white level like white eyes. I assume a certain case.

Therefore, when the unprocessed data D71 shown in FIG. 8 is processed by the sobel filter, the edge of the luminance change is detected at the positions of the boundaries D71a and D71b, and the processed data D72 including this edge information is obtained. However, unlike the case where a normal Sobel filter is applied, the data D72 after this processing holds information on the positive and negative signs of the gradient in addition to the magnitude of the gradient (edge strength).

That is, in the processed data D72, the black value in the range of "-255 to -1" is retained at the edge D72a, and the white value within the range of "1 to 255" is held at the edge D72b. It is retained, and the value of "0" representing gray is retained in other areas.

That is, the boundary between the start position of the iris whose brightness (gradation) changes from white to black and the boundary of the end position of the iris whose brightness changes from black to white, such as the boundaries D71a and D71b of the data D71 before processing. The positive and negative codes are held in the processed data D72 in a form that can be distinguished from each other. In this way, the data of the edge image D73 shown in FIG. 8B can be obtained. The data of the edge image D73 is suitable for detecting the feature of the boundary between the iris and the white of the eye in the scanning direction (horizontal direction in the case of FIG. 8).

<Specific example of "configuration B">
FIG. 9 shows a detailed procedure including characteristic contents for the “sampling process” of step S18d shown in FIG. Further, FIG. 10 shows a specific example of the pattern shape of the particle filter used for detecting the iris. Further, FIGS. 11A and 11B show schematic diagrams showing specific examples of processing data when the processing shown in FIG. 9 is executed.

The pattern shape shown in FIG. 10 corresponds to the sampling circle 93A and the sampling ellipse 94A shown in FIGS. 11A and 11B, and the position of each sampling ellipse 94A in the “sampling process” shown in FIG. It is used to calculate the appropriate likelihood in. Note that FIG. 11A is the same as FIG. 6 except that the edge image D92 is the edge image D73, and therefore detailed description thereof will be omitted here. Further, since FIG. 11B is the same as FIG. 7 except that the edge image D53 is the edge image D73, detailed description thereof will be omitted here.

The filter pattern FP0 shown in FIG. 10 is formed in a circular (or elliptical) ring as a whole, and includes a large number of sampling points 101 arranged side by side at equal intervals on the circumference thereof. The total number of sampling points 101 on the filter pattern FP0 is, for example, 120 points.

Further, the filter pattern FP0 is divided into left and right at the center of the circle, and has a pattern left region FPL corresponding to the left semicircle and a pattern right region FPR corresponding to the right semicircle. The sampling points 101 in the left side region FPL of the pattern are designated as point PGLs in the left side group, and the sampling points 101 in the right side region FPR of the pattern are classified as point PGRs in the right side group.

In the "sampling process" shown in FIG. 9, in steps S101 to S108, the processes described below are sequentially executed for each of the "i" sampling points 101 constituting one selected sampling ellipse 94A. do.

In step S102, it is determined whether or not the corresponding sampling point 101 is on the left side of the center coordinate of the ellipse, that is, whether or not the sampling point 101 is the point PGL of the left side group. If the sampling point 101 is the point PGL of the left group, the process proceeds to step S103, and if the sampling point 101 is the point PGR of the right group, the process proceeds to step S105.

In step S103, the value of one pixel on the edge image D73 shown in FIG. 11B is acquired (sampled) at the position of one sampling point 101 currently selected, and the sign of the sampled value is positive or negative. Identify. Then, if the sign of the sampling value is "negative", the process proceeds from step S103 to S104, and if it is "positive", the process proceeds to S107.

In step S104, the result of multiplying the current sampling value obtained in S103 by "-1" is added to the value of the variable X. That is, in this case, since the sampling point 101 is the point PGL in the left group and belongs to the second point group, the sampling value of the pixel having a negative gradient is the target of the likelihood evaluation. Further, in order to align the sign of the sampling value at the time of calculation to "positive", multiply by "-1" in S104.

In step S105, the value of one pixel on the edge image D73 shown in FIG. 11B is acquired (sampled) at the position of one sampling point 101 currently selected, and the sign of the sampled value is positive or negative. Identify. Then, if the sign of the sampling value is "positive", the process proceeds from step S105 to S106, and if it is "negative", the process proceeds to S107.

In step S106, the current sampling value obtained in S105 is added to the value of the variable X as it is. That is, in this case, since the sampling point 101 is the point PGR of the right side group and belongs to the first point cloud, the sampling value of the pixel whose gradient is positive is the target of the likelihood evaluation.

In step S107, the value of the variable (i) representing the selected position of the sampling point 101 to be processed this time on the circumference of the ellipse is updated to the next position.

When the processing for all the sampling points 101 is completed in steps S101 to S108 shown in FIG. 9, the variable X holds the sum of the likelihood evaluation values obtained at each sampling point 101. That is, the value of the variable X represents the likelihood of one currently selected sampling ellipse 94A (see FIG. 11B).

By executing the "sampling process" shown in FIG. 9 on the data of the edge image D73 generated in "configuration A", the characteristics of the points where the edge intensities are positively or negatively inverted at the boundaries on both sides of the iris are accurately detected. be able to. That is, the sampling ellipse in which the edge strength on the right side of the center of the ellipse is positive and the edge strength on the left side is negative is calculated with higher likelihood.

In this way, the processes from step S101 to step S108 are performed on all the sampling circles 93A, and the sampling circle 93A having the maximum likelihood is detected as an iris. As described above, according to the image processing apparatus of the embodiment of the present invention, there are many points where the edge strength is positive at the right contour position of the iris at the boundary between the white eye and the iris, and the edge strength is high at the left contour position of the iris. Since there are many negative points, the likelihood of the sampling circle 93A scattered near the contour position of the iris is remarkably high. As a result, even if it is erroneously determined that the iris in the eye image is located at the upper eyelid by performing binarization template matching in an environment with a lot of disturbance, each sampling circle scattered based on that position. Since the likelihood of is extremely low, sampling circles placed closer to the upper eyelid side are excluded from the iris candidates. In this way, it becomes possible to detect the iris with high accuracy by avoiding the adverse effect of unnecessary features that become noise with a small amount of calculation.

As described above, even if it is erroneously determined that the iris in the eye image is located at the upper eyelid by performing binarization template matching, the sampling circle arranged at the position closer to the upper eyelid is used. The following points should be noted in order to exclude from the candidates for iris. That is, when spraying the sampling circle 93A, the random number components (Ppd, Pyd) for defining the center of each sampling circle 93A are required to have a certain range in the numerical range of the random number that can be taken as the random number component. If this width is small, the sampling circle 93A will be locally scattered, but it is difficult to improve the detection accuracy of the iris even if a large number of sampling circles 93A are locally scattered at the wrong position. Therefore, even if it is erroneously determined that the iris in the eye image is located at the upper eyelid, when the sampling circle 93A is sprayed around the incorrect position, the sampling circle 93A is located at the correct location where the iris is located. It is desirable to adjust the total number of random components and sampling circles to the extent that they are scattered. By appropriately adjusting the total number of random number components and sampling circles, the process of step S17 shown in FIG. 2 becomes indispensable.

Further, in the "sampling process" shown in FIG. 9, it is assumed that the edge image data (D72) generated while scanning from left to right as shown in FIG. 8A is processed. .. When the scanning direction when generating the edge image is different, the position and the positive / negative of the portion where the edge strength is inverted positive and negative change. Therefore, the contents of steps S102 to S106 in FIG. 9 are adjusted according to the difference in the scanning direction. Need to be changed.

The scanning direction of the filter when generating an edge image is not limited to the alignment direction and the horizontal direction of both eyes, but it is also conceivable to change the scanning direction to, for example, an oblique direction. However, it is easier to detect the edge of the boundary between the white eye and the iris when the filtering process for detecting the edge is performed in the horizontal direction or the alignment direction of both eyes, as compared with the case of scanning in the vertical direction, for example. It is suitable because the accuracy is easily improved.

By the way, the detection of the iris up to this point has been carried out for the first image (first frame), but when the same detection is performed for the second frame image, step S17 shown in FIG. You can cancel the process. This is because the position coordinates of the iris detected in the first frame have already been determined, so that the position coordinates of the first frame may be used as a base for calculating the position coordinates of the iris in the second frame. That is, when detecting the iris in the second frame, the sampling circle 93A may be sprayed in the vicinity of the center position (Pp, Py) of the iris obtained in the first frame. Similarly, in the third and subsequent frames as well, the sampling circle 93A may be sprayed in the vicinity of the center position of the iris detected in the previous frame. In this way, it is possible to reduce the data processing associated with the detection of the iris.

<Processing of the second embodiment>
FIG. 12 shows an example of the pattern shape of the particle filter in the second embodiment. Further, FIGS. 13A and 13B show schematic diagrams showing specific examples of processing data when the pattern shape of the particle filter of FIG. 12 is applied.

In the second embodiment, the “sampling process” shown in FIGS. 13 (a) and 13 (b) is carried out using the semicircular filter pattern FP shown in FIG. That is, the same processing as in the first embodiment is performed except that the shape of the filter pattern is different. FIG. 13A is the same as FIG. 6 except that the edge image D92 is the edge image D73, and therefore detailed description thereof will be omitted here. Further, since FIG. 13B is the same as FIG. 7 except that the edge image D53 is the edge image D73, detailed description thereof will be omitted here.

The filter pattern FP shown in FIG. 12 is a pattern of a semicircular arc that swells downward, and is projected onto an edge image D73 on a two-dimensional plane, for example, as shown in FIG. 13 (b).

Similar to the filter pattern FP0 shown in FIG. 10, this filter pattern FP has a large number of sampling points 101 arranged at equal intervals on the circumference. These sampling points 101 are divided into a left region FPb and a right region FPc with the center FPa of the filter pattern FP as a boundary. That is, the sampling point 101 belonging to the left region FPb is treated in the same manner as the point PGL in the left group shown in FIG. 10, and the sampling point 101 belonging to the right region FPc is treated in the same manner as the point PGR in the right group.

When spraying a plurality of particles in the vicinity of the iris position obtained by the rough search, the position where different random number components (Ppd, Pyd) are added to the center position (Pp, Py) of the iris region 92 is a semicircle. Each sampling circle 93B is arranged so as to be the center of the shape. Further, at this time, the bottom of the most bulging arc in the filter pattern shape is arranged so as to face downward in the eyeball 3D model 30.

Since the filter pattern FP having this radial shape is projected as the sampling ellipse 94B on the edge image D73 on the two-dimensional plane as shown in FIG. 13 (b), the contour portion of the lower half region of the iris is this. The characteristics of the shape can be evaluated efficiently.

In general, the boundary between the iris and the white of the eye tends to appear on the right, left, and lower sides of the iris, and the upper side of the iris tends to be the boundary with the upper eyelid. Therefore, by adopting the semicircular filter pattern FP shown in FIG. 12, matching with the sampling curve is performed for each of the right, left, and lower parts of the iris where the boundary of the iris can be detected accurately. Can be tried. Conversely, the upper side of the iris, which reduces the detection accuracy of the iris boundary, can be excluded from the matching target. Therefore, the iris can be detected with high accuracy.

The shape of the filter pattern FP is not limited to a semicircle, and various arcs can be applied. Further, it may have a C-shape in which a part of the annulus is missing. At this time, it is preferable that the bottom of the most bulging arc in the filter pattern shape faces downward in the eyeball 3D model 30.

<Specific example of the process of searching the pupil from the image>
Subsequently, the process of searching the pupil from the image will be described. In the above-mentioned <Specific example of the process of searching for an iris from an image>, in step S42 of FIG. 4, an image D4t having a black circle shape similar to the shape of the iris is used as a template, and this template is scanned on the image of the data D42. At the same time, it was decided to search for an approximate iris position P41 having similar characteristics and to specify the position and the radius or diameter of the iris. In the process of searching the pupil from the image, the pupil is basically searched by the same process as the process described in <Specific example of the process of searching the iris from the image> described above. That is, an image having a black circle shape similar to the shape of the pupil (a black circle similar to the shape of the pupil is smaller than a black circle similar to the shape of the iris) is used as a template, and this template is scanned on the image of data D42. While searching for the approximate location of the pupil with similar features, the location and the radius or diameter of the pupil are determined. In this way, by changing the shape of the black circle drawn on the image used as the template, the target to be searched can be selected from the iris or the pupil.

<Processing of the third embodiment>
FIG. 15 shows an example of the pattern shape of the particle filter in the third embodiment. Further, FIGS. 16A and 16B show schematic diagrams showing specific examples of processing data when the pattern shape of the particle filter of FIG. 15 is applied.

In the third embodiment, the "sampling process" shown in FIGS. 16A and 16B is performed using the semicircular filter pattern FP2 shown in FIG. That is, the same processing as in the first embodiment is performed except that the shape of the filter pattern is different. FIG. 16A is the same as FIG. 6 except that the edge image D92 is the edge image D73, and therefore detailed description thereof will be omitted here. Further, FIG. 16B is the same as FIG. 7 except that the edge image D53 is the edge image D73, and therefore detailed description thereof will be omitted here.

The filter pattern FP2 shown in FIG. 15 is a circular pattern having three circles C1, C2, and C3 having different radii as the center points, and is, for example, two-dimensional as shown in FIG. 16 (b). It is projected onto the edge image D73 on a plane.

This filter pattern FP2 has a large number of sampling points 201, 202, and 203 arranged at equal intervals on the circumference of each circle C1, C2, and C3. Further, the sampling points 201, 202, and 203 for each of the circles C1, C2, and C3 are set with the same number of sampling points, and are arranged radially so as to be arranged in the radial direction.

When spraying a plurality of particles in the vicinity of the iris position obtained by the rough search, there are three positions in which different random number components (Ppd, Pyd) are added to the center position (Pp, Py) of the iris region 92. Each sampling circle 93C is arranged so as to be a center point common to the circles.

The filter pattern FP2 composed of these three circles is projected as a sampling ellipse 94C on the edge image D73 on the two-dimensional plane as shown in FIG. 16B.

When an image in which an iris or a pupil is photographed is generated, the size of the iris or the pupil (the number of pixels in the vertical and horizontal directions) in the image generally changes even if the same iris or the pupil is photographed as an image target. By changing the pupil diameter according to the amount of light, not to mention the change in the size of the iris or pupil in the captured image, but also by moving the head toward or away from the camera. , Even if the iris or pupil diameter does not change, the size of the iris or pupil in the captured image will change. Thus, it is also important to search for the iris or pupil while responding to changes in the iris or pupil diameter.

In that respect, in the filter pattern FP2 of the third embodiment, it is possible to cope with the change in the iris diameter or the pupil diameter by each of the triplely formed circles. That is, even if the iris or pupil taken in the image frame at a certain time point t1 changes the iris diameter or the pupil diameter in the image frame at the next time point t2, the circles C1 and C2 that capture the iris or the pupil at the time point t1. , C1, C2, C3, circles larger or smaller than C3, can capture the iris or pupil at time point t2. As described above, according to the third embodiment of the present invention, the iris or the pupil can be searched accurately while responding to the change in the iris diameter or the pupil diameter.

The third embodiment of the present invention can be applied to the filter pattern described in the first embodiment or the second embodiment, and can also be applied to the filter pattern described with reference to FIG. ..

The image processing apparatus according to the third embodiment of the present invention can be simply expressed as follows.
An image processing device including a data processing unit that processes data on a face image taken so as to include a face.
The data processing unit
A filter process for detecting an edge in a certain scanning direction is applied to the face image to generate an edge image in which the magnitude of the gradient is retained as information.
A sampling curve in which a plurality of circles having different radii, which are virtually located on the surface of a three-dimensional model, are arranged with the same position as a center point, is projected onto the edge image, and a plurality of points constituting the sampling curve. For each position in the edge image corresponding to each, the magnitude of the gradient is extracted as a sampling value and used as a target for likelihood evaluation.
Of the plurality of sampling curves, the sampling curve having the maximum likelihood is detected as a pupil or an iris.
An image processing device characterized by this.

<Application of image processing equipment technology>
It should be noted that the technique of the image processing apparatus of the present invention can be applied to various applications other than detecting the eyes. Specific examples include head movement tracking, eyelid detection, mouth detection, and the like. When considering the actual application, the tracking of the head movement can be dealt with by adjusting the spray shape size of the particle filter to the head size in the image. In addition, the detection of eyelids and mouth can be handled by applying a sobel filter in the vertical direction and setting the code division of the sampling feature in the vertical direction.

It should be noted that each operation shown in FIGS. 2, 4, 5, 9, etc. can be realized by hardware such as a dedicated logic circuit, or can be read and executed by various control computers. It can also be realized as software composed of programs and data.

<Improved accuracy>
FIG. 14 shows an example of the relationship between the presence or absence of application of the present invention and the detection error. That is, FIG. 14 shows the difference in detection error between the method to which the present invention is applied and the conventional method under the condition that the accuracy is lowered due to the influence of ambient light.

In FIG. 14, "after improvement" corresponds to the detection result of the method to which the present invention is applied, and "before improvement" corresponds to the detection result when the present invention is not applied. With reference to FIG. 14, it can be seen that the error average from the true value was 1.6 pixels in the conventional method, but it is improved to 0.5 pixels in the method of the present invention.

Here, the features of the image processing apparatus, the image processing method, and the embodiment of the image processing program according to the present invention described above are briefly summarized and listed below in [1] to [6], respectively.
[1] An image processing apparatus including a data processing unit (processing unit 13) that processes data on a face image taken so as to include a face.
The data processing unit
A filter process for detecting an edge in a certain scanning direction is applied to the face image to generate an edge image (D73) in which the magnitude of the gradient and the positive / negative are retained as information.
A state in which a sampling curve (sampling circles 93A, 93B) including an arc-shaped curve virtually located on the surface of a three-dimensional model (eyeball 3D model 30) is projected onto the edge image (sampling ellipses 94A, 94B). In, the magnitude and positive / negative of the gradient are extracted as sampling values for each position in the edge image corresponding to each of the plurality of points (sampling points 101) constituting the sampling curve (S103, S105).
In the first point group (point PGR in the right group) located on the front side in the scanning direction among the plurality of points constituting the sampling curve, those having a positive gradient are targeted for likelihood evaluation ( S105, S106), in the second point group (point PGL in the left group) located on the rear side in the scanning direction among the plurality of points constituting the sampling curve, the one having a negative gradient is the likelihood. As an object of evaluation, the likelihood with respect to the sampling curve was calculated (S103, S104), and
Of the plurality of sampling curves, the sampling curve having the maximum likelihood is detected as a pupil or an iris (S18f).
Image processing device.

[2] The scanning direction in which the data processing unit detects an edge by the filtering process is the direction in which the eyes are lined up in the face image (see FIG. 8).
The image processing apparatus according to the above [1].

[3] The sampling curve (filter pattern FP) has an arc shape, and is projected onto the edge image with the bottom of the most bulging arc facing downward in the three-dimensional model (see FIG. 12).
The image processing apparatus according to the above [2].

[4] The data processing unit adds the magnitude of the gradient at the point where the gradient is positive in the first point cloud (S105, S106), and the gradient is in the second point cloud. The likelihood with respect to the sampling curve is calculated by adding the magnitudes of the gradients of the negative points (S103, S104).
The image processing apparatus according to the above [1].

[5] An image processing method including a data processing unit that performs data processing on a face image taken so as to include a face.
A step (S43) of generating an edge image (D73) in which the magnitude of the gradient and positive / negative are retained as information by a filter that detects an edge in a certain scanning direction with respect to the face image.
In a state where a sampling curve (sampling circles 93A, 93B) including an arc-shaped curve virtually located on the surface of the three-dimensional model is projected on the edge image (sampling ellipse 94A, 94B), the sampling curve is used. A step (S103, S105) of extracting the magnitude and positive / negative of the gradient as a sampling value for each position in the edge image corresponding to each of the plurality of constituent points.
In the first point group located on the front side in the scanning direction among the plurality of points constituting the sampling curve, those having a positive gradient are targeted for likelihood evaluation (S105), and the sampling curve is used. In the second point group located on the rear side of the scanning direction among the plurality of constituent points, the likelihood of the sampling curve is calculated with the negative gradient as the target of the likelihood evaluation (S103). Steps to be performed (S104, S106) and
A step (S18f) of detecting the sampling curve having the maximum likelihood among the plurality of sampling curves as a pupil or an iris.
Image processing method having.

[6] An image processing program for causing a computer to execute each step of the image processing method according to the above [5].

10 Vehicle interior 11 People 11a Eye area 11b Eyes 12 Camera 13 Processing unit 30 Eyeball 3D model 31 Eyeball center 32 Pupil or iris 33 Pupil 34 Two-dimensional image 91, 92 Area of pupil or iris obtained by rough search 93 Sampling Circle 94 Sampling ellipse 101 Sampling points FP, FP0 Filter pattern FPL pattern Left side area FPR pattern Right side area PGL Left side group point PGR Right side group point

Claims (7)

An image processing device including a data processing unit that processes data on a face image taken so as to include a face.
The data processing unit
A filter process for detecting an edge in one scanning direction is applied to the face image to generate an edge image in which the magnitude of the gradient and the positive / negative are retained as information.
In a state where a sampling curve virtually located on the surface of the three-dimensional model is projected on the edge image, the magnitude of the gradient is large for each position in the edge image corresponding to each of the plurality of points constituting the sampling curve. Extract the positive and negative as the sampling value,
In the first point group located on the front side in the scanning direction among the plurality of points constituting the sampling curve, those having a positive gradient are targeted for likelihood evaluation, and the plurality of points constituting the sampling curve. In the second point group located on the rear side of the scanning direction among the points, the likelihood with respect to the sampling curve is calculated by using the point having a negative gradient as the target of the likelihood evaluation.
Of the plurality of sampling curves, the sampling curve having the maximum likelihood is detected as a pupil or an iris.
Image processing device. The sampling curve consists of a semicircular arc curve that bulges downward.
The image processing apparatus according to claim 1. The scanning direction in which the data processing unit detects an edge by the filtering process is the direction in which the eyes are lined up in the face image.
The image processing apparatus according to claim 1 or 2 . The sampling curve has an arc shape, and the bottom of the most bulging arc is projected onto the edge image with the bottom facing downward in the three-dimensional model.
The image processing apparatus according to claim 3 . The data processing unit adds the magnitude of the gradient at the point where the gradient is positive in the first point cloud, and the gradient of the point where the gradient is negative in the second point cloud. By adding the magnitudes, the likelihood with respect to the sampling curve is calculated.
The image processing apparatus according to claim 1. It is an image processing method including a data processing unit that performs data processing on a face image taken so as to include a face.
A step of generating an edge image in which the magnitude of the gradient and positive / negative are retained as information by a filter that detects an edge in one scanning direction with respect to the face image.
In a state where a sampling curve virtually located on the surface of the three-dimensional model is projected on the edge image, the magnitude of the gradient is large for each position in the edge image corresponding to each of the plurality of points constituting the sampling curve. Steps to extract positive and negative as sampling values,
In the first point group located on the front side in the scanning direction among the plurality of points constituting the sampling curve, those having a positive gradient are targeted for likelihood evaluation, and the plurality of points constituting the sampling curve. In the second point group located on the rear side of the scanning direction among the points, the step of calculating the likelihood with respect to the sampling curve with the negative gradient as the target of the likelihood evaluation.
A step of detecting the sampling curve having the maximum likelihood among the plurality of sampling curves as a pupil or an iris, and a step of detecting the sampling curve.
Image processing method having. An image processing program for causing a computer to execute each step of the image processing method according to claim 6 .

Images