JP2003168106A - Eye position tracking method, eye position tracking device, and program therefor - Google Patents

Abstract

(57) [Problem] To provide an eye position tracking method capable of tracking the position of eyes from a general face image with high robustness and without unduly restricting a user. A method for tracking the position of an eye includes a step of acquiring an image, a step of predicting an inter-brow position of a face image in the image, and a step of: Step 246 of searching for the best matching point; predicting the position of both eyes in the image based on the position of the searched point and the relative position of both eyes; A step 248 of searching for a center point of each of the two regions satisfying the specified condition, a position between eyebrows, a relative position of both eyes,
And a step 254 of updating the image pattern between the eyebrows.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to man-machine interface technology, and more particularly, to the position of human eyes for the purpose of accurately detecting the direction of the line of sight of a person when operating a computer or the like by the line of sight of the person. A method and apparatus for tracking an object.

[0002]

2. Description of the Related Art With the recent development of computer technology,
Computer-based devices are used in every corner of people's lives. Without the technology to operate a computer,
There is a risk that they will not be able to live a satisfactory social life.

On the other hand, in order to operate a computer, it is necessary to convey the intention of a person to the computer. Various studies have been made and put into practical use as to how efficiently, without error, and easily convey human intentions to a computer. Such a technique is generally called a man-machine interface.

If it is only to operate the computer, a text-based command for operating the computer may be given to the computer via, for example, a keyboard. However, that would require a human to remember those numerous commands along with the syntax of the command and the required parameters. Therefore, in general, GUI
What was called (Graphical User Interface) was invented and is now the mainstream.

In the GUI, a pointing device such as a mouse is generally used to point an icon or the like displayed on the screen and perform a predetermined operation such as click, double click, drag, etc. Can be given to. Therefore, there is a feature that it is not necessary to remember many commands and an error is unlikely to occur.

On the other hand, in the GUI, it is necessary to operate a pointing device. Therefore, for example, it is difficult for a person with a hand movement disorder to operate even a computer that employs a GUI. In addition, depending on the application, the hand may not be used freely, and it may be difficult to use a pointing device such as a mouse.

Therefore, various techniques for transmitting the intention of a human to a computer by using the line of sight of the human have been considered. When a human consciously operates the line of sight, the line of sight can be used as a pointing device to give the human the intention of the human to the computer. In addition, when a human unconsciously moves the line of sight, the computer can detect the movement of the line of sight to estimate the intention of the human.

[0008] A technology which is a prerequisite for this is an apparatus for estimating a line-of-sight direction by imaging a human eyeball. For that purpose, an eye camera is generally used. By estimating the line-of-sight direction of a person using an eye camera, the intention of the person can be estimated and can be used for computer operation.

In the case of estimating the line-of-sight direction using an eye camera, it is necessary to image the eyeball with high resolution in order to improve the estimation accuracy. However, because of this, the imaging range of the eye camera is narrowed. The depth of field is also shallow. As a result, when the eye camera is used, the user has to almost fix the face so that the face does not move forward, backward, left or right. Therefore, when the eye camera is used alone, its usage is very limited. Further, it is desirable that the system can be used only for a specific individual and that the eyes can be detected and tracked even with a strange face.

In order to solve the problems associated with the use of such an eye camera, the inventor of the present application first detects the position of the user's eyes using a camera with a wide field of view, and uses the information to detect the eye camera. We have found that controlling pan, tilt, and focus is effective. In other words, by using a subsystem that discovers and tracks the position of the human eye by some method and sequentially outputs the direction and distance to the control device of the eye camera, Is a way to increase.

It is considered that the user is usually sitting in a chair at a distance of about 50 to 100 cm from the eye camera when using the computer. Therefore,
By detecting the eye position as described above and controlling the eye camera by using the information, it is considered that the accuracy of the eye gaze direction detection by the eye camera becomes extremely high.

In the assumed situation, the range of movement of the face is quite limited. Therefore, it is possible to capture an image as large as the face fills the height of the screen. Therefore, the inventor of the present invention has come to the idea that the positions of both eyes can be detected by detecting the movement of the eyelids in the face image captured in such a large size. Humans are blinking unconsciously, and by waiting for a natural blink, the position of their eyes can be detected without making the user aware. Especially in the situation where the eye camera is used, it is possible to obtain the cooperation of the user.

Blinking is not a continuous movement. for that reason,
Once the eye position is detected, it is necessary to somehow track the eye position. One way to do that is to prepare a large number of face image templates,
It is to estimate the eye position by matching the captured face image with the template.

However, in this case, a simple template matching cannot cope with a change in the orientation of the face. Even if the template is updated, it is inevitable that the tracking point gradually shifts from the actual eye. Therefore, it is necessary to overcome the weak points caused by the template matching and improve the robustness of the tracking process.

Regarding the information on the distance given to the eye camera, sufficient accuracy can be obtained by measuring it using an ultrasonic sensor or the like as long as the direction of the eyes is known. However, in the following embodiments, the distance to the user's face is estimated using a twin-lens stereo image. Of course, in addition to this, various methods can be used to measure the distance to the user's face, but since it is not directly related to the detection and tracking of the eye position in the present invention, it will be described below. A detailed description of distance measurement will not be given.

There are already some examples of eye position tracking methods. One of them is JP 2000-16.
3564. In this example, the position of the eyes is tracked using a so-called template matching method with an image pattern centering on the eyes as a template. Another example is the Technical Report PRMU9.
9-151 (1999-11), pp. It was disclosed in a paper entitled "Development of a real-time eye-gaze detection / motion recognition system" in 9-14. In this example, face tracking is performed from a face image obtained by using two video cameras by stereo processing of an image of a facial feature region including both ends of the left and right eyes and an image using three-dimensional coordinates. The center position of the eyeball is estimated by the face position thus estimated and the offset vector from the midpoint of both ends of the left and right eyes to the eyeball.

[0017]

However, Japanese Patent Laid-Open No. 2000-2000
In the method disclosed in 163564, a normal template matching method is used, and there is a problem that it is weak against pattern rotation and scale change. In addition, in the example published in IEICE, the eye position is estimated only from the stereo processing using two images, so the distance from the camera to the eye is limited, and the pattern to be tracked is registered in advance. There is a problem that needs to be kept.

In other words, there has been a problem that a general user's eye position cannot be accurately tracked without constraining the user's face position.

Therefore, an object of the present invention is to enable tracking of the eye position from a general face image with high robustness, and a method and apparatus for tracking the eye position without improperly restraining the user. And to provide a program therefor.

Another object of the present invention is to detect a user's eye position with high accuracy and robustness from a general face image even if the face rotates or moves, which makes the user unfair. (EN) Provided are an eye position tracking method, an apparatus, and a program therefor which are not restricted by the above.

[0021]

The invention according to the first aspect of the present application is a computer to which an image pickup means for picking up a video image of a subject's face is connected. A method for tracking the position of an eye and a computer-executable program therefor, which are set in advance according to a position between the eyebrows of a face image output by the imaging unit, a relative position of both eyes with respect to a position between the eyebrows, and a position between the eyebrows. Given a defined image pattern between the eyebrows, the method comprises a first step of obtaining an image subsequent to the image, a step of predicting a position between the eyebrows of a facial image in the subsequent image, In the vicinity of the position between the eyebrows, the step of searching for a point that is the center of the region that best matches the image pattern between the eyebrows, the position of the searched point, and The position of both eyes in the subsequent image is predicted based on the relative position of each of the two, and the center point of each of the two regions satisfying a predetermined condition is searched with the predicted region as the center. The position of the midpoint between the step and the two center points that have been searched is set as a new position between the eyebrows, and an area defined by the position of the new eyebrows and the relative position of the two center points is set to the position between the eyebrows. The relative position of both eyes, and the step of updating the image pattern of the area defined by the new position between the eyebrows as the new image pattern between the eyebrows, the position between the eyebrows, the relative position between the eyes, and the image pattern between the eyebrows, respectively. Performing the process from the first step on the image subsequent to the image to be processed.

Preferably, the step of predicting the position between the eyebrows is based on the position between the eyebrows in the image output by the image pickup means and the position between the eyebrows in the image preceding the image, Extrapolating the position is included.

More preferably, the method comprises the step of searching for a center point of each of the two regions,
In response to the fact that the point could not be searched, the position between the eyebrows of the face image output by the imaging means, the relative position of both eyes to the position between the eyebrows, and the image pattern between the eyebrows determined by the position between the eyebrows were predetermined. Extracted by the method
The method further includes the step of restarting the process from the first step.

Further, the step of searching for the center point of each of the two regions is predetermined in the vicinity of each of the positions determined by the relative positions of the eyes with respect to the position of the searched point. It may include a step of searching for the center of an area having a different shape and having the darkest average pixel value in the area and setting it as a candidate point for both eyes.

More preferably, the step of searching for the center point of each of the two areas further comprises: 1) the candidate points searched in the step of searching the center of the darkest area are It is greater than or equal to a predetermined minimum value, 2) the distance between candidate points is less than or equal to a predetermined maximum value, and 3) the angle between the straight line connecting the candidate points and the scanning line direction is predetermined. Satisfying all of the above conditions, it may be determined whether all the conditions are satisfied, and if any of the conditions is not satisfied, the search may be failed.

According to a second aspect of the present invention, in a computer to which an image pickup means for picking up a video image of a subject's face is connected, the eye position of the subject is tracked within a series of images output by the image pickup means. And an image pattern between the eyebrows determined by the position of the eyebrows relative to the position of the eyebrows, and the position of the eyebrows of the face image output by the imaging unit, Image acquisition means for acquiring an image, a prediction means for predicting a position between the eyebrows of a face image in a subsequent image, and in the vicinity of the predicted position between the eyebrows,
Based on the first searching means for searching the center point of the region that best matches the image pattern between the eyebrows, the position of the searched point, and the relative position of both eyes, Second search means for predicting the positions of both eyes and searching for a center point of each of the two regions satisfying a predetermined condition with the predicted region as the center, and two searched points. The position of the middle point of the center point is set as the position of the new eyebrow, and the area determined by the position of the new eyebrow and the relative position of the two center points is set as the new relative position of both eyes with respect to the position of the eyebrow. The image pattern of the area defined by the new position between the eyebrows is used as a new image pattern between the eyebrows, an updating unit for updating the position between the eyebrows, the relative position of the eyes, and the image pattern between the eyebrows, and subsequently. Image obtaining means to repeat the processing from the image acquisition unit to the image for further subsequent to the image, and means for controlling the prediction means, first and second search means, and updating means.

Preferably, the predicting means extrapolates the position of the eyebrows of the subsequent image from the position of the eyebrows in the image output by the image pickup means and the position of the eyebrow in the image preceding the image. Including means for.

More preferably, in response to the second searching means not being able to search for the point, the position between the eyebrows of the face image output by the image pickup means, the relative position of both eyes to the position between the eyebrows, and the space between the eyebrows. The image pattern between the eyebrows defined by the position is extracted by a predetermined device, and the image acquisition means, the prediction means, the first and second search means, and the update means are controlled so as to restart the processing from the image acquisition means. Further included are means for:

The second searching means is a region having a predetermined shape in the vicinity of each of the positions determined by the relative positions of the eyes with respect to the position of the searched point, and Means may be included for searching the center of the region where the average of the pixel values in the region is darkest and using it as a candidate point for both eyes.

More preferably, in the second searching means, the candidate points searched by the means for searching the center of the darkest area are: 1) The distance between the candidate points is equal to or more than a predetermined minimum value. 2) the distance between the candidate points is less than or equal to a predetermined maximum value, and 3) the angle between the straight line connecting the candidate points and the scanning line direction satisfies a predetermined relationship. Means may be included for determining whether or not all the conditions are satisfied, and failing the search if any of the conditions is not satisfied.

[0031]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [Hardware Configuration] FIG. 1 shows the external appearance of a computer system 20 which implements an eye position tracking method and apparatus according to the present invention. The computer system 20 detects and tracks the position of the eyes, and a computer (not shown) controls the eye camera by using the information about the distance to the face of the user and the direction of the eyes provided from the computer system 20. In the present specification, the “subject” refers to a user whose system detects and tracks the position of the eyes, and includes not only humans but also animals and the like that have “eyes” equivalent. May be included.

Referring to FIG. 1, the computer system 20 includes a CD-ROM (Compact Disc Read Only Memo).
computer 40 equipped with a CD-ROM drive 50 for ry), an FD drive 52 for a flexible disk (FD), a mouse 48 which is a first pointing device connected to the computer 40, and a keyboard 46. And a monitor 42 and a stereo video camera 56 disposed between the monitor 42 and the keyboard 46,
58 and an eye camera 54 controlled by the separate computer described above.

FIG. 2 shows the internal configuration of the computer 40 together with other components of the computer system 20. Referring to FIG. 2, the computer 40 has a CPU (Ce
ntral Processing Unit) 70, a memory 72 connected to the CPU 70, and a hard disk 74 connected to the CPU 70. The CPU 70 is connected to a network 80 via a network board (not shown), and the above-described computer 82 that controls the eye camera 54 is connected to the network 80. Since the configuration of the computer 82 is similar to that of the computer 40, its details will not be described here.

An eye position tracking method and apparatus according to the present invention, which will be described in detail below, includes the computer hardware described with reference to FIGS.
It is realized by software executed by the PU 70. This software is distributed in the market by being stored in a storage medium such as a CD-ROM or via a network. When this software is installed on this computer 40, it is typically stored on hard disk 74. Then, when the user starts this software, or when it is started by the operating system (OS) of the computer 40, it is read from the hard disk 74 to the memory 72 and executed by the CPU 70.

It should be noted that this software may utilize a function originally incorporated as a part of the OS of the computer 40, or may utilize a part of a function of another software installed separately from the OS. obtain. Therefore, what is circulated as software for implementing the present invention may lack such a part of the functions, and has a configuration considering that such functions are realized by other software at the time of execution. If you have. Of course, all necessary functions may be distributed as system software in which all necessary functions are incorporated in advance.

[Principle] In the present invention, the basis of the method for detecting the position of the eyes is based on the difference between the frame images, and in particular, the difference in brightness due to blinking. When a blink is to be detected from the inter-frame difference, if the face is moving, pixels with large brightness changes occur in many portions other than the moving blinking portion. Therefore, it is necessary to distinguish the change in brightness caused by the movement of the eyelid and the change in brightness caused by the movement of the entire face. Although problems may occur due to changes in the mouth shape and eyebrow movements, those brightness changes are rejected by the conditional test after extracting eye candidate points.

The movement of the whole face is a translation in the image plane,
It can be divided into rotation within the image plane (shaking the neck) and rotation outside the image plane (turning around, nodding). The frame rate should be fast enough and the movement between frames should be small. Hereinafter, a method of canceling the movements of these will be described in order.

Movement when the face is translated in the image plane
The way to cancel is as follows. Figure 3
Image f shown in (A)_t-1The color shown in
The disc 90 moves to the right by the distance d and moves to the next frame.
Image f shown in FIG. _tIf it becomes like
think of. The disk has a white small circle mark. background
Are not necessarily shown, but they are not shown.

When the difference image between the images of FIGS. 3A and 3B is taken, the image is as shown in FIG. In FIG. 4, a shaded area is a portion where a large difference in pixel value appears, and a white area is a small difference in pixel value. In the difference image D shown in FIG. 4, considering the pixel positions where a large difference in pixel value appears, returning to the original image shown in FIG. 3A or 3B, these pixel positions are Further, it can be classified into regions F and B shown on the left and right of the upper part of FIG. The area F corresponds to Foreground, that is, a point on the disk, and the area B corresponds to Background, that is, the background. The area B is a part that is visible in one image but not in the other image. The area F is a portion visible in both images and having pixels corresponding to each other. However, in the actual image, since it is unknown which part the moving object is, it is not possible to distinguish which is the region F and which is the region B.

Here, the pixels corresponding to the regions F and B on the image f _t shown in the upper right of FIG. 5 are shifted by -d and overlapped on the image f _t-1 . Then, as shown in the lower left of FIG. 5, it can be seen that there is a portion of the region M where the pixel values match and a portion of the region U where the pixel values can and cannot be said to match. Similarly, when pixels corresponding to the regions F and B on the image f _t-1 are shifted by + d and overlapped on the image f _t , the regions M and U are as shown in the lower right of FIG. . The movement amount ± d here is a two-dimensional vector amount.

Assuming that the shift amount is a shift amount ± d 'different from the actual shift amount ± d, the number of pixels belonging to the region M becomes smaller than when the actual shift amount ± d is used. Conceivable. Therefore, if a shift amount that maximizes the number of pixels in the region M is searched, it can be said that the shift amount is the movement amount d of the disc 90 between frames. In counting the number of pixels in the region M, both must always have the same size in opposite directions so that the shift amount of the image f _t becomes −d when the shift amount of pixels of the image f _t−1 is d. I have to.

As described above, among the pixels extracted from the difference image between frames, the shift amount ± d that maximizes the number of pixels belonging to the region M is found, the images are shifted and superimposed, and the pixel value It can be said that when a pixel in which is not matched is extracted, it is a portion that has moved other than the parallel movement with the movement amount d. If the amount of movement of the entire face is d, d
The parts that have moved other than are parts derived from blinks, changes in the mouth shape, and the like. Therefore, by canceling the movement of the entire face from the inter-frame difference image, it is possible to extract the blinking portion, that is, the eye position.

Next, consider the turning motion. With reference to FIG. 6, consider a case where the face facing the cameras 56, 58 is about to turn to the left. In this case, the hidden part appears from the right end (left end in FIG. 6) of the face,
At the left end of the face (right end in FIG. 6), the visible part is hidden by self occlusion. Since such a portion is hidden in one of the two images for which the inter-frame difference is calculated, it is meaningless to shift and see the coincidence of the pixel values unlike the case of the parallel movement.

However, such a portion appears only in the peripheral portion of the moving area. Therefore, after calculating the inter-frame difference image (FIG. 4), the influence can be removed by removing a certain number of pixels from the left and right of the hatched area. On the other hand, since the center part of the face can be regarded as parallel movement even in the case of a turning motion, the above-described method regarding parallel movement (a method of moving two images by ± d to see matching pixels) can be applied. .

The movements of squeezing the neck and the nod are
Since the center of rotation is off the face, it can be regarded as a parallel movement. However, in either case, it is premised that the movement between frames is small.

[Control Structure of Software] The structure of software for realizing the eye detection processing and the tracking processing based on the above principle will be described below. As described above, when executing the software described below, it may be premised that the functions of the OS or the functions already installed in the computer are used, and as a result, the distributed software does so. It may not include the function itself. However, even in that case, such software falls within the technical scope of the present invention as long as it has information about where in the control structure such a function is used.

In the processing described below, in addition to the processing for detecting eye candidate points already described, on the assumption that there are two eyes, blinks occur simultaneously in the left and right eyes, and there is a distance between both eyes. By using conditions such as being in the range, the angle of inclination of the face being within a certain range, and the grayscale pattern near the midpoint of both eyes being almost symmetrical, the candidate point pairs of the extracted eye positions are paired. It tests whether or not is an eye.

Further, various thresholds are used in the following processing, but the values are empirically determined depending on the application and the required accuracy.

Referring to FIG. 7, the overall structure of this software is as follows. First, the eye position is extracted using a procedure described in detail later with reference to FIG. 8 and subsequent figures (step 100). In the following step 102, the extracted eye position is transmitted to the computer 82 which controls the eye camera 54. Then, in step 104, the eye position tracking process is performed. It is determined whether or not the eye position has been tracked, that is, whether or not the eye position has been lost (step 106). If the tracking fails, the control returns to step 100, and the control is executed again from the eye position extraction processing of step 100. If the tracking is completed, the control returns to step 102, and the information such as the direction of the eyes and the distance to the eyes obtained by the tracking is transmitted to the computer 82. Thereafter, by repeatedly performing the processing of steps 100 to 106, it is possible to constantly transmit the information regarding the direction and the distance of the user's eyes to the computer 82.

Note that the eye positions may not be extracted in step 100 of FIG. In that case, the process of step 100 is executed again, the eye position extraction process is executed for the next frame, and then the tracking process is started again.

Referring to FIG. 8, the eye position extraction processing performed in step 100 of FIG. 7 will be described. The following processing uses the principle described with reference to FIGS. 4 to 6. It is assumed that the previous image has been captured before this processing. Especially at the beginning of the process,
The previous image may be matched with the current image. First,
At step 120, the current image from the cameras 56, 58 is captured. The following processing is actually performed on the images of the two cameras 56, and after detecting the eyes, the distance to the eyes is calculated by stereo processing together with the images of the cameras 58. The roles of the camera 56 and the camera 58 may be changed.

Subsequently, at step 122, the inter-frame difference binarized image D between the previous image and the current image is calculated. More specifically, the absolute value of the difference between the values of the previous image and the current image is calculated for each pixel, and the value is determined by a predetermined threshold value N.
If it is above, the pixel value is set to "1", and if it is less than the threshold value N, the pixel value is set to "0". Since the video signal fluctuates to some extent even when capturing a stationary object,
It is necessary to use such a threshold value N. This threshold N can be considered as a noise level.

Subsequently, at step 124, it is judged whether or not the number of pixel values "1" in the difference image D is equal to or larger than a predetermined threshold value G. If the determination result is “YES”, the control jumps to step 168 shown in FIG. 9, and it is determined that no eyes have been detected, and the process ends. In this case, since motion has been detected in too many pixels, it is assumed that there is more motion than expected, and the next frame input is awaited.

When the result of the determination in step 124 is "NO", it is determined in step 126 whether the number of pixel values "1" in the difference image D is less than or equal to the threshold value E. This threshold E
Is a value smaller than the above-mentioned threshold value G and is for determining whether or not the face is substantially stationary. If the decision result in the step 126 is "YES", the control jumps to the step 152 shown in FIG. In this case, the number of pixels extracted by the interframe difference is small, and the face is considered to be in a stopped state.

When the result of the determination in step 126 is "NO", the control proceeds to step 128. In step 128, the pixel group having the pixel value “1” of the difference image D is subjected to the process of deleting the pixels at both ends thereof. This processing considers the turning motion as described with reference to FIG.

More specifically, each scanning line of the difference image D is searched in the rightward and leftward directions from the left and right ends, and the pixel having the pixel value "1" that appears first is searched for. Then, the pixel values of k pixels from that pixel are rewritten to "0". By this processing, a predetermined amount of the left and right regions of the pixel value "1" in the image is removed.

Subsequently, at step 130, the pixel corresponding to the pixel value "1" of the difference image D is extracted from the previous image and the current image. This process is a process for obtaining the image shown in the middle part of FIG. Thereafter, in step 132, the above-mentioned movement amount ±
The process of calculating d is performed. Details of this processing will be described later with reference to FIG.

When the movement amount ± d is calculated, subsequently, in step 150 (FIG. 9), a process of deleting the pixels corresponding to the region M shown in the lower part of FIG. 5 from the difference image D is performed. Further, in step 152, isolated point removal and smoothing processing is executed on the difference image D. More specifically, if the pixel value of 4 pixels or more in total of 8 neighborhoods of the pixel to be processed and the central pixel is “1”, the pixel value of the pixel to be processed is set to “1”. . Otherwise, the pixel value is set to "0". By this processing, the pixel value of the isolated point becomes “0” and is deleted from the difference image D.

Subsequently, in step 154, the labeling process is performed on the difference image D. By this processing, a label is given to each of the regions made up of pixels having pixel values of "1" that are continuous with each other. The center of each element thus obtained is set as the eye point candidate point (step 156).

It is determined whether the number of obtained candidate points is less than 2 (step 158). Since it is generally assumed that there are two eye positions, if the number of candidate points is less than two, it is determined that no eye has been detected, and the process proceeds to step 168. If the number of candidates is 2 or more, the control proceeds to step 160.

In step 160, for each candidate point, the sum S of the absolute values of the differences between the pixel value of the previous image and the pixel value of the current image of each pixel in the rectangle a × b centered on the candidate point is set. Ask for. Here, the rectangle a × b is a rectangle having a size corresponding to the expected size of the eyes. If there is a blink between the previous image and the current image, one image is the image of the eye and the other image is the image of the eyelid, so the absolute value of the difference between the pixel values is large and its size is large. Should be close to the expected eye size. Therefore, S can be obtained for each candidate point in this way, and the two with the largest value can be used as the second candidate point (pair of candidate points).

Subsequently, in step 162, it is judged whether or not the two candidate points obtained as described above satisfy the following conditions. If all the conditions are not satisfied, it is determined that no eye has been detected (step 168), and the process ends.

1) The sum S of the absolute values of the differences obtained in step 160 is equal to or greater than the predetermined threshold value F at any of the two candidate points.

2) The distance between the candidate points is greater than or equal to a predetermined value Lmin. 3) The distance between the candidate points is less than or equal to a predetermined value Lmax.

4) The angle formed by the straight line connecting the candidate points and the scanning line direction is less than or equal to a predetermined threshold value A degrees.

The threshold value F is a value that is experimentally or statistically determined in advance, and is used to prevent portions other than the eyes from being extracted as the eyes. Lmin and Lmax are the minimum and maximum values that are assumed as the distance between the centers of the eyes, respectively. These values are statistically determined beforehand. An appropriate value for the threshold angle A is also determined experimentally, but normally it is 30 to 4 degrees.
Values around 5 degrees are used.

When all of the above four conditions are satisfied, it is further determined in step 164 whether the grayscale pattern near the midpoint of the obtained two eye candidate points is substantially symmetrical. If it is determined that they are substantially symmetrical, these two eye candidate points are set as eye positions. If the eyes are not symmetrical, the processing is terminated because the eyes cannot be found.

A method for determining whether or not the patterns near the midpoints of the two eye candidate points are substantially symmetrical will be described. Referring to FIG. 11, generally in a region 220 centered on the midpoint of both eyes (hereinafter referred to as “between eyebrows”), the gray pattern on the left and right between the eyebrows is a line segment connecting two eye candidate points. It should be almost symmetric with respect to a straight line that is perpendicular to the line between the eyebrows (the midpoint between the two eye candidate points). Therefore, first, the current image is rotated about the midpoint of the candidate points of both eyes so that the candidates for both eyes are arranged horizontally on the scanning line, and g × h pixels (region 220 is set) centered on the midpoint of the candidate points. .) Is cut out. As shown in the lower part of FIG. 11, the sum of the pixel values of the vertical h pixels in this area 220 is obtained for each column, and the sum of each column with respect to the sum of all pixels is calculated as a percentage. , A projection profile such as the graph 222 shown in FIG. 11 is obtained. Then, in this profile, a total of three columns adjacent to each other, which are symmetrical with respect to the midpoint of both candidates, is calculated for both the left and right sides. Then, the sum of the differences between the left and right sums is calculated for all combinations of three adjacent columns, and if all the values are equal to or less than the threshold value p, the pattern near the midpoint of the two eye candidate points is , And determine that they are almost symmetrical.

Specifically, referring to the lower part of FIG. 11, for example, the sum of the values of the three columns 224L at the leftmost end and the value of 3 at the rightmost end.
The sum of the values in the column 224R is calculated, and the difference p ₁ is calculated. Similarly, the sum of the values of the third column 226L from the second column to the fourth column from the left and the sum of the values of the third column 226R from the second column to the fourth column from the right are calculated, and the difference p ₂ is obtained. . 3rd column from the left
The sum of the values of the 3rd column 228L of the 5th column and the 3rd to 5th columns from the right
The sum of the values in the third column 228R in the third column is calculated, and the difference p ₃ is calculated. Similarly, the values p _{4 to} pg _-2 are calculated, and p ₁ to
By obtaining the sum of the values of p _g-2 and comparing the sum with the threshold value p, it is possible to determine whether or not the grayscale pattern near the midpoint of the two eye candidate points is substantially symmetrical.

In the above description, the two eye candidate points are once rotated about their midpoints so as to be aligned with the horizontal scanning line, and then it is determined whether the light and shade pattern is symmetrical.
However, the symmetrical determination method is not limited to this. For example, if the angle formed by the line segment connecting the two eye candidate points with respect to the horizontal scanning line is known, the total pixel value can be calculated in the direction perpendicular to the line segment.
Processing similar to that shown in 1 can be realized. In the above example, the total is calculated for each column, but the difference is calculated for each pixel that is symmetrical about the midpoint, and the sum is calculated. It may be determined to be symmetrical.

Now, returning to step 132 of FIG. 8 again,
The process of calculating the movement amount d will be described in detail with reference to FIG. Movement amount d (two-dimensional vector amount) = (x,
y), x = -m, -m + 1, ..., m, y = -n,-
An area M as shown in the lower part of FIG. 5 in the range of n + 1, ..., N.
The one showing the maximum value of the total number of pixels belonging to is searched. First, the initial value 0 is substituted for the variable max (step 180). This variable max is for storing the maximum value of the total number of pixels belonging to the region M in the difference image calculated for each movement amount in the calculation process described below.

Substitute -m for variable x (step 18
2) Substitute -n for variable y (step 184). Hereinafter, iterative processing is performed for each value of y.

First, the total number M (d) of pixels belonging to the region M for d = (x, y) is obtained (step 18).
6). Then, it is determined whether or not this M (d) is larger than the variable max (step 188). M (d) is the variable ma
If x or less, nothing is done and the control proceeds to step 192. When M (d) is larger than the variable max, the value of M (d) is substituted for the variable max in step 190, and X, Y
The x and y values at that time are respectively substituted into. Control continues to step 192.

At step 192, 1 is added to y, and it is determined whether or not y exceeds n (step 19).
4). If y does not exceed n, the control is step 1
Returning to 86, steps 186-194 for the new y.
The process of is repeated. If y exceeds n, control proceeds to step 196.

At step 196, 1 is added to x.
Then, it is determined whether or not the result x exceeds m (step 198). If it is determined that x does not exceed m, the control returns to step 184, and the processing of steps 184-198 is repeated for the new x. x is m
If it is determined that the value exceeds the limit, control proceeds to step 200.
Then, the process ends with the value of the variable max and the values of x and y when the value of the variable max is obtained.

In this way, the movement amount d that maximizes the number of pixels included in the area M is obtained. Note that various other methods of calculating the movement amount d are possible. For example, x, y
It is also possible to find the number of pixels to be included in the region M in a brute force in advance while making a change in the range of m × n to make a table, and find the cell having the maximum value in the table.

In order to track the eye detected as described above from the next frame with high reliability and robustness, the following tracking method is realized by software. In the present embodiment, the eye is not directly tracked but the pattern between the eyes (between eyebrows) is tracked. Then, the eye is searched for in the vicinity of the point where the relative position from the eyebrow thus tracked is the same as the eye position of the previous frame.

The eyebrow pattern is relatively stable even when the facial expression changes. Further, since a light and shade pattern in which eyes and eyebrows are cut in from both sides like wedges is formed in the forehead and the bright part of the nose, it is easy to obtain accuracy in determining the position by pattern matching.

As described below, when performing pattern matching using the template between the eyebrows, it is necessary to update the template. In the present embodiment, a rectangular pattern having the center of both eyes as the midpoint is adopted as a template,
The rectangular pattern detected in the current frame is used to update the template.

Details of the tracking process will be described below with reference to FIG. In addition, the midpoint positions of both eyes extracted in the previous frame, the eyebrow pattern (s × t pixels),
Relative position e _r , e _l (e _r =
-E _l , both are two-dimensional quantities) and are already saved.

First, in step 240, a process of taking in the midpoint position, the eyebrow pattern, and the relative positions of both eyes saved as initial values is performed. Then, the current image (current frame) is captured (step 242).

In the next step 244, the midpoint position of both eyes in the current image is predicted from the midpoint positions of both eyes in the previous image and the image before that (previous image). That is, the positions of the eyebrows between the previous frame and the previous-previous frame are set to X _t-1 and X _t-2 , respectively (where X _t-1 and X _t-2 are both 2).
Dimension amount) and the predicted position X _t in the current frame are X _t = 2X
Extrapolation is possible with _t-1 ―X _t-2 . However, at the first detection, X _-1
= X ₀ . In this way, the midpoint position of both eyes in the current image is predicted when the face image is moving, because the amount of movement is considered to be almost constant. This is because it is efficient to match the pattern between the eyebrows.

In step 246, a matching process is performed in the vicinity of the midpoint position predicted in step 244 to search for an eyebrow pattern that best matches the saved eyebrow pattern. The position at which the best match is obtained is X _t0 . Then, in step 248, the positions of the right eye and the left eye are searched in the neighborhood (i × j pixels) centered on X _{t0 +} e _r and X _{t0 +} e _l , respectively (step 2
48). In the search, 5 × 5 centered on that point
The point having the darkest average pixel value of the pixels is determined as the eye.

Next, it is determined whether or not the eye position of the search result satisfies the conditions 2) 3) 4) used in step 162 of FIG. If any one of the conditions is not satisfied, it is determined that the eye tracking is wrong or lost, and the tracking failure is determined (step 252).
The process ends. In this case, the control returns from step 106 of FIG. 7 to step 100, and the processing is performed again from the detection of the eye position.

If it is determined that all the conditions are satisfied, it is determined that the tracking has succeeded,
The tracking process is performed by saving (updating) the midpoint position of both eyes obtained by tracking, the eyebrow pattern (s × t pixel) centered on that point, and the relative positions e _r and e _{l of} both eyes with respect to the midpoint. To finish.

If the positions of both eyes are detected in the image of the camera 56 in this manner, the distance to the eye is obtained by stereo processing together with the image of the camera 58, and the eye camera 54 is controlled. Parameters (distance, direction) can be determined by calculation. A well-known technique can be applied to the processing, and thus the detailed description will not be repeated here.

[Operation] The eye position detecting device according to the present invention, the structure of which has been described above, operates as follows. The stereo cameras 56 and 58 shown in FIG. 1 capture a face image of the user, output video signals to the computer 40, and apply the video signals to the computer 40.

When the software for eye position detection and tracking is activated as shown in FIG. 8, the eye position is extracted for each of the video images obtained from the cameras 56 and 58. Specifically, the current image is captured (step 120) and the difference image D is calculated (step 1).
22), the determinations of steps 124 and 126 are made. If the determination result in step 124 or 126 is “YES”, it is determined that the eye position cannot be detected, and the eye position extraction process is executed again for the next frame image.

If the determination results at steps 124 and 126 are both "NO", step 1
At 28, the pixels at both ends of the area having the pixel value “1” are deleted.
Then, after the movement amount d is calculated according to the method already described (step 132), the movement amount d thus obtained is used to move to the region M obtained by performing the movement and superposition shown in FIG. The corresponding pixels are removed from the difference image D (step 150).

Subsequently, the center of each element obtained through isolated point removal, smoothing processing (step 152) and labeling processing (step 154) is set as an eye position candidate (step 1).
56). If the number of candidates is less than 2, it is determined that the extraction has failed, and the process is restarted from the beginning. When the number of candidates is two or more, the two points having the largest sum of absolute values of the differences between the images in the vicinity thereof are candidate points (step 160). It is determined that this candidate point satisfies all of the above four conditions (step 162) and that the pattern near the midpoint is symmetrical (step 164).
Then, with the two candidate points as eye positions, the eye position extraction process ends. If the test in steps 162 and 164 fails, the eye position cannot be extracted,
The process is restarted from the beginning.

When the direction of the eyes of both the stereo cameras 56 and 58 is determined in this way, information for controlling the eye camera 54 using those information (eye camera 54
To the visual direction and the distance to the eye). Such information is transmitted to the computer 82 (step 102 in FIG. 7), and the control shifts to eye position tracking processing (step 104).

At the beginning of the tracking process, the information about the midpoint position of both eyes (that is, the position between the eyebrows) obtained by the eye position extraction process, the image pattern between the eyebrows, and the relative position of both eyes viewed from the midpoint position. Has been saved. First, the information thus saved is fetched as an initial value (step 240). Then, the current image is captured (step 242), and the midpoint position of the eye in the current image is predicted based on the midpoint position of the eye (position between the eyebrows) in the previous image and the midpoint position of the eye in the previous image ( Step 24
4). This prediction is performed, for example, by extrapolating the midpoint positions of the two images before and after. In the vicinity of the predicted midpoint position, search for the center position of the pattern that best matches the already saved eyebrow pattern in the previous image, and locate the positions of both eyes centering on the position showing the best match. Search (step 248). If the obtained binocular position satisfies the predetermined condition (YES in step 250), the midpoint position of both eyes is calculated based on the binocular position, and the value and the eyebrow centered around the midpoint position are calculated. The pattern and the relative position from the midpoint position to both eyes are saved, and the tracking is temporarily terminated. However, in this case, the next tracking process is performed through the route from step 106 to step 102 in FIG.

If the tracking fails, the process proceeds from the step 106 to the step 100 in FIG. 7 and the eye position extraction processing is performed again.

Thus, the positions (directions) of the eyes are extracted and tracked using the stereo cameras 56 and 58, and the parameters (directions and distances of the eyes) for controlling the eye camera 54 are obtained by using these values. The eye camera 54 can be calculated and controlled.

13 to 15 show screen displays of specific examples of eye position extraction and tracking processing according to the present embodiment.

FIG. 13 shows an example of the input image. FIG. 14 shows a difference image between this image and the previous image. A pattern 270 cut out to determine left-right symmetry is displayed on the upper left of the screen, and a pattern between eyebrows 272 of the input image is displayed on the lower left. In the difference image, two places 274 and 276 are extracted as eye position candidates and displayed as white circles.

FIG. 15 shows an example in which the positions of both eyes 292 and 294 and the center position 290 are superimposed and displayed on the image. As shown in FIG. 15, the positions of both eyes are correctly tracked with the center between the eyebrows.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

[Brief description of drawings]

FIG. 1 is an external view of a computer system 20 that implements an eye position tracking method and apparatus according to the present invention.

FIG. 2 is a block diagram of the computer system 20 and peripheral devices shown in FIG.

FIG. 3 is a diagram showing the principle of eye position extraction according to the embodiment of the present invention.

FIG. 4 is a diagram showing an example of a difference image.

FIG. 5 is a diagram showing the principle of eye position extraction according to the embodiment of the present invention.

FIG. 6 is a diagram showing the principle of canceling the movement of the user turning around in the embodiment of the present invention.

FIG. 7 is a flowchart showing the overall control structure of software that realizes the eye position extraction method and apparatus according to the embodiment of the present invention.

FIG. 8 is a flowchart of software that implements eye position extraction processing according to the embodiment of the present invention.

FIG. 9 is a flowchart of software that implements eye position extraction processing according to an embodiment of the present invention.

FIG. 10 is a flowchart of software that implements a process of calculating a movement amount d in the eye position extraction process according to the embodiment of the present invention.

FIG. 11 is a diagram for explaining processing for determining left-right symmetry in the vicinity of the eyebrow in the embodiment of the present invention.

FIG. 12 is a flowchart of software that implements eye tracking processing according to an embodiment of the present invention.

FIG. 13 is a diagram showing a display example of a current image according to the embodiment of the present invention.

FIG. 14 is a diagram showing a display example of a difference image according to the embodiment of the present invention.

FIG. 15 is a diagram showing a display example of eye position tracking according to an embodiment of the present invention.

[Explanation of symbols]

20 computer system, 40,82 computer, 42 monitor, 46 keyboard, 48 mouse,
54 eye camera, 56, 58 stereo camera.

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 5B057 AA20 CA12 CA16 CH08 CH11                       CH12 DA07 DA08 DB02 DC09                       DC34                 5L096 CA02 FA69 HA04 HA07 JA03                       JA09

Claims (15)

[Claims] 1. A method for tracking the position of the subject's eyes in a series of images output by the image capturing means in a computer to which image capturing means for capturing a video image of a subject's face is connected, An image pattern between the eyebrows determined by the position between the eyebrows of the face image output by the imaging means, the relative position of both eyes with respect to the position between the eyebrows, and the position between the eyebrows is given in advance, and an image subsequent to the image is acquired. And a step of predicting a position between the eyebrows of the face image in the subsequent image, and a center of a region that best matches the image pattern between the eyebrows in the vicinity of the predicted position between the eyebrows. A step of searching for a point, the position of the searched point, and the position of both eyes in the subsequent image based on the relative position of the both eyes,
A step of searching for a center point of each of the two areas satisfying a predetermined condition centered on the predicted area, and a new position of the midpoint of the searched two center points is newly set. The position between the eyebrows, the area defined by the new eyebrow position and the relative position of the two central points is the new relative position of both eyes with respect to the eyebrow position, and the area defined by the new eyebrow position Updating the image pattern as a new image pattern between the eyebrows, the position between the eyebrows, the relative positions of both eyes, and the image pattern between the eyebrows, and the first image with respect to an image further following the subsequent image. The method of tracking an eye position, the method including: 2. The step of predicting the position between the eyebrows comprises:
The method according to claim 1, further comprising extrapolating a position between the eyebrows of the subsequent image from a position between the eyebrows in the image output by the imaging unit and a position between the eyebrows in the image preceding the image. The method described. 3. In the step of searching for a point that is the center of each of the two areas, in response to the fact that the point cannot be searched, the position between the eyebrows of the face image output by the image pickup means, and the space between the eyebrows. 2. The method according to claim 1, further comprising the step of extracting an image pattern between the eyebrows defined by the relative positions of both eyes with respect to the position of the eyebrow and the position between the eyebrows by a predetermined method, and restarting the process from the first step. Item 2. The method according to Item 2. 4. The step of searching for a center point of each of the two regions, in the vicinity of each of the positions defined by the relative positions of the eyes with respect to the position of the searched point, 4. The method according to claim 1, further comprising a step of searching a center of an area having a predetermined shape and having a darkest average pixel value in the area and setting the center as a candidate point for both eyes. The method described in crab. 5. The step of searching for a center point of each of the two areas further includes: 1) the candidate points searched in the step of searching for the center of the darkest area; It is greater than or equal to a predetermined minimum value, 2) the distance between candidate points is less than or equal to a predetermined maximum value, and 3) the angle between the straight line connecting the candidate points and the scanning line direction is predetermined. The method according to claim 4, further comprising the step of determining whether all the conditions are satisfied, and failing the search if any of the conditions is not satisfied. 6. An apparatus for tracking the eye position of the subject in a series of images output by the image capturing means in a computer to which image capturing means for capturing a video image of the face of the subject is connected, An image pattern between the eyebrows determined by the position between the eyebrows of the face image output by the imaging means, the relative position of both eyes with respect to the position between the eyebrows, and the position between the eyebrows is given in advance, and an image subsequent to the image is acquired. Image acquisition means for performing, a prediction means for predicting the position between the eyebrows of the face image in the subsequent image, a region near the predicted position between the eyebrows, the region that best matches the image pattern between the eyebrows A first searching means for searching for a point that is the center of the, and the position of the searched point and the relative position of both eyes, in the subsequent image. To predict the position of the eyes,
Second searching means for searching a center point of each of two areas satisfying a predetermined condition centering on the predicted area, and among the searched two center points The position of the point is a new position between the eyebrows, a region defined by the position of the new eyebrows and the relative position of the two central points is a new relative position of both eyes with respect to the position of the eyebrow, and the new position between the new eyebrows. An image pattern of a region defined by the position of the eyebrow as a new image pattern between the eyebrows, the position between the eyebrows, the relative position of both eyes, and updating means for updating the image pattern between the eyebrows, further to the subsequent image The image acquisition unit, the prediction unit, the first and second search units, and the image acquisition unit so as to repeat the process from the image acquisition unit for the subsequent images, and And means for controlling the serial updating means, the tracking device locations eye. 7. The predicting means excludes the position between the eyebrows of the subsequent image from the position between the eyebrows in the image output by the imaging means and the position of the eyebrow in the image preceding the image. 7. The device of claim 6, including means for inserting. 8. The position of the eyebrow of the face image output by the imaging unit, the relative position of both eyes with respect to the position of the eyebrow, and An image pattern between the eyebrows defined by the position between the eyebrows is extracted by a predetermined device, and the image acquisition means, the prediction means, the first and second search means, so as to restart the processing from the image acquisition means, 8. An apparatus according to claim 6 or claim 7, further comprising means for controlling the updating means. 9. The second searching means is a region having a predetermined shape in the vicinity of each of the positions determined by the relative positions of the eyes with respect to the position of the searched point. 9. The apparatus according to claim 6, further comprising means for searching for a center of a region where the average of pixel values in the region is darkest to be a candidate point for both eyes. 10. The candidate points searched for by the means for searching the center of the darkest region are further 1) the distance between the candidate points is equal to or greater than a predetermined minimum value. 2) the distance between the candidate points is less than or equal to a predetermined maximum value, and 3) the angle between the straight line connecting the candidate points and the scanning line direction satisfies a predetermined relationship. 10. The apparatus of claim 9, including means for determining if all conditions are met and failing the search if any of the conditions are not met. 11. A method for tracking the eye position of the subject in a series of images output by the imaging means in a computer connected to an imaging means for capturing a video image of the subject's face. Is a computer-executable program for controlling the computer, wherein the position between the eyebrows of the face image output by the imaging means, the relative position of both eyes with respect to the position between the eyebrows, and the eyebrows determined by the position between the eyebrows. Image pattern is given, the method comprises: a first step of obtaining an image subsequent to the image; a step of predicting a position between eyebrows of a face image in the subsequent image; A point near the position of the center of the region that best matches the image pattern between the eyebrows, The position of the cord has been point, based on the relative position of the eyes, we predict the positions of the eyes in the subsequent image,
A step of searching for a center point of each of the two areas satisfying a predetermined condition centered on the predicted area, and a new position of the midpoint of the searched two center points is newly set. The position between the eyebrows, the area defined by the new eyebrow position and the relative position of the two central points is the new relative position of both eyes with respect to the eyebrow position, and the area defined by the new eyebrow position Updating the position of the eyebrow, the relative position of the eyes, and the image pattern of the eyebrow as the new image pattern of the eyebrow. Eye position tracking program, including the steps of performing processing from step 1. 12. The step of predicting the position between the eyebrows includes the position between the eyebrows in the image output by the imaging unit,
From the position of the eyebrows in the image preceding the image, including the step of extrapolating the position of the eyebrows of the subsequent image,
The program according to claim 11. 13. The method, in the step of searching for a center point of each of the two areas, in response to the fact that the point could not be searched, between the eyebrows of the face image output by the imaging means. The method further includes: extracting a position, a relative position of both eyes with respect to the position between the eyebrows, and an image pattern between the eyebrows defined by the position between the eyebrows by a predetermined method, and restarting the process from the first step. The program according to claim 11 or 12. 14. The step of searching for a center point of each of the two regions, in the vicinity of each of the positions defined by the relative positions of the eyes with respect to the position of the searched point, 14. The method according to claim 11, further comprising a step of searching a center of a region having a predetermined shape and having a darkest average pixel value in the region and setting the center as a candidate point for both eyes. The program described in Crab. 15. The step of searching for a center point of each of the two regions further includes: 1) the candidate points searched in the step of searching for the center of the darkest region; It is greater than or equal to a predetermined minimum value, 2) the distance between candidate points is less than or equal to a predetermined maximum value, and 3) the angle between the straight line connecting the candidate points and the scanning line direction is predetermined. 15. The program according to claim 14, further comprising the step of determining whether all the conditions are satisfied, and failing the search if any of the conditions is not satisfied.