JP2991134B2 - Attention point detection system on screen - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gazing point detecting system for detecting a gazing point on a screen of a person to be detected without contact.

[0002]

2. Description of the Related Art Conventionally, a gaze point detection system on a screen is generally used for detecting a human motion, particularly a point of interest. In this system for detecting human motion, it is important to detect the position of the face, especially the motion of the line of sight, in order to know what or where the person is watching.

For this reason, in the gaze direction detection system of the prior art 1, for example, as disclosed in JP-A-6-086758, an eyeball position detection device is fixed to a viewfinder of a camera, and the gaze direction is adjusted by irradiating the eyeball with light. The contact type that takes the method of obtaining is the mainstream.

On the other hand, in a non-contact type environment in which the position of the eyeball is not fixed, for example, Japanese Unexamined Patent Publication No. Hei.
No. 7 “Gaze direction detection method” is available. In this conventional example, since it is necessary to first obtain the position of the eyeball (or iris / pupil), a method of detecting the information near the iris part in the eye from the image above the shoulder of the person and detecting the gaze direction is described. Has been disclosed. More specifically, the difference in the intensity of the reflected light at multiple wavelengths is obtained, the position of the iris is calculated by simple image processing, and then changes in the apparent area of the iris and pupil and the reflected light from the cornea are tracked. By doing so, the gaze direction is obtained.

[0005] The detection of the direction of the line of sight is disclosed in Japanese Patent Application Laid-Open No. 5-232373, entitled "Device for detecting the direction of the line of sight" of JP-A-5-232373, which has a similar technical content. This conventional example proposes a device that enables visual axis detection corresponding to a change in magnification of a finder optical system.

[0006]

However, each of the above-described conventional examples has various problems. For example, the first problem is that it is difficult to fix the eyeball position. In the case of a camera viewfinder as in Conventional Example 3, it is relatively easy. However, it is generally difficult to fix the eyeball position as a pointing device on a graphic screen of a display, which is an interface between a computer and a user.

[0007] The reason is that the user works in various postures in a work environment on an office floor or at home, regardless of a special environment such as an aircraft or a dedicated machine. For this reason, it is difficult to fix the eyeball position. Furthermore, many people dislike wearing a helmet-type sensor. In particular, when operating while looking at the display screen, the user may not always face the front of the screen, and the position of his head often shakes constantly. These are very serious problems for gaze detection.

[0008] A second problem is that a human always blinks his eyes or temporarily closes his eyes. The reason is that blinking or closing your eyes can temporarily detect the position of your eyes, sometimes for a few seconds,
It is difficult to detect. If the head moves and the position of the eyes moves during that time, it will take some time until the position of the eyes is determined again. In particular, in the case of a display using a CRT or the like, eyestrain generally tends to accumulate more than a paper document or the like. For this reason, operations such as blinking and pinching of the eyes and rubbing the eyes with the hand are likely to occur. It is necessary to take measures against this point.

[0009] The third problem is that the amount of opening the eyelids differs depending on the individual. The reason is that if you have a small amount of eyelids to open, you cannot catch the iris enough.
For this reason, the detection of the reflected light becomes insufficient, and it becomes difficult to track the position of the eye.

A fourth problem is that when the non-contact type is selected, the system becomes large-scale. The reason for this is that two input systems are required to obtain the difference, and furthermore, the same sensor is used to detect the position of the iris and change the area of the iris. For this reason, it is necessary to prepare a high-resolution sensor capable of detecting the iris and the pupil for an area several times as large as the entire face.

More specifically, as shown in FIG. 5, when 1280 dots are displayed horizontally on a 17-inch display screen, if it is attempted to identify the position of the point to be watched with an accuracy of 4 dots, An accuracy of about 1 mm is required. There is a face 400mm away from the face, 1mm on the screen
Is obtained from the movement of the eyeball, the radius of the eyeball is about 2
Assuming 0 mm, detection of a movement of about 0.05 mm is required. If the position of the face fluctuates by 300 mm for the width of the display, the resolution of the sensor for detecting the iris and the pupil needs a resolution of 6000 points only in the horizontal direction. This is a very expensive sensor.

Further, a device for processing a high-resolution and large-area image is required to have a sufficient processing capability, which is also a factor for increasing the scale and cost of the system. As a pointing device on a graphic screen which is an interface between a computer and a user, a system capable of specifying a position with a line of sight is required. In particular, as a pointing device for persons with disabilities with limbs, an interface that can specify a position by watching a point on a screen is very effective from a humane and industrial viewpoint.

An object of the present invention is to provide a non-contact type, high-speed response, small-sized, and inexpensive gaze point detection system on a screen.

[0014]

In order to achieve the above object, a gaze point detection system on a screen according to the present invention uses a gray-scale image used for specifying the position of an eyeball with the entire face of a person to be detected as one image area. Means for obtaining a wide range of images, eyeball position determining means for determining the position of the eyeball by blinking the eye, eyeball imaging means for obtaining a precise image of the eyeball of the face looking at a predetermined display based on the determined position of the eyeball, Gaze point determination means for determining the gaze point of the display from the precise image of the above, so that the gaze point of the eyeball with respect to the display can be detected.

The above-mentioned eyeball imaging means determines the position of the eyeball by using the characteristic that the reflected light of the cornea of the eyeball changes by blinking of the eye, and further tracks the left and right eyeballs independently. The system comprises: a system tracking mirror and a liquid crystal shutter; an optical system having two optical axes as one system; and a narrow-range image sensor for imaging the optical system. It should be detected.

Further, the above-mentioned gaze point detection system on the screen has infrared light emitting means for emitting infrared light, and the gaze point of the eyeball is determined based on a change in the amount of reflected infrared light emitted from the infrared light emitted by the infrared light emitting means. It should be detected.

The eyeball position determining means includes a peak point detecting means for detecting a peak point of the light amount of the eyeball imaged by the eyeball imaging means, a grayscale image history storage means for storing and storing the grayscale image history data, Change amount peak detecting means for obtaining an amount of change from the history data and the newly obtained grayscale image,
It is preferable to include an eyeball position calculation unit that outputs a control signal for controlling the tracking operation of the eyeball imaging unit based on the detected peak point and the peak position of the change amount, and calculates the position of the eyeball.

The amount of reflected light from the face for detecting the point of gaze of the eyeball may be the amount of reflected light from the cornea of the pupil of the eye.

[0019]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of a system for detecting a gazing point on a screen according to the present invention; Referring to FIG. 1, there is shown an embodiment of the on-screen gaze point detection system of the present invention.

As described above, the gaze point detection system of this embodiment is a non-contact gaze point detection system on a non-contact screen for use as a pointing device on a graphic screen of a display which is an interface between a computer and a user. Make up.

FIG. 1 is an explanatory diagram showing one configuration example of a gaze point detection system on a screen applied to the present embodiment. In FIG. 1, a gazing point detection system according to the present embodiment includes a face 1 of a person to be detected, a light emission timing designation unit 2, an infrared light emission unit 3, a wide-area imaging sensor 4, an eyeball position determination unit 5, an eyeball imaging unit 6, A fixation point determination unit 7;

The light emission timing designation unit 2 and the infrared light emission unit 3 of each of the above components emit predetermined infrared light to the face 1 of the subject. With this infrared ray, the wide-area imaging sensor 4 images a wide area (the whole face and its surroundings) and simultaneously measures the distance. The eyeball position determination unit 5 is an eyeball position detection unit that uses the blinking of an eye. For this detection, a narrow-range imaging sensor captures a grayscale image of the eyeball. The gaze point determination unit 7 calculates the gaze point of the detected person from the grayscale image of the eyeball whose position is detected.

FIG. 2 is a block diagram showing an example of the internal configuration of the eyeball position determining unit 5 of the present embodiment. In FIG.
The eyeball position determination unit 5 according to the present embodiment includes a peak point detection unit 51 that detects a portion of a grayscale image exceeding a threshold, a grayscale image history storage unit 53 that records the history of the obtained grayscale image, and a grayscale image history. And a change amount peak detecting unit 52 that obtains a change amount from a newly obtained gray image, and an eyeball position calculating unit 54 that calculates the direction of the eyeball from the peak point of the gray image and the peak point of the change amount.

In a wide-area imaging sensor 4 for imaging a wide area (whole face and its surroundings) and measuring a distance at the same time, an infrared light emitting section 3
Measures the reflected light of the infrared light emitted periodically (at least 30 Hz or more), measures the grayscale image of the face 1 of the subject, and the time from when the light is emitted until when the peak point is received. hand over.

Inside the eyeball position determining section 5, the obtained grayscale image is converted into a peak point detecting section 51 and a variation peak detecting section 5.
Hand over to 2. The peak point detection unit 51 detects a position (peak coordinate) of a grayscale image having a value exceeding the threshold based on a threshold set in advance by a calibration operation when the apparatus is installed. FIG. 4 is a graph showing a pattern example of the intensity of light reflected from the eyeball. As FIG. 4 shows,
The face is characterized in that the reflected light (first Purkinje image) of the eyeball, particularly the cornea, is strong. Because of this,
Two peak points corresponding to the corneal portions of both eyes can be detected.

The change peak detector 52 compares the previous measured value of the gray image stored in the gray image history storage 53 with the latest measured value to obtain the amount of change in the gray image. From this amount of change, the position of the peak point (change amount peak coordinate) at which the value exceeds a preset threshold is obtained. When a blink of an eye is detected, this peak value is detected, so that it can be estimated that an eyeball is present in that direction.

Based on the peak coordinates and the change amount peak coordinates, the eyeball position calculation unit 54 calculates the peak coordinate positions if two peak coordinates are obtained, and the change amount peak coordinates (past values) if three or more points are obtained. The peak coordinate position that coincides with the detected coordinate within a certain period of time) is selected as the eyeball position when the number of points is equal to or less than one, and the latest change amount peak coordinate is selected. The distinction between the left eye and the right eye is determined from the positional relationship. Further, the distance to the eyeball is calculated based on the peak detection time. This is the distance to the eyeball on the side closer to the screen, and the other is regarded as the same distance or its vicinity (difference between eyes or less).

Based on the determined position of the eyeball, the eyeball imaging section 6 captures a precise gray-scale image of the eyeball. FIG.
3 is a block diagram illustrating a configuration example of an eyeball imaging unit 6. FIG.
In FIG. 3, the eyeball imaging unit 6 of the present embodiment includes two driving units 61, a tracking mirror 62, and a liquid crystal shutter 65 corresponding to the left and right eyes, and sets the two optical axes to one optical axis. , A plurality of mirrors and a half mirror, an infrared light emitting unit 63 and a narrow range imaging sensor 64.

The eyeball imaging unit 6 configured as described above directs the tracking mirror 62 to the eyeball of the person to be detected by the drive unit 61 based on the tracking mirror setting direction to obtain a precise shaded image. Further, the narrow-range imaging sensor 64 and the infrared light emitting unit 63 are simplified. Therefore, the right eye and the left eye are imaged in a time-division manner using the half mirror and the liquid crystal shutter 65. By using a half mirror, the infrared light emitting section 6
Since the light-emitting axis 3 and the light-receiving axis of the narrow-range imaging sensor 64 coincide with each other, adjustment such as alignment can be simplified.

According to the on-screen gazing point detection system of the above-described embodiment, the wide-area imaging sensor 4 captures an image of a wide area (the whole face and its surroundings), and at the same time measures the distance. The position of the eyeball is detected using blinking, and the eyeball imaging unit 6 captures a grayscale image of the eyeball. The gazing point determining unit 7 calculates the gazing point from the gray-scale image of the eyeball.

In particular, as shown in FIG. 2, the eyeball position determining unit 5 includes a peak point detecting unit 51 for detecting a portion of the grayscale image exceeding a threshold value, and a grayscale image history storage for recording the history of the obtained grayscale image. Unit 53, a change amount peak detection unit 52 that obtains a change amount from the history of the grayscale image and a newly obtained grayscale image, and an eyeball position calculation unit that calculates the direction of the eyeball from the peak point of the grayscale image and the peak point of the change amount 54.

When light is applied to the face, the reflected light of the eyeball, particularly the cornea, is characterized by being strong. Therefore, the eyeball position detecting means usually detects two peak points corresponding to the cornea of both eyes. Is done. However, at the moment of the blink of an eye, one loses his position. This is because the head may move when the eyes blink or pinch. However, since blinking of an eye is usually repeated several to several tens of times per minute by a person, the change in reflected light due to the blinking of the eye is detected by eye position detecting means, particularly a change amount peak detecting unit, so that the movement of the image is detected. The position of the eyeball can be directly estimated without performing tracking and prediction. This process is much simpler and faster than the process of detecting image motion.

The above embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications can be made without departing from the gist of the present invention.

[0034]

As is clear from the above description, the gaze point detection system on the screen according to the present invention applies the entire face of the subject to the detection of one face.
Then, a gray-scale image and a precise image of the eyeball of the face looking at a predetermined display are obtained, the position of the eyeball is determined by blinking of the eye, and the point of gaze of the display is determined from the precise image of the eyeball.

Therefore, by focusing on only the change in the reflected light using the blinking of the eye, it is possible to use the blinking of the eye performed several to several tens of times per minute with a smaller number of circuits to achieve high-speed
Simple eyeball position determination becomes possible. Further, by dividing the detection portion into two systems, a sensor covering a wide range can be used with a coarse accuracy, and an expensive precision sensor only needs to cover a minimum range. This makes it possible to keep both the size and the price of the system low.

[Brief description of the drawings]

FIG. 1 is a system configuration block diagram illustrating an embodiment of a gaze point detection system on a screen according to the present invention.

FIG. 2 is a block diagram illustrating a more detailed configuration example of an eyeball position detection unit in FIG. 1;

FIG. 3 is a block diagram illustrating a more detailed configuration example of an eyeball imaging unit in FIG. 1;

FIG. 4 is a diagram showing an example of a reflection intensity pattern from an eyeball.

FIG. 5 is a conceptual diagram for explaining a gazing point on a display screen and a required resolution.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Subject's face 2 Emission timing specification part 3 Infrared light emission part 4 Wide-area imaging sensor 5 Eyeball position determination part 6 Eyeball imaging part 7 Eye fixation point determination part 51 Peak point detection part 52 Change amount peak detection part 53 Gray scale image history storage Unit 54 eyeball position calculation unit 61 tracking mirror drive unit 62 tracking mirror 63 infrared light emitting unit 64 narrow range imaging sensor 65 liquid crystal shutter

Claims (6)

(57) [Claims] 1. An eye including an entire face of a subject as one image area
Means for wide-area imaging for obtaining a gray-scale image used for specifying the position of a sphere, and eyeball positioning for determining the position of the eyeball by blinking an eye
And a predetermined display based on the determined position of the eyeball.
Eyeball imaging means for obtaining a precise image of the eyeball of the face, and gazing point determination means for determining a gazing point of the display from the precise image of the eyeball , wherein the gazing point of the eyeball with respect to the display can be detected. A gaze point detection system on a screen characterized by the following. 2. An eyeball imaging means, comprising: a cornea of the eyeball.
Using the characteristic that the reflected light of the eye changes due to the blinking of the eye
The position of the eyepiece is determined.
Attention point detection system on the screen. 3. The eyeball imaging means comprises: two systems of tracking mirrors and liquid crystal shutters for independently tracking the left and right eyeballs; an optical system having two optical axes as one system; 3. The gaze point detection system on a screen according to claim 1, further comprising a narrow-range imaging sensor configured to perform imaging of the system, and detecting the blink of the eye. 4. 4. The system for detecting a gazing point on a screen further includes infrared light emitting means for emitting infrared light, and the eye gaze point of the eyeball is changed based on a change in the amount of reflected infrared light emitted from the infrared light emitting means from the face. gaze point detection system on the screen according to any one of claims 1 to 3, characterized in that for detecting the viewpoint. 5. An eyeball position determining means, a peak point detecting means for detecting a peak point of a light amount of the eyeball imaged by the eyeball imaging means, and a grayscale image history storage means for storing history data of the grayscale image. And a change amount peak detection unit that obtains a change amount from the history data of the gradation image and the newly obtained gradation image; and a tracking operation of the eyeball imaging unit from the detected peak point and the peak position of the change amount. outputs a control signal for controlling, and note on the screen according to any one of claims 1 to 4, characterized in that it is constructed and a eyeball position calculating means for calculating the position of the eye Viewpoint detection system. 6. The on-screen display according to claim 4 , wherein the amount of reflected light from the face for detecting the point of gaze of the eyeball is the amount of reflected light from the cornea of the pupil of the eye. Gaze point detection system.