CN106066696A - The sight tracing compensated based on projection mapping correction and point of fixation under natural light - Google Patents

Abstract

The invention discloses a line-of-sight tracking method based on projection mapping correction and fixation point compensation under natural light. The method firstly extracts the iris center of both eyes, the inner and outer corners of the eyes, and the features of the corners of the mouth, and then calculates the rectangle formed by the inner and outer corners of the eyes and the corners of the mouth. The projection mapping relationship between the rectangular information when the head is at the calibration position, and then the projection mapping correction is performed on the center position of the iris and the inner and outer corners of the eye to eliminate the impact of head movement; then, the corrected left and right eyes The center of the iris forms 4 vectors with the inner and outer corners of the left and right eyes respectively, combined with the polynomial mapping model to obtain the real-time fixation point, and finally compensates the fixation point through the support vector regression model. The invention provides a high-precision solution capable of reducing the impact of head movement for line-of-sight tracking under natural light.

Description

Gaze Tracking Method Based on Projection Mapping Correction and Gaze Compensation under Natural Light

technical field

The invention relates to the technical field of gaze tracking, in particular to a gaze tracking method based on projection mapping correction and gaze point compensation under natural light.

Background technique

Gaze tracking has changed the way humans interact with machines and equipment, has become the source of new technologies or system ideas, and has expanded the use of various information systems. It is an important field of human-computer interaction research.

Gaze tracking methods are mainly classified into contact methods and non-contact methods. The camera-based non-contact method is more user-friendly, has natural and direct advantages, and is currently the mainstream research direction of eye-tracking as a human-computer interaction method. In the non-contact eye-tracking method based on camera, the eye-tracking algorithm under natural light does not need other auxiliary light sources, which can be better popularized and applied. However, the main difficulty of this method lies in: (1) How to accurately extract eye movement feature information without the aid of auxiliary infrared light source and the image has illumination changes and low contrast; (2) How to find An eye movement vector that can represent eye movement and is robust; (3) Solve the problem that the eye movement vector changes with the head movement, which leads to the inability to accurately estimate the fixation point.

Contents of the invention

The invention discloses a line-of-sight tracking method based on projection mapping correction and fixation point compensation under natural light. Under natural light, by extracting iris center, inner and outer corners of eyes and mouth corners, establishing a fixation point on the screen based on iris and eye corner point information mapping model. This method can effectively eliminate the influence of free movement of the head on the line of sight estimation results. At the same time, only one monocular camera is required in terms of hardware requirements, which improves the accuracy and real-time performance of line of sight tracking under ordinary cameras.

The present invention is realized through the following technical solutions:

A line-of-sight tracking method based on projection mapping correction under natural light, which requires an ordinary camera without the assistance of an additional light source, and includes the following steps: (1) The camera collects images, performs face positioning and eye movement information extraction.

(2) Correction of eye movement information: Calculate the projection mapping matrix through the information of the corners of the eyes and the corners of the mouth, and correct the positions of the center of the iris and the inner and outer corners of the eyes.

(3) Preliminary fixation point estimation: Use the corrected iris center position and the inner and outer corners of the eyes to form a two-dimensional eye movement vector, and establish a mapping relationship between the two-dimensional eye movement vector and the fixation point on the screen. According to the real-time two-dimensional vector Calculates the real-time screen gaze point.

(4) Gaze point compensation: The support vector regression model is used for gaze point compensation, and the gaze point deviation caused by head movement is corrected, so as to obtain the final gaze point estimation result.

In the above method, the step (1) includes:

a. Use the face detection algorithm based on Adaboost to locate the face of the collected image, and then use the local binary feature regression method (Face Alignment via Regressing Local Binary Features) to determine the regions of interest in the inner and outer corners of the eyes and the corners of the mouth;

b. Carry out precise positioning according to the specific physiological shapes of different corner features, obtain the inner corners of the eyes and the corners of the mouth through the Fast corner detection and screening method, and use the curve fitting method to locate the outer corners of the eyes;

c. Determine the eye image according to the position of the inner and outer corners of the eye, then extract the gradient feature of the eye image, and locate the iris search starting point; secondly, start from the search starting point, search the edge of the iris through a sliding window, and finally locate according to the ellipse fitting method iris center. In the above method, the step (2) includes:

Take the position point at the set distance from the midpoint of the screen as the head calibration position, record the face image obtained when the head is at the head calibration position and stare at the screen frontally as the calibration image, and calculate according to the step (1) to locate The projection mapping matrix between the outer corners of the eyes and the corners of the mouth and the corresponding feature points in the calibration image is used to correct the real-time inner and outer corners of the eyes and the center of the iris.

In the above method, the step (3) includes:

a. According to the corrected left and right iris center positions in the step (2), respectively form 4 eye movement vectors with the corrected left and right eye inner and outer corner positions, and obtain the corrected two-dimensional eye movement vectors after superposition;

b. When the head is still at the calibration position, the eyes are fixed on the calibration point on the screen, the corrected two-dimensional eye movement vector is calculated, and the parameters of the polynomial mapping model are calculated according to the corresponding relationship between the vector and the calibration point; the correction is obtained in real time The final two-dimensional eye movement vector is combined with the polynomial mapping model to calculate the preliminary fixation point estimation result.

In the above method, the step (4) includes:

To train the support vector regression model, the input is the coordinates of the midpoint of the line between the inner corners of the eyes on the image and the offset of the same point in the calibration image, and the distance between the inner and outer corners of the left and right eyes on the image and the distance between the inner and outer corners of the eyes corresponding to the calibration image The ratio of the distance and the angle difference between the connection line between the inner corner points on the image and the connection line between the inner corner points of the calibration image; the output is the displacement deviation between the corresponding fixation point estimation result and the real calibration point; the support vector regression model is used for fixation point compensation, so as to obtain the final fixation point estimation result. Advantage and positive effect of the present invention are:

1. There is no effective reference point under natural light, and binocular data represents eye movement information. The present invention uses the iris centers of the left and right eyes and the inner and outer corners of the eyes to construct eye movement vectors, which can represent eye movement information more effectively. Using the sliding window search method to locate the edge of the iris, the accuracy of iris center location is improved.

2. The present invention corrects the position of feature points such as the center of the iris through the projection mapping correction method, and at the same time uses the support vector regression model to compensate the fixation point, which can effectively reduce the impact of head movement on eye movement characteristics, and the line of sight tracking method has better Anti-jamming ability of head movement.

3. This method requires less calculation, and the hardware only needs one camera.

Description of drawings

FIG. 1 is a schematic diagram of the arrangement of a display screen and a camera in an embodiment of the present invention.

Fig. 2 is a schematic flowchart of a gaze tracking method in an embodiment of the present invention.

Fig. 3 is a schematic diagram of a sliding window in an embodiment of the present invention.

Fig. 4a and Fig. 4b are distribution diagrams of two kinds of calibration points on the screen in the embodiment of the present invention.

detailed description

The specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

As shown in Fig. 1, the hardware configuration of the present invention requires an ordinary camera located directly above the center of the screen to capture face images in real time.

As shown in Figure 2, the specific implementation steps of the present invention are as follows:

Step 1. Collect images in real time and extract eye movement feature information;

Step 2, dynamic eye movement vector correction;

Step 3, constructing a polynomial mapping model;

Step 4: Train the gaze point compensation model to calculate the gaze point.

Wherein the concrete implementation of step 1 is:

1. Initial positioning of face feature points

The face image is obtained from the camera, and the shape regression method based on local binary features (FaceAlignment via Regressing Local Binary Features) is used for the initial positioning of the face feature points, and the initial positions of the eye contours and mouth corners are obtained. Secondly, on the basis of the preliminary positioning, the eye area and the mouth area are respectively obtained as the area of interest for positioning.

2. Eye movement feature extraction

The eye movement feature information includes eye inner corner feature, eye outer corner feature, mouth corner feature and iris center. The specific implementation steps are:

2.1 Positioning of the inner corner of the eye

First, determine the inner corner point interest region by the eye inner corner point candidate points obtained by initial positioning, and perform FAST corner point detection on this region to obtain corner point candidate points; the present invention thinks that the points where the candidate points gather are more likely to be eye corner points, according to each The number of candidate points around each candidate point is screened, and the inner corner of the eye is quickly and accurately located according to the relative position of the corner of the eye in the eye.

2.2 Positioning of the outer corner of the eye

Firstly, the eyes and the background area are separated by adaptive threshold segmentation, so as to extract the eye contour. Curve fitting is performed on the upper and lower eye contours, and the intersection point of the two curve fittings is calculated. The intersection point on the left is the outer corner of the eye.

2.3 Positioning of mouth corners

Firstly, the a component of the mouth area image in the Lab color space is extracted, and the lip area is segmented. Secondly, determine the leftmost and rightmost points of the lip area as the initial position of the corner of the mouth, and perform OTSU segmentation on the area, perform local corner detection on the segmented image, and filter out the isolated points, and consider the candidate point as the leftmost or The rightmost candidate point is the position of the corner of the mouth.

2.4 Iris Center Positioning

Under partial illumination, the contrast between the iris and the sclera is not very obvious, and it is difficult for traditional edge detection operators to accurately detect the edge of the iris. Secondly, the change of illumination also makes it difficult for the binarization method to completely segment the iris block. Therefore, the present invention proposes a sliding window-based iris edge search method, which can locate the iris edge under the condition of eye movement and posture changes, thereby obtaining iris center location.

a. Search starting point positioning

First, extract the red component image of the original eye image, and perform a morphological opening operation on the image to reduce the influence of reflection spots; second, calculate the gradient feature of the eye image after image preprocessing, and find the points through which the most gradient vectors pass, Set as the starting point of the search. Since this method is directly calculated by gradient features, good positioning results can be obtained in situations such as blurred images and severe eye deformation.

b. Iris edge search and iris center positioning

The sliding window is constructed as shown in Figure 3, which consists of two continuous rectangles of the same size, respectively representing the left window 1 and the right window 2, and representing the center of the sliding window 3. The sliding window starts from the center of the iris gradient and searches for the edges of the iris around. According to the connection between the corners of the eyes, the rotation angle θ ₀ of the eyes can be judged, so as to set the search range of the iris contour, and avoid the parts blocked by the upper and lower eyelids, so as to avoid contour fitting errors.

Calculate the average value of pixels within a window and define the energy of the sliding window:

Wherein, I _i and I _j (i=1,2,..N,j=1,2,..N) are pixel values of pixels in the left and right windows respectively, and N is the number of pixels in the window. θ is the current sliding window search direction. After setting the search direction, starting from the search starting point, the energy function curve of the search window is obtained. Find the largest peak of the function curve, i.e. the edge of the iris.

After the precise contour point set of the iris is searched by using the sliding window, the ellipse is fitted by the least square method, and the center of the ellipse is the center of the iris.

Wherein the specific implementation steps of step two are:

1. Eye movement vector construction based on binocular data

First, according to the left and right iris centers and the left and right inner and outer corners, a total of 4 eye movement vectors are formed, and the formed vector set is:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; l</span>}}\\ {\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; l</span>}}\\ {\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}}\\ {\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; r</span>}}\end{array}\right.$

Among them, v _il and v _ol represent the eye movement vectors formed by the iris center I _l of the left eye and the inner and outer corner positions C _il and C _ol of the left eye respectively, and v _ir and v _or represent the iris center I _r of the right eye and the inner and outer corners of the right eye respectively. The eye movement vector formed by the corner position C _ir and C _or . In the case of free head movement, the eyes move with the head movement, and the vector directly formed by the corner of the eye and the center of the iris also changes accordingly. Therefore, the present invention uses binocular data to construct a new type of eye movement vector to represent the gaze point, and eliminates the influence of different head postures by correcting the binocular data. The eye movement vector formed by binocular data is as follows:

$\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

2. Dynamic Eye Movement Vector Correction

In the process of line of sight estimation, the depth movement of the head will cause the deformation of the eye image, and the yaw, rotation, and pitch movements will cause the rotation and deformation of the eye image. The eye movement vector extracted from the eye image can directly fix the fixation point in the case of head movement. Mapping can cause large errors. Therefore, the present invention proposes a method based on projection mapping to dynamically correct eye movement vectors.

First, the coordinates of the four points in the image coordinate system of the eye corner point and mouth corner point feature points extracted from the face image captured in real time in step 1. The coordinates of the feature points captured during recording are (x ₁ , y ₁ ), and the coordinates of the feature points on the face image acquired when the head is at the head calibration position and looking at the screen frontally are (x ₂ , y ₂ ). The projection mapping relationship is as follows:

$\mathrm{<span\; class="notranslate">\; the\; s</span>}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]$

Among them, s is also a parameter, and H _p is a 3×3 matrix, recorded as:

${\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}\end{array}\right]$

Normalize the matrix H _p so that h ₃₃ =1, then:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; sx</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}\\ {\mathrm{<span\; class="notranslate">\; sy</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}\\ \mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right.$

Then the coordinates of the feature points captured in real time can be mapped and corrected by the following formula:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; sx</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}}{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; sy</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}}{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}\end{array}\right.$

The four feature points of the outer corners of the eyes and the corners of the mouth captured in real time and the corresponding feature points on the face image obtained when the head is at the head calibration position and looking at the screen are substituted into the above formula to form 8 linear equations. Therefore, by solving the equation, the projection transformation matrix H _p corresponding to the calibration position at the current head position and posture can be calculated. Then the position of the center of the iris imaged on the image after the head movement can also be corrected according to the projection transformation matrix _Hp :

$\mathrm{<span\; class="notranslate">\; the\; s</span>}\left[\begin{array}{c}\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; I</span>}}\\ \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; I</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]$

Calculate the corrected iris center position according to the above formula Among them, the symbol Represents the result of feature points or eye movement vector I corrected by projection mapping. Therefore, similarly, the positions of the inner and outer corners of the eye at the calibration position can be obtained, and the corrected eye movement vector can be expressed as:

$\left\{\begin{array}{c}\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; l</span>}}}<span\; class="notranslate">\; =</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; l</span>}}}\\ \stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; l</span>}}}<span\; class="notranslate">\; =</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; l</span>}}}\\ \stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; =</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}}}\\ \stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; =</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; r</span>}}}\end{array}\right.$

At the same time, since the corrected inner and outer corners of the eye are the same as the calibration position, that is, the distance between the inner and outer corners remains unchanged. Therefore, under the condition of keeping the information unchanged, the eye movement vector of the present invention can be simplified as:

$\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; l</span>}}}<span\; class="notranslate">\; +</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; )</span>$

That is, the corrected eye movement vector is obtained.

The specific implementation steps of Step 3 are as follows:

1. As shown in Figure 4a, set 9 points of 3×3 on the screen as calibration points. Second, select the mapping function with calibration as follows:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 7</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}\\ {\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}\end{array}\right.$

Where (v _x ,v _y ) is the corrected eye movement vector, (P _x ,P _y ) is the fixation point estimation result, a _i (i=0,1,...7), b _i (i=0 ,1,...6) totally 15 unknown parameters.

2. The user gazes at 9 calibration points respectively, extracts eye movement features in real time through step 1, and constructs and corrects eye movement vectors through step 2. The eye movement vectors extracted from each calibration point can respectively construct 2 equations, a total of 18 equations, and solve the parameters by the least square method.

The specific implementation steps of Step 4 are:

There is a deviation between the estimated result obtained by polynomial mapping of the corrected eye movement vector and the actual fixation point position, and the deviation is related to the head movement. The present invention uses support vector regression to compensate for this error. The input sample vector X=[v _x ,v _y ,M _x ,M _y ,R _l ,R _r ,θ _Δ ], the output vector is Y=[Y _x ,Y _y ]. Among them, v=(v _x ,v _y ) is the corrected eye movement vector, M=(M _x ,M _y ) is the offset between the coordinates of the middle point of the line segment between the inner corner points on the image and the same point as the calibration position, R _l and R _r are the ratios of the distance between the inner and outer corners of the left and right eyes on the image to the distance between the inner and outer corners of the eyes corresponding to the calibration image. θ _Δ is the angle difference between the connection line between the inner corner points on the image and the connection line between the inner corner points of the calibration image.

1. Training data construction.

As shown in Figure 4b, when the experimenter gazes at the designated calibration point on the screen, he moves his head while maintaining his gaze at the same point, collects feature information in real time according to step 1, and forms an input sample vector X. At the same time, according to Step 2, the eye movement vector is corrected, and then the polynomial mapping model obtained in Step 3 is used to obtain the estimated fixation point, and the displacement deviation from the real coordinate value is (Δx, Δy). Construct two training sets respectively: {(X ₁ ,Δx ₁ ),…,(X _i ,Δx _i ),…,(X _N ,Δx _N )},{(X ₁ ,Δy ₁ ),…,(X _i ,Δy _i ),…,(X _N ,Δy _N )}. Among them, X _i (i=1,2,...,N) are different sample vectors, (Δxi ,Δy _i )( _i =1,2,...,N) are the fixations corresponding to different sample vectors Point displacement deviation. N is the number of samples. The two training sets are trained separately.

2. Model parameter selection. The support vector regression model of the present invention adopts the RBF radial basis function, and can perform regression on complex relationships. Secondly, the grid search method is used to search for parameters. The main search parameters are the balance parameter C, the loss function parameter ε, and the kernel parameter γ.

3. Carry out support vector regression training according to the two training sets respectively to obtain the optimal regression model, and calculate the corresponding gaze point compensation offset (Y _x , Y _y ) through the real-time input vector X.

4. Gaze point estimation. The estimated fixation point calculated by the polynomial mapping equation in step 3 is (P _x , P _y ), which is superimposed with the offset (Y _x , Y _y ) calculated by the fixation point compensation model, and the final fixation point estimation result is :

(S _x ,S _y )=(P _x ,P _y )+(Y _x ,Y _y ).

Claims (5)

1. the sight tracing compensated based on projection mapping correction and point of fixation under natural light, it is characterised in that the method needs One common camera, it is not necessary to additional light source assists, and specifically comprises the steps of

(1) camera acquisition image, carries out Face detection and eye moves information retrieval；

(2) eye moves information correction: calculate projection mapping matrix, to iris center, eyes by eyes angle point and corners of the mouth dot information Inside and outside corner location is corrected；

(3) preliminary point of fixation is estimated: utilize calibrated after iris center, corner location constitutes two dimension eye and moves inside and outside eyes Vector, and set up the two dimension eye moving vector mapping relations to screen point of fixation, calculate screen in real time according to real-time bivector Curtain point of fixation；

(4) point of fixation compensates: uses support vector regression model to carry out point of fixation compensation, revises the point of fixation that head movement brings Deviation, thus obtain final point of fixation estimated result.

The sight tracing compensated based on projection mapping correction and point of fixation under natural light the most according to claim 1, It is characterized in that described step (1) including:

A. use Face datection algorithm based on Adaboost carries out Face detection to collection image, secondly uses based on office Portion's binary features homing method determines the inside and outside angle point of eyes and the area-of-interest of corners of the mouth point;

B. it is accurately positioned respectively, by Fast Corner Detection and screening according to the concrete physiology shape of different Corner Features Method obtains angle point and corners of the mouth point location in eyes, and uses the curve-fitting method location outer angle point of eyes；

C. determining eye image according to corner location inside and outside eyes, then extract the Gradient Features of eye image, location iris is searched Rope starting point；Secondly, from initial search point, by sliding window, iris edge is scanned for, intend finally according to ellipse Iris center, conjunction method location.

The sight tracing compensated based on projection mapping correction and point of fixation under natural light the most according to claim 2, It is characterized in that described step (2) including:

With the location point at distance screen midpoint setpoint distance for head calibration position, note head is in head calibration position and just The facial image obtained when screen is watched in face attentively is uncalibrated image, calculate the outer angle point of the eyes navigated to according to described step (1) and Projection mapping matrix between characteristic of correspondence point position in mouth corner location and uncalibrated image, utilizes this projection mapping matrix pair Inside and outside the eyes obtained in real time, corner location, iris center are corrected.

The sight tracing compensated based on projection mapping correction and point of fixation under natural light the most according to claim 3, It is characterized in that described step (3) including:

A. according to described step (2) correct after iris center, left and right, respectively with correction after left and right eyes inside and outside angle point Position constitutes 4 eye moving vectors, the two-dimentional eye moving vector after being corrected after superposition；

B. head is at calibration position during transfixion, and the fixed point on eye gaze screen, the two-dimentional eye after calculating correction moves Vector, according to the parameter of this vector Yu the corresponding relation evaluator mapping model of fixed point；Obtain two after correction in real time Dimension eye moving vector, in conjunction with polynomial map model, calculates preliminary point of fixation estimated result.

The sight tracing compensated based on projection mapping correction and point of fixation under natural light the most according to claim 4, It is characterized in that, it is characterised in that described step (4) including:

Training support vector regression model, inputs as angle point top-stitching section midpoint coordinate on image in eyes and uncalibrated image phase With the skew put, angle point spacing inside and outside the eyes that inside and outside right and left eyes, between angle point, the distance on image is corresponding with uncalibrated image Differential seat angle between line between angle point in line between angle point and uncalibrated image on ratio and image；It is output as the note of correspondence Offset deviation between viewpoint estimated result and true fixed point；Support vector regression model is utilized to carry out point of fixation compensation, thus Obtain final point of fixation estimated result.