TWI533234B - Control method based on eye's motion and apparatus using the same - Google Patents

Description

Eye movement control method and device thereof

The present invention relates to a control method and an apparatus therefor, and more particularly to a method based on an eye movement control method and an application thereof.

In the current technology, human eye tracking technology can be mainly divided into two types: invasive and non-invasive. Invasive human eye tracking technology is mainly to set a search coil in the eye or use an electrooculogram to sense the location of the human eye. Non-invasive human eye tracking technology is based primarily on vision based and can be divided into free-head or head-mount implementations.

With the development of technology, human eye tracking technology has been widely used in various fields such as neuroscience and computer science, and is commonly used in security systems (such as eye movement locks) or eye control computers. The human eye tracking technology can track the movement of the eyeball to obtain the line of sight of the eye gaze, and can realize the eye control function according to the control system or the computer device, or trigger the security system by gazing. Eye control computer, etc. Control of the device.

The invention provides a control method based on eye movement and a device thereof, which can enable a user to gaze and move a gaze point on a preset view pattern, and detect a movement trajectory of a user by detecting a line of sight. To determine whether to let the device perform the corresponding operation. Thereby, the device can be operated by the user's line of sight movement trajectory.

The present invention proposes a control method based on eye movements. This control method includes the following steps. Capture the image sequence through the image capture unit. The image sequence is analyzed to obtain eye image information of the user's eye region in each image of the image sequence. Based on the eye image information, the trajectory of the line of sight of the user looking at the view pattern is detected. It is determined whether the line of sight movement track conforms to the preset track relative to the view pattern to generate a judgment result. And, according to the judgment result, the corresponding operation is performed on a device.

In an embodiment of the invention, the view pattern includes a plurality of identification patterns. The preset trajectory is a geometric connection corresponding to the sequence formed by at least one of the identification patterns.

In an embodiment of the invention, the step of detecting the trajectory of the line of sight of the user watching the view pattern based on the eye image information comprises: analyzing the eye image information to determine whether the user is looking at the preview in the view pattern. Set the position, or look at the preset position for more than the preset time, and when the user looks at the preview in the view pattern When the position is set or the preset position is over the preset time, the trajectory of the line of sight of the user watching the view pattern is detected.

In an embodiment of the invention, the step of detecting the trajectory of the line of sight of the view pattern by the user based on the eye image information comprises: detecting the eye part in the eye area based on the eye image information The object performs an eye tracking operation according to the eye object, sequentially obtains a plurality of first positions of the eye object position in the image sequence, and maps the first position to a plurality of second according to the coordinate system corresponding to the view pattern. Position to obtain the trajectory of the line of sight of the user looking at the view pattern.

In an embodiment of the invention, the eye object includes a pupil and a first light reflecting point, and performing an eye tracking action according to the eye object to sequentially obtain the first position of the eye object corresponding to the image sequence. The steps include: positioning the pupil of the image sequence and the first reflective point to perform an eye tracking action, and sequentially obtaining the first position of the eye object corresponding to the image sequence.

In an embodiment of the present invention, the step of mapping the first position to the plurality of second positions according to the coordinate system corresponding to the view pattern to obtain the line of sight movement of the user in the view pattern includes: Calculating the distance parameter with the first reflecting point, calculating the angle parameter according to the vector of the first reflecting point relative to the pupil, and analyzing the distribution relationship between the distance parameter and the angle parameter, so as to convert the first position into the coordinate system corresponding to the view pattern The second position, and the second position is used as a line of sight movement trajectory of the user watching the view pattern.

In an embodiment of the invention, the eye object further includes a second reflective Point, and according to the coordinate system corresponding to the view pattern, the mapping conversion first position is the plurality of second positions, to obtain the user's gaze movement track of the view pattern, including: according to the pupil, the first reflective point And calculating a region parameter according to the second reflection point, calculating an angle parameter according to at least one of the first connection point of the first reflection point and the pupil and the second connection line of the second reflection point and the pupil, and analyzing the area parameter and the angle parameter And distributing the relationship to convert the first position to the plurality of second positions in the coordinate system corresponding to the view pattern, and to use the second position as a line of sight movement trajectory of the user in the gaze view pattern.

In an embodiment of the present invention, the step of performing the eye tracking action according to the eye object to sequentially obtain the first position of the eye object corresponding to the image sequence further comprises: obtaining the first position After one of them, the eye area is adjusted to detect the eye object again, and the eye tracking action is performed accordingly.

In an embodiment of the invention, the step of performing the corresponding operation on the device according to the determination result comprises: when the line of sight movement track conforms to the preset track, the device unlocks the locked state of the lock, and when the line of sight movement track does not conform to the preset When the track is in motion, the device maintains the locked state of the lock.

The present invention proposes a device for controlling based on eye movements, the device comprising an image capturing unit, a storage unit and a processing unit. The image capturing unit is used to capture an image sequence. The storage unit is configured to record a plurality of program modules, and the storage unit further includes a database. The processing unit is coupled to the image capturing unit and the storage unit to access and execute the module recorded in the storage unit. The program module includes an image analysis module, a line of sight detection module, a judgment module, and a control module. The image analysis module analyzes the image sequence, In this way, in each image of the image sequence, the eye image information of the user's eye area is obtained. The line-of-sight detection module is based on the eye image information to detect the trajectory of the line of sight of the user watching the view pattern. The determining module determines whether the line of sight movement track conforms to the preset track relative to the view pattern to generate a determination result. The control module performs a corresponding operation by the device according to the judgment result.

In an embodiment of the invention, the view pattern includes a plurality of identification patterns, and the preset trajectory is a geometric connection corresponding to the at least one of the identification patterns.

In an embodiment of the invention, the line-of-sight detecting module analyzes the eye image information to determine whether the user is looking at the preset position in the view pattern or watching the preset position exceed the preset time. And when the user looks at the preset position in the view pattern or looks at the preset position for more than the preset time, the line-of-sight detecting module detects the line of sight movement of the user watching the view pattern.

In an embodiment of the invention, the line of sight detection module includes an eye detection module, an eye tracking module, and a mapping conversion module. The eye detection module is based on the eye image information to detect eye objects in the eye region. The eye tracking module performs an eye tracking action according to the eye object to sequentially obtain a plurality of first positions of the eye object in the image sequence. The mapping conversion module maps the first position to a plurality of second positions according to the coordinate system corresponding to the view pattern, so as to obtain a line of sight movement track of the user watching the view pattern.

In an embodiment of the invention, the eye object includes a pupil and a first light reflecting point, and the eye tracking module determines the pupil of the image sequence and the first light reflecting point. Positioning to perform an eye tracking action, and sequentially obtaining the first position of the eye object corresponding to the sequence of images.

In an embodiment of the invention, the mapping conversion module calculates a distance parameter according to the pupil and the first reflection point, and calculates an angle parameter according to the vector of the first reflection point relative to the pupil, and analyzes the distance parameter and the angle parameter. And distributing the relationship to convert the first position to the plurality of second positions in the coordinate system corresponding to the view pattern, and to use the second position as a line of sight movement trajectory of the user in the gaze view pattern.

In an embodiment of the invention, the eye object further includes a second reflective point, and the mapping conversion module calculates the area parameter according to the pupil, the first reflective point and the second reflective point, and according to the first reflective point and the pupil The first connection line and at least one of the second reflection point and the second connection line of the pupil, calculate an angle parameter, and analyze a distribution relationship between the area parameter and the angle parameter to convert the first position into a coordinate system corresponding to the view pattern The plurality of second positions, and the second position as a line of sight movement trajectory of the user in the gaze view pattern.

In an embodiment of the invention, the eye tracking module further adjusts the eye region after obtaining one of the first positions, so that the eye detection module detects the eye object again, and the eye tracking is performed. The module performs the eye tracking action accordingly.

In an embodiment of the invention, when the line of sight movement trajectory conforms to the preset trajectory, the control module control device releases the locked state of the lock, and when the line of sight movement trajectory does not conform to the preset trajectory, the control module control device maintains the lock Lock status.

Based on the above, the eye movement based control proposed by the embodiment of the present invention The method and the device thereof can detect the trajectory of the line of sight of the user watching the view pattern, and perform corresponding operations by the device when determining that the line of sight movement trajectory conforms to the preset trajectory relative to the view pattern. Therefore, the embodiment of the present invention can be used as a trigger of the signal by the line of sight movement track, and can be widely applied to various fields such as a security system and an eye control computer.

In order to make the above features and advantages of the present invention more comprehensible, the following embodiments are described in detail with reference to the accompanying drawings.

100‧‧‧ device

110, 710, 810, 910‧‧‧ image capture unit

120, 720, 920‧ ‧ processing unit

130‧‧‧storage unit

131‧‧‧Image Analysis Module

132‧‧‧Sight Detection Module

1321‧‧‧Eye detection module

1322‧‧‧Eye tracking module

1323‧‧‧Map Conversion Module

133‧‧‧Judgement module

134‧‧‧Control Module

135‧‧‧Database

200‧‧‧ view pattern

210‧‧‧ identification pattern

220‧‧‧Geometry connection

230‧‧‧Preset trajectory

510‧‧‧Eye objects

512‧‧‧ pupil center point

514, 614, 616‧‧‧ Reflective points

602‧‧‧ vector

612‧‧‧瞳孔

630‧‧‧ punctuation

640‧‧‧ sector

650‧‧‧Rectangle

660‧‧‧ coordinate system

700‧‧‧Access Control System

730‧‧‧Locks

740‧‧‧Cue unit

750‧‧‧

760‧‧‧ Cover

800‧‧‧Handheld eye control eyepiece

820‧‧‧ display unit

830‧‧‧ housing

840‧‧‧Mirror

850‧‧‧Light source

860‧‧‧ safe

900‧‧‧ eye control device

H‧‧‧ horizontal line

IS‧‧·image sequence

L‧‧‧ distance

Θ‧‧‧ angle

1~18‧‧‧Area

S310~S350, S410~S480, S510~S540‧‧‧ steps

1 is a functional block diagram of a control device based on eye movements in accordance with an embodiment of the present invention.

2A and 2B are schematic diagrams of a method of controlling an eye movement according to an embodiment of the present invention.

3 is a flow chart showing the steps of a method based on eye movement control in accordance with an embodiment of the present invention.

4A is a flow chart showing the steps of a method based on eye movement control in accordance with an embodiment of the present invention.

FIG. 4B is a functional block diagram of a line-of-sight detecting module according to an embodiment of the invention.

4C is a schematic diagram of a method of controlling an eye movement based on an embodiment of the present invention.

5A is a flow chart showing the steps of eye tracking in accordance with an embodiment of the present invention.

FIG. 5B is a schematic diagram of eye tracking in accordance with an embodiment of the present invention.

6A-6G are schematic diagrams of coordinate conversion of a coordinate system in accordance with an embodiment of the present invention.

FIG. 7 is a schematic diagram of an access control system based on eye movement control according to an embodiment of the invention.

8A and FIG. 8B are schematic diagrams of a handheld eye control device according to an eye movement control according to an embodiment of the invention.

9 is a schematic diagram of a method of controlling an eye movement based on an embodiment of the present invention.

At present, the eye control technology mostly detects the user's line of sight, and can only control the security system or the eye control computer according to the position it gaze on the screen. The eye movement control method and the application device thereof according to the embodiments of the present invention can implement the eye control technology in a trajectory manner by using the line of sight movement track as a trigger of the signal. Thereby, the embodiment of the invention can provide a safer password input mode, and can also realize more convenient and more intuitive eye movement control. Hereinafter, the eye movement-based control method and the device using the same according to the embodiments of the present invention will be described in detail.

1 is a functional block diagram of an apparatus for controlling based on eye movements in accordance with an embodiment of the present invention. The apparatus 100 for controlling based on eye movements includes an image capturing unit 110, a processing unit 120, and a storage unit 130. Here, the The device 100 can be, for example, a safe, an access control system, or other type of qualification that requires verification of the user to determine whether to provide a security system/device for a specific user, or an electronic device such as a computer with an eye control function. The invention does not limit the specific type of device 100.

In this embodiment, the image capturing unit 110 can be used to capture an image sequence IS in a specific direction (depending on the arrangement position/angle of the image unit 110), that is, the image capturing unit 110 continuously captures multiple images, and Provided to the processing unit 120.

The processing unit 120 is coupled to the image capturing unit 110, and is, for example, a central processing unit (CPU), a graphics processing unit (GPU), or other programmable microprocessor (microprocessor). .

The storage unit 130 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, or the like. Or a combination of the above elements. The storage unit 130 includes a database 135 and records a plurality of program modules. The processing unit 120 is coupled to the storage unit 130 to access and execute the above program module. The program modules include an image analysis module 131, a line of sight detection module 132, a determination module 133, and a control module 134. The image analysis module 131 performs image processing and analysis on the image sequence IS captured by the image capturing unit 110, so that the eye image information of the eye region of the user is obtained in each image of the image sequence IS. In this way, the line-of-sight detecting module 132 can detect the eye movement of the user according to the obtained eye image information, and then the determining module 133 determines that the user is watching the view pattern. Whether the line of sight movement track conforms to a preset track. Here, the correspondence between the "view pattern", the "predetermined track", and the above will be described with reference to the examples of FIGS. 2A and 2B. As shown in FIG. 2A, the view pattern 200 includes a plurality of identification patterns 210. The identification pattern 210 may be arranged in a matrix of M×N. In this embodiment, both M and N are 4, but not limited thereto. It should be noted that the preset trajectory may be a geometric connection corresponding to at least one of the sequence of the identification patterns 210. For example, the geometric connection 220 illustrated in FIG. 2A may be composed of a plurality of identification patterns 210, and FIG. 2B illustrates the preset trajectory 230 stored in the database 135. It can be seen that the geometric connection 220 described above can correspond to the preset trajectory 230.

Thereby, the control module 134 determines whether to perform a corresponding operation on the device 100 according to the determination result of whether the line of sight movement track conforms to the preset track. For example, when the device 100 is a security device and the preset track is set to decode the lock in the security device, the control module 134 can switch the lock to the locked state or unlock according to whether the line of sight movement track conforms to the preset track. status.

The method of controlling eye movement based on the embodiment of the present invention will be described below. FIG. 3 is a flow chart showing the steps of a method based on eye movement control according to an embodiment of the invention. The method of the present embodiment is applicable to the electronic device 100 described above, and is combined with the components of the device 100 of FIG. 1 , and the concept of the eye motion based on the embodiment of the present invention is used to describe the control concept of the embodiment of the present invention. .

In step S310, the device 100 captures the image sequence IS through the image capturing unit 110. In step S320, the image analysis module 131 analyzes the image sequence IS, thereby obtaining the eye portion of the user's eye region in each image of the image sequence IS. Image information. In step S330, the line-of-sight detecting module 132 detects the trajectory of the line of sight of the user watching the view pattern based on the eye image information.

It should be noted that, in some embodiments, the visual line detection module 132 may first analyze the eye image information to determine whether the user is looking at the preset position in the view pattern, or whether the user is looking at the above. The preset position exceeds the preset time. When the user looks at the preset position in the view pattern, or the user looks at the specific preset position for more than the preset time, the line of sight detection module 132 correspondingly starts to detect the line of sight movement of the user in the gaze view pattern. . The manner in which the line-of-sight detecting module 132 determines when to start detecting the trajectory of the user's line of sight is only an exemplary description. The embodiment of the present invention can also provide a method for the user to externally trigger, for example, pressing a physical button or transmitting The invention is not limited to the detection action of the line of sight detection module 132 for the line of sight movement track.

The process of FIG. 3 is continued. In step S340, the determining module 133 determines whether the line of sight movement track conforms to the preset track relative to the view pattern to generate a determination result, and in step S350, the control module 134 can determine the result according to the determination. A corresponding operation is performed on the device 100. In detail, please refer to the examples of FIG. 2A and FIG. 2B again. When the gaze detection module 132 detects the trajectory of the line of sight of the user watching the view pattern, and the line of sight trajectory is the geometric connection 220 as shown in FIG. 2A, the determination module 133 can be based on the database 135. The preset trajectories recorded in the comparison are compared to determine whether the above-mentioned line of sight movement trajectory conforms to the preset trajectory recorded in the database 135. Thereby, the control module 134 can track the trajectory according to the line of sight. The determination result of the preset trajectory 230 in FIG. 2B, and corresponding operations are performed on the device 100.

Further, in the apparatus 100 of the present embodiment, it is possible to selectively provide a prompting unit. The prompting unit may be used to prompt whether the current device 100 conforms to the preset trajectory according to the user's line of sight movement, thereby performing a corresponding operation. In this case, the prompting unit can use the text display, the light signal display, the voice prompt or any other feasible prompting manner to enable the user to distinguish the current state of the device 100, which is not limited by the present invention.

Next, the detailed flow of the above-described control based on the eye movement will be further described with reference to the embodiment of FIGS. 4A to 4C. 4A is a flow chart of steps of a method for controlling eye movement based on an embodiment of the present invention, and FIG. 4B is a detailed block diagram of a line of sight detection module controlled based on eye movement according to an embodiment of the invention. 4C is a schematic view of an eye region in accordance with an embodiment of the present invention. Hereinafter, please refer to the step flow of FIG. 4A, and the components in FIG. 1 and the visual line detection module 132 of FIG. 4B are used for description. In this embodiment, the device 100 is a security device and the eye motion based control method can be used to control the lock in the device 100. It should be noted that the control method based on the eye movement can also be applied to any type of device, and the invention is not limited thereto.

In step S410, the device 100 captures the image sequence IS through the image capturing unit 110, and in step S420, the image analyzing module 131 analyzes the image sequence IS, thereby obtaining the user's image in each image of the image sequence IS. Eye image information in the eye area. The detailed description herein is similar to the embodiment of Fig. 3, so please refer to the foregoing.

As shown in FIG. 4B, the visual line detection module 132 of this embodiment includes an eye detection module 1321, an eye tracking module 1322, and a mapping conversion module 1323. Therefore, in step S430, the eye detection module 1321 detects the eye object in the eye region based on the eye image information. In step S440, the eye tracking module 1322 performs an eye tracking operation according to the eye object to sequentially obtain a plurality of first positions of the eye object position in the image sequence. In step S450, the mapping conversion module 1323 maps and converts the plurality of first positions into a plurality of second positions according to the coordinate system corresponding to the view pattern to obtain a line of sight movement trajectory of the user in the gaze view pattern. In other words, the embodiment can use the eye detection module 1321, the eye tracking module 1322, and the mapping conversion module 1323 to analyze the continuous motion of the image sequence corresponding to the user's eye object, and utilize the The conversion between the coordinate system corresponding to the image sequence and the view pattern respectively is obtained according to the trajectory of the line of sight of the user watching the view pattern.

After obtaining the trajectory of the user's line of sight, the determining module 133 can compare the line of sight trajectory with the preset trajectory of the view pattern, and the control module 134 performs the corresponding operation by the device 100 according to the determination result. The preset track can be set to the preset unlock code of the lock of the device 100. Therefore, in step S460, the determination module 133 determines whether the line of sight movement trajectory conforms to the preset trajectory with respect to the view pattern. When the lock of the device 100 is in the locked state, if the determination in step S460 is YES, the control module 134 releases the locked state of the lock (ie, the lock is switched to the unlocked state); otherwise, if the determination is no, the control mode is Group 134 will maintain the locked state of the lock, and can also send a warning tone or text through the prompting unit to remind the use By. Further, from another point of view, if the lock is in the unlocked state, the lock will remain in the unlocked state regardless of whether the result of the determination in step S460 is YES.

It is worth mentioning that, in an embodiment, the user can also make a set of continuous eye movements by using a line of sight corresponding to the continuous eye movements while the lock is unlocked. To define the default unlock password. Of course, the preset unlock password may also be predefined by the designer and stored in the database 135, and the invention is not limited thereto.

It is also worth mentioning that, in an embodiment, the device 100 further includes an identification module for identifying the biometric features of the user's eye object after detecting the eye object (step S430). Information to confirm the identity of the user. The above biometric information is, for example, iris or retina information. If the user's identity is correct, the identification module allows the device 100 to further perform the subsequent steps S440-S480. If not, the identification module stops or rejects the device 100 for subsequent eye motion recognition steps.

The above embodiment will be specifically described below with reference to the schematic diagram of Fig. 4C. In this embodiment, after the eye detection module 1321 searches for the eye object 410 in the eye region ER, the eye detection module 1321 can also perform image processing on the eye image information to obtain clear The image information indicating the boundary of the eye object 410 is indicated. Thereby, the eye tracking module 1322 can track the eye object 410 according to the image information, thereby obtaining a trajectory of the user's line of sight. For example, by adjusting the gain value and the offset value, the eye detection module 1321 can adjust the contrast of the eye image information and obtain a enhanced image. Then, the eye detection module 1321 can further perform processing such as de-noising, edge sharpening, binarization, and edge sharpening on the enhanced image to obtain an image of the eye object 410 as shown in FIG. 4C.

After obtaining the above-mentioned eye object 410, the eye tracking module 1322 performs an eye tracking operation according to the eye object 410 to obtain the precise position of the user's eye object 410 in each image sequence. In an embodiment, the eye tracking module 1322 can transmit the relative positional relationship between the pupil and the reflective spot in the eye object 410 to obtain precise positioning of the eye object 410. The above-mentioned light reflecting point is, for example, a purkinje image which can be formed by a fixed light source in the form of a light beam and reflected by the human eye. Depending on the position of the fixed light source, the reflective spot will also be present in a specific position of the human eye in the image, so it can be used as a reference feature for locating the eye object 410. Moreover, by using the reflective point as the positioning reference feature, the eye tracking module 1322 can be prevented from being subjected to excessive noise interference due to insufficient light source of the reference point when tracking the eye object. It should be noted that in the following embodiments, the pupil position is represented by the position of the center point of the pupil, and the number of the reflection points according to the positioning reference feature is different (for example, in the example of FIG. 5B, the eye object) 510 includes a reflective spot 514, while in the example of FIG. 6E, eye article 610 includes two reflective spots 614, 616), and eye tracking can be implemented in different forms.

The flow of the steps of eye tracking shown in FIG. 5 and FIG. 5A is first described. In step S510, the eye tracking module 1322 calculates a black and white contrast change in the eye region to obtain a pupil position in the image sequence. In step S520, the eye tracking module 1322 obtains the reference position of the reflection point. And, in step S530, the eye tracking mode The group 1322 adjusts and obtains the first position of the eye object position in the image sequence according to the relative relationship between the pupil position and the reference position of the reflection point.

In detail, FIG. 5B illustrates an example in which the eye object 510 includes a pupil (corresponding to the pupil center point 512) and a light reflecting point 514, and the eye object 510 corresponds to one of the images of the image sequence. The eye detection module 1321 can use the statistical information of the pupil to obtain the eye object 510. The above statistical information is, for example, the area, length, width, aspect ratio, shape, and black and white contrast statistics of the eyeball and/or the pupil. At least one of the types of data or a combination thereof. After the eye detection module 1321 obtains the eye object 510, the eye tracking module 1322 can calculate the black and white contrast change in the eye region. The use of the pupil and the eyeball in the human eye (here, the iris in the human eye structure, the non-white area such as black, blue or brown of the eye) corresponds to the extreme value of the black-and-white contrast change (generally, relative to the neighborhood) The eyeball, the pupil is a darker (or darker) image, and the eye tracking module 1322 can obtain the approximate location of the pupil center point 512. For the reflective spot 514, the eye tracking module 1322 can also obtain the reference position of the reflective spot 514 in a manner similar to that described above, and can also consider the condition of the white contrast region located at the nearest center of the pupil as a determination of whether it is reflective. The basis of point 514. Thereafter, the eye tracking module 1322 performs edge detection on the pupil and uses the reference position of the reflective point 514 as a reference point for edge detection to obtain an accurate pupil edge to estimate the position of the pupil center point 512, thereby correcting the pupil center point 512. Position for precise pupil positioning. The above edge detection method can utilize the pixel to detect the edge from white (corresponding to the eyeball) to black (corresponding to the pupil) or from the black to white. In addition, the eye tracking module 1322 can also be opposite to the reflective point 512. The reference position is fine-tuned to provide a more accurate reference position as a reference point for pupil positioning. The eye tracking module 1322 can use the center of the reflective point as a reference point for edge detection, and adjust the reference position of the reflective point according to a process similar to performing edge detection on the pupil. Thereby, by repeating and continuously performing the above-described flow, the eye tracking module 1322 can sequentially acquire the respective positions of the eye objects in the image sequence. These positions correspond to the trajectory of the line of sight of the user looking at the view pattern.

It is worth mentioning that the flowchart of FIG. 5A further includes step S540. In step S540, after the eye tracking module 1322 obtains the first position of the eye object 510, the range of the eye region is further adjusted so that the eye tracking module 1322 can execute accurately and efficiently. One eye tracking action. For example, in one embodiment, the eye tracking module 1322 can adjust the extent of the eye region based on the maximum range that the human eye can move. In addition, in some embodiments, the eye tracking module 1322 can also adaptively filter out noise in the eye object before performing the eye tracking operation to improve positioning and track the accuracy of the eye object 510. The manner of adjusting the eye area and filtering the noise can be appropriately designed according to requirements, and the present invention is not limited thereto.

Hereinafter, a specific embodiment of the line of sight movement trajectory of the user in the gaze view pattern is obtained in step S450 by using the schematic diagrams of FIG. 5B and FIG. 6A to FIG. 6D. In this embodiment, the eye tracking module 1322 can utilize the implementations of steps S510, S520 in the previous embodiment to obtain the position of the pupil center point 512 and the reflection point 514. Then, the mapping conversion module 1323 can utilize the pupil center point 512 to The distance L between the reflection points 514, and a reference line passing through the pupil center point 512 (i.e., the horizontal axis in Fig. 6A), and the angle θ defined by the reflection point 514 with respect to the vector 602 of the pupil center point 512, Therefore, through the relationship between the distance L and the angle θ, the pupil displacement position of the user in the image sequence (ie, corresponding to the first position described above) is transferred to the coordinate system of the view pattern, and the corresponding line of sight movement is obtained. Track. The parameters of the distance L and the angle θ are applied to perform the coordinate conversion, and the group correspondence method and the interpolation correspondence method are provided as an explanation.

First, the group correspondence method will be described with reference to the embodiment of Figs. 6B and 6C. In this embodiment, the group correspondence method may include a training phase and an application phase, wherein the mapping conversion module 1323 passes through the training phase to obtain the coefficients required for the coordinate conversion, and may according to the current eye object in the application phase. Convert to the coordinate system of the view pattern. In detail, during the training phase, the mapping conversion module 1323 can display one of the regions from the regions 1 to 16 of FIG. 6B for viewing by the user each time (in FIG. 6B, the region 1 is displayed to In the case of the user viewing, the mapping conversion module 1323 can locate and obtain the distance and angle of the pupil center and the reflection point corresponding to each group of regions, so as to draw the distance-angle distribution of each group region of FIG. 6B. The distance-angle diagram shown in Figure 6C. Therefore, in the application phase of the group correspondence method, the mapping conversion module 1323 can transmit the eye object currently acquired by the image capturing unit 110, and analyze the distance L corresponding to the pupil and the reflective point in the eye object. Angle θ and obtain a coordinate point in the distance-angle diagram, such as coordinate point 630 of Figure 6C. Then, the mapping conversion module 1323 can find the closest region to the coordinate point 630 by using the minimum target function to locate the current eye object to the coordinate system of the view pattern, thereby completing the coordinate conversion. The formula (1) of the above minimum objective function is as follows: Min Obj.=W ₁ |Dis_t-Dis_i|+W ₂ |Angle_t-Angle_i|, (i=1~16)......(1)

Among them, Dis_i and Angle_i are the target distance value and the angle value, and Dis_t and Angle_t are the distance value and angle value corresponding to the current eye object. The above distance value and angle value can be obtained by calculating the x-axis and y-axis coordinates of the coordinate point, and W ₁ and W ₂ are the distribution weights. It can be seen that the number of regions in FIG. 6B will determine the accuracy of the mapping conversion module 1323 to convert coordinates in a grouping correspondence method. Therefore, when the number of regions included in the packet correspondence method of this embodiment is larger, the mapping conversion module 1323 can obtain a more accurate conversion coordinate correspondingly.

On the other hand, the embodiment of Fig. 6D is used to explain the interpolation correspondence method. In this embodiment, the mapping conversion module 1323 can perform an un-warping operation to convert the fan shape 640 drawn by the coordinate points in the distance-angle diagram into a relatively regular rectangle 650, and the rectangle 650 The location corresponds to the coordinate system 660 in which the view pattern is located. This embodiment may also include a training phase and an application phase, and the implementation of the training phase may also be similar to the previous embodiment. The difference is that the mapping conversion module 1323 of the embodiment further utilizes the above-mentioned twisting operation to obtain a normalized distribution map, and can perform motion correction on the normalized distribution map by an affine transform technique (moving calibration) ). The formulas (2) and (3) of the normalized distance-angle distribution map obtained by the twisting operation are as follows:

The twisting operation converts the untwisted sampling point (X, Y) = {(x ₁ , y ₁ )...(x _n , y _n )} to the target point (X _T , Y _T )={( x _T1 , y _T1 ) (x _Tn , y _Tn )}, where X and Y represent the above-described distance L and angle θ, n is the number of samples, and a ₀ ~ a ₅ , b ₀ ~ b ₅ are Twist the conversion factor. Thereby, the mapping conversion module 1323 can obtain the optimal solutions of the coefficients a ₀ to a ₅ and b ₀ to b ₅ by inverse matrix operations. In this way, the mapping conversion module 1323 can use the above-mentioned coefficients to perform the twisting and transformation of the currently unknown coordinate points.

On the other hand, the formula (4) of the map conversion module 1323 for performing motion correction using the affine technique is as follows:

Among them, x', y' are the new coordinates after movement correction, and a~f is the affine conversion coefficient. Therefore, the mapping conversion module 1323 can also calculate through the inverse matrix, or by inputting three pairs of coordinate points of any three corners of the uncorrected front display unit and three pairs of coordinate points of the corrected three corners. The above affine conversion coefficients a~f are obtained. Thereby, the mapping conversion module 1323 can perform motion correction on the currently unknown coordinate points to eliminate the influence of image scaling, translation, rotation and other factors on the coordinate conversion. With respect to the embodiment of the packet correspondence method proposed in FIGS. 6B and 6C, the mapping conversion module 1323 can perform coordinate conversion with a small number of samples using the interpolation correspondence method.

It should be noted that, in an embodiment, when the eye tracking module 1322 is in order After obtaining the plurality of first positions of the eye objects in the image sequence, the mapping conversion module 1323 converts the first positions into a continuous motion form into a line of sight movement trajectory of the user watching the view pattern. In another embodiment, the mapping conversion module 1323 can synchronize with the actions of the eye tracking module 1322 to obtain the first positions of the eye objects, so as to coordinately convert the first position to the first position. Obtain the user's line of sight movement. The method for obtaining the line of sight movement track can be determined according to the design requirement of the embodiment, and the present invention is not limited thereto.

On the other hand, the example of FIG. 6E illustrates the case where the eye object 610 includes the pupil 612 and the two reflective spots 614, 616. Similar to the previous embodiment, the eye tracking module 1322 can utilize the black and white contrast change of the image (eg, binarizing the image of the eye object 610) to find the pupil 612 and the reflective spot 614 in the eye object 610. , 616, and obtain their individual center points to indicate the respective positions of the pupil 612 and the reflective spots 614, 616. The mapping conversion module 1323 can calculate the triangular area formed when the pupil 612 and the reflective spots 614, 616 are not collinear, and can adaptively correct errors that may be caused by the movement of the user's head. In detail, for the area algorithm considering the above error, the calculated area can be further normalized by a normalization factor after calculating the triangular area formed by the pupil 612 and the reflection points 614, 616. . For example, the normalization factor may be the square of the distance between the reflection points 614 and 616 divided by 2, and the mapping conversion module 1323 divides the calculated triangle area by the normalization factor, thereby performing error. Correction to obtain the area parameters.

On the other hand, the mapping conversion module 1323 is provided by the pupil 612 and the reflective point. 614, 616 to calculate the angle parameter. In one embodiment, the mapping conversion module 1323 can transmit a transverse line H transversely passing through a center point of the pupil 612 and a line connecting the pupil 612 and the reflective point 614 to obtain a first angle α 1 between the two lines. In addition, the mapping conversion module 1323 can also obtain the second angle α 2 between the two lines by the horizontal line H and the connection of the pupil 612 and the reflection point 616. In other words, the angle parameter may be the first angle α 1 , the second angle α 2 , or may be the difference between the second angle α 2 and the first angle α 1 (ie α 2−α 1 to pupil 612 is the center, the line connecting the pupil 612 with the light reflecting point 614, and the angle between the line connecting the pupil 612 and the reflecting point 616).

Therefore, the mapping conversion module 1323 can perform correction on the first-time use through the correction areas 1 to 18 or the correction points as shown in FIG. 6F to obtain the corrected coordinates on the angle-area plan. Similar to the training phase of the foregoing embodiment, the distribution of the area and angle formed by the user corresponding to the pupil 612 and the reflective spots 614, 616 of each group of regions is shown in FIG. 6G, and the mapping conversion module 1323 described above. The original coordinates are mapped to coordinates corresponding to the correction areas 1-18 to obtain the coefficients required for coordinate conversion (eg, affine transformation methods). In this way, after entering the application phase, the mapping conversion module 1323 can complete the coordinate conversion for the pupil 612 and the reflective points 614, 616 by the coefficients used for coordinate conversion.

The embodiment of FIGS. 7 to 9 is a practical application example of the above-described eye movement-based control method. First, FIG. 7 is taken as an example for explanation. FIG. 7 is a schematic diagram of an access control system based on eye movement control according to an embodiment of the present invention.

The access control system 700 includes an image capturing unit 710, a processing unit 720, a lock 730, a prompting unit 740, and a door body 750. The lock 730 is disposed on the door body 750. And used to control the opening or closing of the door body 750. In the present embodiment, the prompting unit 740 is, for example, a light number display device, and indicates that the current configuration state of the lock 730 is a locked state or an unlocked state by a light signal, but the embodiment of the present invention does not limit the form of the prompting unit.

In this embodiment, the image capturing unit 710 is disposed on the door body 750, and the image capturing unit 710 is covered by a cover 760, and an image capturing area is exposed only in the image capturing direction of the image capturing unit 710. Therefore, the user can align the eye area with the image capturing area, so that the image capturing unit 710 can capture the image sequence for the user's eye area to avoid peers, thereby improving the safety of the access control system 700. . Herein, the arrangement of the cover 760 is also the designer can choose whether to join according to the design requirements, and the invention is not limited thereto.

Based on the architecture of the access control system 700, the processing unit 720 can process the image sequence captured by the image capturing unit 710, and can read the program module recorded in the storage unit as described in the previous embodiment, according to FIG. 2 to FIG. The step flow of the 6D embodiment detects the eye movements made by the user, and determines whether the user's line of sight movement trajectory conforms to the preset trajectory. Therefore, the processing unit 720 can determine whether to issue the corresponding control signal to control the unlocking state of the lock 730 according to the determination result, so that the user can open the door body 750 to enter the door body 750 when the line of sight movement track conforms to the preset track. After the area.

8A and 8B are diagrams showing another example of the present invention, and illustrating a handheld eye control device 800 controlled based on eye movements in accordance with an embodiment of the present invention. The handheld eye control device 800 includes a processing unit and an image capturing unit 810. The display unit 820, the housing 830, the mirror 840, and the light source 850 are connected to a security device (such as the safe 860) to authenticate the user. In this embodiment, the handheld eye control device 800 can transmit through a wireless transmission interface (for example, Short Distance Wireless Communication, Radio Frequency Identification (RFID), Bluetooth, or Wi-Fi, etc. is connected to the safe 860. The image capturing unit 810 and the light source 850 are both disposed in the housing 830 and adjacent to the position of the window of the handheld eye control device 800. The light source 850 can be turned on when the image capturing unit 810 captures the image of the user's eye, or is turned on when the user approaches, thereby providing sufficient brightness, which is not limited by the present invention. The display unit 820 is disposed within the housing 830 and can be used to display a screen with information about the password input. The mirror 840 can reflect the image displayed by the display unit 820 to the window of the handheld eye control device 800, so that the user can view the screen content displayed by the display unit 820 through the window. The password input information is, for example, a "please enter a password" or the like, which prompts the user to start the eye movement input, or indicates whether the user password outputs the correct text, and the present invention is not limited thereto.

Therefore, when the user wants to use the safe 860, the user can carry the handheld eye-contacting device 800 against the eyes by hand, and make an eye movement. Similarly, based on the architecture of the handheld eye control device 800, the image capturing unit 810 captures the user's eyes and captures the image sequence, and the processing unit can read the program recorded in the storage unit as described in the previous embodiment. Module, and according to the step flow of the embodiment of FIG. 2 to FIG. 6D, the eye movements made by the user are detected, and The line of sight movement trajectory of the view pattern displayed by the user gaze display unit 820 is obtained, and the line of sight movement trajectory is used as an input password of the safe 860, and is compared with a preset security code in the form of a track. When the security password above matches the input password, the processing unit may generate verification success information and transmit it to the safe 860 to control the opening of its lock. Other details of the operation of the embodiment can be similar to the above embodiment, so please refer to the foregoing.

9 is a schematic diagram of an eye control device 900 that is controlled based on eye movements according to still another embodiment of the present invention. The eye control device 900 includes an image capturing unit 910 and a processing unit. In this embodiment, the eye tracking module provides another implementation for tracking eye objects. In detail, the image capturing unit 910 may be provided with a lens that rotatably adjusts the direction and angle to adjust the lens to a state of looking up to the user's face. For example, the lens of the image capturing unit 910 can face the face of the user at an angle of 45 degrees to enhance the recognition of the user's nostril image, and help to obtain the nostril image, thereby serving as a reference for defining the position of the eye object. feature.

In detail, the processing unit can pass through the characteristics of the nostril region which is significantly darker relative to other regions, and utilize the intersection of the longest horizontal axis and the longest longitudinal axis of the nostril region as the center point of the nostrils. Next, the eye tracking module calculates the spacing between the center points of the two nostrils to determine the coordinates of the points (s ₁ , t ₁ ), and then calculates the coordinates of the reference point based on the spacing and the starting point coordinates (s ₂ , t _{2 ).} Wherein s ₂ = s ₁ + k ₁ × D, t ₂ = t ₁ + k ₂ × D, k ₁ = 1.6 to 1.8, k ₂ = 1.6 to 1.8, and preferably k ₁ = k ₂ . The values of k ₁ and k ₂ described above can be obtained according to statistical results, and the reference point coordinates (s ₂ , t ₂ ) obtained through the above relationship can just fall on an eye object close to the facial image. Center point. In this way, through the nostril features and eye statistics, the eye tracking module can accurately locate the user's eye objects.

Thereby, based on the hardware architecture of the above-described eye control device 900, the processing unit can read the program modules recorded in the storage unit as described in the previous embodiment in a similar manner, and according to the embodiment of FIG. 2 to FIG. 6D The step process is to detect the eye movements made by the user, and correspondingly obtain the trajectory of the line of sight of the user watching the view pattern to determine whether the preset trajectory is met, so as to be based on the user by the eye control device 900. The line of sight moves the trajectory to perform corresponding operations, such as detecting the user's line of sight movement track to unlock the computer screen, or operating the cursor of the computer window.

In summary, the eye movement control method and the device using the same according to the embodiments of the present invention can detect the trajectory of the line of sight of the user watching the view pattern, and determine that the line of sight movement trajectory is in accordance with the view pattern. When the preset trajectory is performed, the corresponding operation is performed by the device. Therefore, the embodiment of the present invention can be used as a trigger of the signal by the line of sight movement track, and can be widely applied to various fields such as a security system and an eye control computer.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S310~S350‧‧‧Steps

Claims (12)

An eye movement-based control method includes: capturing an image sequence through an image capturing unit; analyzing the image sequence to obtain one of an eye region of a user in each image of the image sequence Eye image information; based on the eye image information, to detect a line of sight movement of the user while looking at a view pattern; determining whether the line of sight movement track conforms to a predetermined track relative to the view pattern to generate a Determining a result; and performing a corresponding operation on a device according to the determination result, wherein the step of detecting the movement track of the line of sight of the user in the view pattern based on the eye image information comprises: based on the eye portion Image information for detecting an eye object in the eye region; performing an eye tracking action according to the eye object to sequentially obtain a plurality of first positions of the eye object in the image sequence; According to the labeling system corresponding to the view pattern, mapping converts the first positions into a plurality of second positions to obtain the user viewing the view pattern The line of sight movement track, wherein the eye object comprises a pupil and a first light reflecting point, and the eye tracking action is performed according to the eye object, to sequentially obtain the first part of the eye object corresponding to the image sequence The steps of a location include: Positioning the pupil of the image sequence and the first light reflecting point to perform the eye tracking action; and sequentially obtaining the first positions of the eye object corresponding to the image sequence, wherein the corresponding position according to the view pattern The coordinate system, mapping the first positions to the second positions to obtain the line of sight movement trajectory of the user looking at the view pattern comprises: calculating a distance according to the pupil and the first reflective point An angle parameter is calculated according to a vector of the first reflection point relative to the pupil; and the distance parameter and a distribution relationship of the angle parameter are analyzed to convert the first position to the coordinate system corresponding to the view pattern The second positions of the second position; and the second positions are used as the line of sight movement trajectory of the user looking at the view pattern. The control method of claim 1, wherein the view pattern comprises a plurality of identification patterns, and the preset tracks are geometric lines corresponding to the sequence of the at least one of the identification patterns. The control method of claim 1, wherein the step of detecting the line of sight movement of the view pattern by the user based on the eye image information comprises: analyzing the eye image information to determine the Whether the user is looking at the view pattern a preset position, or watching the preset position for more than a preset time; and detecting when the user looks at the preset position in the view pattern or watches the preset position exceed the preset time The line of sight movement of the user looking at the view pattern is measured. The control method of claim 1, wherein the eye object further comprises a second reflective point, and according to the coordinate system corresponding to the view pattern, mapping and converting the first positions to the second positions The step of obtaining the line of sight movement of the user in the view pattern includes: calculating an area parameter according to the pupil, the first reflection point, and the second reflection point; and the pupil according to the first reflection point and the pupil Calculating an angle parameter of a first connection line and at least one of the second reflection point and a second connection line of the pupil; analyzing the area parameter and a distribution relationship of the angle parameter to convert the first Positions are the second positions in the coordinate system corresponding to the view pattern; and the second positions are used as the line of sight movement trajectory of the user looking at the view pattern. The control method of claim 1, wherein the step of performing the eye tracking operation according to the eye object to sequentially obtain the first positions of the eye object corresponding to the image sequence further comprises: After obtaining one of the first positions, the eye area is adjusted to detect the eye item again, and the eye tracking action is performed accordingly. For example, the control method described in claim 1 of the patent scope, wherein the judgment is based on the judgment As a result, the step of performing the corresponding operation on the device includes: when the line of sight movement track conforms to the preset track, the device releases a locked state of a lock; and when the line of sight moving track does not conform to the preset track, The device maintains the locked state of the lock. An apparatus for controlling based on eye movements, comprising: an image capturing unit for capturing an image sequence; a storage unit for recording a plurality of program modules, and including a database; and a processing unit coupled to the image capturing a unit and the storage unit for accessing and executing the program module recorded in the storage unit, the program module comprising: an image analysis module, analyzing the image sequence, thereby each of the image sequences In the image, an eye image information of one of the user's eye regions is obtained; a line of sight detection module is configured to detect a line of sight movement of the user while looking at a view pattern based on the eye image information; a determining module, determining whether the line of sight movement track conforms to a predetermined track relative to the view pattern to generate a determination result; and a control module to perform a corresponding operation by the device according to the determination result, wherein The line of sight detection module includes: an eye detection module that detects an eye object in the eye area based on the eye image information; An eye tracking module performs an eye tracking operation according to the eye object to sequentially obtain a plurality of first positions of the eye object in the image sequence; and a mapping conversion module according to the view a mapping system corresponding to the pattern, the mapping converts the first positions into a plurality of second positions to obtain the line of sight movement track of the user looking at the view pattern, wherein the eye object comprises a pupil and a first reflection a point, and the eye tracking module positions the pupil of the image sequence and the first light reflecting point to perform the eye tracking action, and sequentially obtain the first of the eye object corresponding to the image sequence a position, wherein the mapping conversion module calculates a distance parameter according to the pupil and the first reflection point, and calculates an angle parameter according to a vector of the first reflection point relative to the pupil, and analyzes the distance parameter and the a distribution relationship of the angle parameters to convert the first positions to the second positions in the coordinate system corresponding to the view pattern, and the second positions as the use In this view looking at the visual line movement track pattern. The apparatus of claim 7, wherein the view pattern comprises a plurality of identification patterns, and the preset tracks are geometric lines corresponding to the sequence of the at least one of the identification patterns. The device of claim 7, wherein the line of sight detection module analyzes the eye image information to determine whether the user is looking at a preset position in the view pattern or watching the preset position exceeds a preset time, and when the use When the gaze is in the preset position in the view pattern or the gaze is over the preset time, the gaze detection module detects the trajectory of the line of sight of the user watching the view pattern. The device of claim 7, wherein the eye object further comprises a second reflective point, and the mapping conversion module calculates an area according to the pupil, the first reflective point and the second reflective point. And calculating an angle parameter according to the first line connecting the first reflection point and the first line of the pupil and the second line of the second reflection point and the second line of the pupil, and analyzing the area parameter and the a distribution relationship of the angle parameters to convert the first positions to the second positions in the coordinate system corresponding to the view pattern, and the second positions as the line of sight of the user looking at the view pattern Move the track. The apparatus of claim 7, wherein the eye tracking module further adjusts the eye area after obtaining one of the first positions to be detected by the eye detection module again The eye object, and the eye tracking module performs the eye tracking action accordingly. The device of claim 7, wherein the control module controls the device to release a locked state of the lock when the line of sight movement track conforms to the preset track, and when the line of sight movement track does not meet the preset When the trajectory is set, the control module controls the device to maintain the locked state of the lock.