TW201516892A - Method, apparatus and system for monitoring - Google Patents

Abstract

A method, an apparatus and a system for monitoring are provided. A user interface having a plurality of seat block corresponding to a plurality of location is displayed on a display unit. An image sequence is obtained from each image capturing unit where is disposed on each location in a space. A face recognition algorithm is executed for the image sequence so as to determine whether the image sequence has a human image of a user. If the image sequence has the human image, an eye tracking algorithm is executed for the image sequence. And a corresponding marking action is executed for each seat block based on whether the user gazes a specific direction.

Description

Monitoring method, device and system

The present invention relates to a monitoring mechanism, and in particular to an eye tracking based monitoring method, apparatus and system.

At present, eye tracking technology can be mainly divided into two types: invasive and non-invasive. Invasive eye tracking technology is mainly to set up a search Coil in the eye or use an electrooculogram. Non-invasive eye tracking technology can be distinguished as a free-head or head-mount eye tracking technology.

With the development of technology, eye tracking technology has been widely used in various fields, such as neuroscience, psychology, industrial engineering, human factors engineering, marketing advertising, computer science and so on. For example, US Patent No. US 2010/0092929 proposes a cognitive and language evaluation system that utilizes eye tracking technology, which utilizes eye tracking technology to obtain the eyeball gaze position and duration of the patient, thereby testing language understanding and work. Memory, ability to assign attention, and the ability to initiate a semantic connection.

The invention provides a monitoring method, device and system, which are based on an eye tracking algorithm to assist in monitoring the degree of concentration of a user.

The monitoring method of the present invention is for use in a monitoring device. Here, the user interface is displayed in the display unit, wherein the user interface includes a plurality of seat blocks, and the seat blocks respectively correspond to a plurality of positions in the same space as the monitoring device, and each of the positions is provided with an image capturing unit. . Obtaining an image sequence from respective image capturing units at these positions, performing a face recognition algorithm on the image sequence to determine whether the image sequence has a user's portrait; if the image sequence has a portrait, performing an eye tracking algorithm on the image sequence; And performing a corresponding marking action on each seat block based on whether the user is looking at the specified direction.

The monitoring system of the present invention includes a plurality of image capturing units and monitoring devices disposed in the same space. The image capturing units are respectively disposed at a plurality of positions in the space. The monitoring device includes a display unit, a communication unit, a storage unit, and a processing unit. The display unit is configured to display a user interface, wherein the user interface includes a plurality of seat blocks, and the seat blocks respectively correspond to the positions. The communication unit is used to obtain an image sequence from each image capturing unit. The storage unit is used to store the image sequence. The processing unit is coupled to the display unit and the communication unit, and is configured to drive the monitoring module. The monitoring module performs a face recognition algorithm on the image sequence to determine whether the image sequence has a user portrait; if the image sequence has a portrait, the monitoring module performs an eye tracking algorithm on the image sequence; and the monitoring module A corresponding marking action is performed on each seat block based on whether the user is looking at the specified direction.

The monitoring device of the present invention comprises a display unit, a communication unit, a storage unit and a processing unit. The display unit is configured to display a user interface, wherein the user interface comprises a plurality of seat blocks, and the seat blocks respectively correspond to a plurality of positions in the same space as the monitoring device, and each position is provided with an image capturing unit. The communication unit is used to obtain an image sequence from each image capturing unit. The storage unit is used to store the image sequence. The processing unit is coupled to the display unit and the communication unit, and is configured to drive the monitoring module. The monitoring module performs a face recognition algorithm on the image sequence to determine whether the image sequence has a user portrait; if the image sequence has a portrait, the monitoring module performs an eye tracking algorithm on the image sequence; and the monitoring module A corresponding marking action is performed on each seat block based on whether the user is looking at the specified direction.

In an embodiment of the invention, the monitoring module further includes: a face recognition module for performing a face recognition algorithm on the image sequence; and an eye movement tracking module for performing eye movement on the image sequence A tracking algorithm; a marking module for performing a corresponding marking action on each seat block.

In an embodiment of the invention, the monitoring module further includes: a state recognition module, configured to determine, according to the result of the eye tracking module, that the user is currently in a state of concentration, dozing or not focusing; A detection module for performing a closed-eye detection algorithm on an image sequence. Wherein, when the eye tracking module determines that the user is not looking at the specified direction, the state recognition module determines that the user is currently in a focused state. When the eye tracking module determines that the user is not looking at the specified direction, and the closed eye detection module detects that the user is in the closed state, the state recognition module determines that the user is currently in a doze state and is detected to be in use. State in the non-closed state The recognition module determines that the user is currently in an unfocused state.

In an embodiment of the invention, the marking module displays a fourth mark in the corresponding seat block when the face recognition module determines that there is no portrait in the image sequence.

In an embodiment of the invention, the monitoring module further includes: a oscillating head detecting module, which determines whether the user turns the head based on the information of the nostril position, thereby obtaining facial swing information, wherein the face recognition mode When the group obtains the face region, the face recognition module detects the nostril region of the face region to obtain the information of the nostril position. Wherein, when the oscillating head detecting module determines that the user faces the specified direction based on the face swaying information, the eye tracking algorithm is performed on the image sequence through the eye tracking module.

Based on the above, the eye movement tracking algorithm is used to determine the degree of concentration of the user, and the corresponding seat block in the user interface presents a corresponding mark, thereby being more intuitively transmitted through the monitoring device (eg, the teacher device). Know the attendance and concentration of users (eg, listeners).

The above described features and advantages of the invention will be apparent from the following description.

100‧‧‧Monitoring system

110‧‧‧Monitor

120‧‧‧Display unit

130‧‧‧Communication unit

140‧‧‧ storage unit

150‧‧‧Processing unit

160‧‧‧Monitoring module

170‧‧‧Image capture unit

201‧‧‧Face recognition module

203‧‧‧Eye tracking module

205‧‧‧Marking module

207‧‧‧State Identification Module

209‧‧‧Closed eye detection module

211‧‧‧Swing head detection module

400‧‧‧User interface

401‧‧‧ first mark

402‧‧‧Second mark

403‧‧‧ third mark

404‧‧‧ fourth mark

S‧‧‧ Space

S305~S325‧‧‧ steps of the monitoring method

S505~S550‧‧‧Steps of another monitoring method

1 is a block diagram of a monitoring system in accordance with an embodiment of the present invention.

2 is a block diagram of a monitoring module in accordance with an embodiment of the present invention.

3 is a flow chart of a monitoring method in accordance with an embodiment of the present invention.

4 is a schematic diagram of a user interface in accordance with an embodiment of the present invention.

FIG. 5 is a flow chart of a monitoring method in accordance with another embodiment of the present invention.

1 is a block diagram of a monitoring system in accordance with an embodiment of the present invention. Referring to FIG. 1 , the monitoring system 100 includes a monitoring device 110 and a plurality of imaging units 170 . The monitoring system 100 is installed, for example, in a space S such as a classroom, a lecture hall, or an auditorium, that is, the monitoring device 110 and the plurality of image capturing units 170 are disposed in the same space S.

The image capturing unit 170 is, for example, a camera or a camera, and has a charge coupled device (CCD) lens, a complementary metal oxide semiconductor transistor (CMOS) lens, or an infrared lens. The image capturing unit 170 is separately disposed in a plurality of positions of the space S to capture an image sequence of the user in the position. In the case where the space S is a classroom, the classroom includes a plurality of seats, and an image capturing unit 170 is provided for each seat, and the lens of the image capturing unit 170 is oriented to be photographable in a direction in which the user sits on the seat.

The monitoring device 110 includes a display unit 120, a communication unit 130, a storage unit 140, a processing unit 150, and a monitoring module 160. The processing unit 150 is coupled to the display unit 120, the communication unit 130, the storage unit 140, and the monitoring module 160.

The display unit 120 is, for example, a liquid crystal display (LCD), a plasma display, a vacuum fluorescent display, a Light-Emitting Diode (LED) display, a Field Emission Display (FED), and / or other suitable type of display, here is not limited to its type.

The communication unit 130 is configured to receive its respective image sequence from the plurality of image capturing units 170. For example, the communication unit 130 is a physical Ethernet card, or a wireless network card, or a third generation (3G) mobile communication module, and a General Packet Radio Service (GPRS) module. Group or Wi-Fi module.

The storage unit 140 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory (Flash memory), hard Disc or other similar device or a combination of these devices. The storage unit 130 includes a plurality of electronic files, and temporarily stores the image sequence captured by the image capturing unit 110.

The processing unit 150 is, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (DSP), programmable Controllers, Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), or other similar devices or combinations of these devices.

The processing unit 150 is coupled to the display unit 120, the communication unit 130, and the storage unit 140, and is configured to drive the monitoring module 160. The monitoring module 160 is, for example, a driver, firmware or software written by a computer programming language, which is stored in the storage unit 140. The monitoring module 160 is basically composed of a plurality of code segments (for example, creating an organization code code segment, signing a form code segment, setting a code segment, and deploying a code segment), and the code segments are Loading monitoring device After being executed in 110, the monitoring function can be realized. Alternatively, the monitoring module 160 is a wafer set or the like formed by various logic gates. The processing unit 150 drives the monitoring module 160 to perform the monitoring method.

For example, the monitoring module 160 performs a face recognition algorithm on the image sequence to determine whether the image sequence has a user's portrait, and the monitoring module 160 performs an eye tracking algorithm on the image sequence to obtain a direction in which the user looks. . In addition, the monitoring module 160 performs a corresponding marking action on each seat block based on whether the user is looking at the specified direction (for example, looking at the front). For example, a corresponding color on each seat block mark, or a corresponding mark (for example, an image) or the like on each seat block.

2 is a block diagram of a monitoring module in accordance with an embodiment of the present invention. Referring to FIG. 2 , the monitoring module 160 mainly includes a face recognition module 201 , an eye tracking module 203 , and a marking module 205 . The face recognition module 201 is configured to perform a face recognition algorithm on the image sequence, for example, an AdaBoost algorithm based on the Haar-like feature, thereby detecting whether a face exists in the image sequence. Whether the user exists in the corresponding position is determined by detecting the presence or absence of the face of the image sequence. The eye tracking module 203 is configured to perform an eye tracking algorithm on the image sequence to track the movement trajectory of the user's eyeball. The marking module 205 is configured to perform a corresponding marking action on the seat block.

In addition, the monitoring module 160 may further include a state recognition module 207, a closed eye detection module 209, and a oscillating head detection module 211. The state recognition module 207 is configured to determine, according to the result of the eye tracking module 203, that the user is currently in a state of concentration, a doze state, or an unfocused state. The closed eye detection module 209 is configured to execute the image sequence Closed eye detection algorithm. The oscillating head detecting module 211 determines whether the user turns the head based on the nostril position information, thereby obtaining the face swing information. For example, after detecting the face, the face recognition module 201 can further search for the nostril area, that is, the position of the two nostrils. The information on the position of the nostrils is, for example, the position of the two nostrils.

In addition, in other embodiments, in each position of the space S, a display may be disposed with respect to each of the image capturing units 170 to display information such as course content on the display. Accordingly, the correction program may be performed on the image capturing unit 170 before the eye tracking calculation is performed. If the hardware specifications of the image capturing units 170 are the same, correction may be performed by one of the image capturing units 170. If the hardware specifications of the image capturing units 170 are different, the image capturing units are corrected one by one. For example, the image capturing unit 170 sequentially receives a plurality of corrected images before detecting the position of the eyeball. Here, the corrected images are respectively obtained by the user viewing a plurality of correction points of the display unit 120. For example, four points of the upper left, upper right, lower left, and lower right of the display unit 120 are used as correction points. When the calibration procedure is performed, the user is prompted to view the above four correction points in the display unit 120, thereby obtaining four corrected images. The calibration module 314 obtains the reference correction parameters according to the two bright spot positions of the eye region in each of the corrected images. The above two bright spot positions are formed due to the reflection caused by the light-emitting module provided in the image capturing unit 170 on the eyeball. The reference correction parameters are obtained by correcting the two bright spot positions in each image. The correction parameters are, for example, vectors based on the bright spot positions G1, G2. And, a coordinate transformation matrix is generated by a perspective transformation method based on the corrected image, and the coordinate transformation matrix is used to convert the coordinate position of the eye region into a coordinate position of the display.

The eye tracking module 203 detects the eye region of the current image in the image sequence to obtain the pupil position and the two bright spot positions in the current image (hereinafter referred to as the bright spot positions G1', G2'). Moreover, the eye tracking module 203 obtains the alignment correction parameter according to the bright spot positions G1', G2' of the current image, thereby further obtaining the dynamic correction parameter according to the reference correction parameter (C1) and the comparison correction parameter (C2). (C3). For example, the dynamic correction parameter is the ratio of the reference correction parameter to the comparison correction parameter, that is, C3=C2/C1. Thereafter, the eye tracking module 203 calculates the eye movement coordinate according to the bright spot position G1' (or the bright spot position G2') in the current image, the pupil position (for example, calculated by the coordinates of the pupil center), and the dynamic correction parameter. For example, the eye movement coordinates are (X', Y'). The eye tracking module 203 uses the coordinate conversion matrix to convert the eye movement coordinate (X', Y') to the line-of-sight coordinates of the display unit (for example, the line-of-sight coordinates are (Xs, Ys)), and then , record the line of sight coordinates (Xs, Ys). Accordingly, the movement trajectory of the eyeball can be obtained by recording the plurality of line-of-sight coordinates, and the direction in which the user is currently gazing can be obtained according to the line-of-sight coordinates.

The steps of the monitoring method are illustrated below by way of example. 3 is a flow chart of a monitoring method in accordance with an embodiment of the present invention. Referring to FIG. 1 to FIG. 3 simultaneously, in step S305, the monitoring device 110 displays the user interface in the display unit 120. Here, the user interface includes a plurality of seat blocks, and the seat blocks respectively correspond to a plurality of positions of the space S, and each of the positions is provided with an image capturing unit 170. The space S is used as a classroom, and the seating table of the classroom is displayed in the user interface.

Next, in step S310, the monitoring device 110 obtains an image sequence from the image capturing unit 170 through the communication unit 130. After that, the monitoring module 160 can be utilized. Analyze the image sequence.

In step S315, the monitoring module 160 executes the face recognition algorithm through the face recognition module 201, thereby determining whether the image sequence has a user's portrait. That is, whether there is a user at the corresponding location, for example, to determine whether the student is present.

Next, in step S320, the monitoring module 160 performs an eye tracking algorithm through the eye tracking module 203. Specifically, the monitoring module 160 activates the eye tracking module 203 when the image of the user is detected in the image sequence.

Then, in step S325, the monitoring module 160 performs a corresponding marking action on each seat block based on whether the user is looking at the specified direction. For example, it is detected whether the user has a display on the table (which is used in conjunction with the image capturing unit 170), or whether the user is viewing the display or whiteboard (blackboard) disposed at the forefront.

Color or pattern in the seating chart indicates whether the student is absent, attentive, or dozing off. In addition, the state recognition module 207 can also determine whether the user is currently in a state of concentration, a doze state, or an unfocused state according to the result of the eye tracking module 203.

In addition, if the students who have used the respective locations have been allocated in advance and a database is established in the storage unit 140 of the monitoring device 100 in advance, the registration action can be further adopted for the absence status.

4 is a schematic diagram of a user interface in accordance with an embodiment of the present invention. Referring to FIG. 4, the user interface 400 is a seating table including a 5×4 seat block. B1~B20. Here, only 5×4 is taken as an example. However, in other embodiments, the adjustment may be made according to the number of positions in which the image capturing unit 170 is disposed in the actual space S.

In the case of the seat blocks B5, B8, B16, and B9, the first mark 401, the second mark 402, the third mark 403, and the fourth mark 404 are respectively displayed. That is, the first mark 401 is displayed in the seat block B5, indicating that there is a user in the corresponding actual position, and the user is currently in a focused state. A second mark 402 is displayed in the seat block B8, indicating that there is a user in its corresponding actual position, and the user is currently in a doze state. A third mark 403 is displayed in the seat block B16, indicating that there is a user in the corresponding actual position, and the user is currently in an unfocused state. A fourth marker 404 is displayed in the seat block B9 indicating that there is no user in the corresponding actual location. According to this, in the teaching, the user interface 400 allows the teacher to quickly know the attendance of the student and the concentration of the class.

FIG. 5 is a flow chart of a monitoring method in accordance with another embodiment of the present invention. Referring to FIG. 1 to FIG. 2 and FIG. 4 to FIG. 5 simultaneously, in step S505, the monitoring module 160 performs a face recognition algorithm through the face recognition module 201. In step S510, it is determined whether or not there is a portrait of the user in the video sequence, thereby determining whether or not the user exists in the actual location. If there is no portrait in the image sequence, it is determined that the location of the image capturing unit that transmits the image sequence is not provided by the user, and in step S515, the marking module 205 displays the fourth marker 404 in the corresponding seat block.

Next, in step S520, the monitoring module 160 performs an eye tracking algorithm through the eye tracking module 203. Thereafter, in step S525, the eye tracking module 203 determines whether the user is looking at the specified direction. When the eye tracking module 203 judges The user is in the specified direction, and the state recognition module 207 determines that the user is currently in the focused state. In step S530, the marking module 205 displays the first marker 401 in the corresponding seat block.

In addition, when it is determined that there is a human image in the image sequence, the swing head detecting module 211 may first determine whether the user faces the specified direction based on the face swing information, so as to determine when the user faces the specified direction, The image sequence performs an eye tracking algorithm. However, whether or not to use the oscillating head detecting module 211 can be determined depending on the situation.

If the user does not look at the specified direction, step S535 is executed to perform a closed-eye detection algorithm on the image sequence through the closed-eye detection module 209. Next, in step S540, the closed-eye detection module 209 determines whether the user is in the closed-eye state. For example, it is determined whether the eye object is in a closed state by detecting the size of the eye object. For example, when the height of the eye object is less than the height threshold (eg, the height threshold is in the range of 5-7 pixels), and the width of the eye is greater than the width threshold (eg, the width threshold is outside the range of 60). When it is ~80 pixels, it is judged to be in the closed eye state. If the above conditions are not met, the state of the eye is not closed.

When it is detected that the user is in the closed state, the state recognition module 207 determines that the user is currently in a doze state, and in step S545, the marking module 205 displays the second mark 402 in the corresponding seat block. When it is detected that the user is in the non-closed state, the state recognition module 207 determines that the user is currently in an unfocused state, and in step S550, the marking module 205 displays the third mark 403 in the corresponding seat block.

In addition, the first mark to the fourth mark may also be ground colors of different colors. The user's concentration and absence status are indicated by displaying the background colors of different colors in the seat block.

In summary, the eye tracking algorithm is used to determine the degree of concentration of the user, and the corresponding seat block in the user interface presents a corresponding mark. According to this, when applied to teaching, teachers can be more intuitive to know the attendance rate and concentration of students. In addition, it can also be applied to general lectures, so that the lecturer can know the degree of concentration of the listener through the mark on the user interface, and then appropriately adjust the manner of the lecture to attract the attention of the listener.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S305~S325‧‧‧ steps of the monitoring method

Claims (13)

A monitoring method for a monitoring device, wherein the monitoring device is located in a space, the method comprising: displaying a user interface in a display unit, wherein the user interface comprises a plurality of seating blocks, and the The seating blocks respectively correspond to a plurality of positions of the space, and each of the positions is provided with an image capturing unit; and the image capturing unit of each of the positions obtains an image sequence; performing a face recognition algorithm on the image sequence Determining whether the image sequence has a portrait of a user; if the image sequence exists in the image sequence, performing an eye tracking algorithm on the image sequence; and based on whether the user is looking at a specified direction, These seating blocks perform corresponding marking actions. The method of claim 1, wherein the step of performing the eye tracking algorithm on the image sequence further comprises: if the user is in the specified direction, determining that the user is currently in a focused state And if the user is not looking at the specified direction, performing a closed-eye detection algorithm on the image sequence to determine whether the user is currently in a doze state or an unfocused state. The method of claim 2, wherein after the step of performing the closed-eye detection algorithm on the image sequence, the method further comprises: If it is detected that the user is in a closed state, it is determined that the user is currently in the doze state; and if the user is not detected to be in the closed state, it is determined that the user is currently in the unfocused state. The method of claim 2, wherein the step of performing a corresponding marking action on each of the seating blocks based on whether the user is looking at the specified direction comprises: when the user is currently in the focus a state, displaying a first mark in the corresponding seat block; when the user is currently in the doze state, displaying a second mark in the corresponding seat block; when the user is currently in the unfocused state Displaying a third mark in the corresponding seat block. The method of claim 1, wherein after the step of performing the face recognition algorithm, the method further comprises: if the image sequence does not exist, displaying a fourth mark in the corresponding seat block. . The method of claim 1, wherein the step of performing the face recognition algorithm on the image sequence further comprises: detecting a nostril region of the face region when obtaining a face region, a nostril position information; based on the nostril position information to determine whether the user turns his head, thereby obtaining a The face swing information; and determining whether the user faces the specified direction based on the face swing information, and executing the eye tracking algorithm on the image sequence when determining that the user faces the specified direction. A monitoring device includes: a display unit displaying a user interface, wherein the user interface comprises a plurality of seat blocks, and the seat blocks respectively correspond to a plurality of positions in the same space as the monitoring device, and Each of the positions is provided with an image capturing unit; a communication unit obtains an image sequence from the image capturing units of the respective positions; a storage unit stores the image sequence; and a processing unit coupled to the display unit The communication unit and the storage unit drive a monitoring module; wherein the monitoring module performs a face recognition algorithm on the image sequence to determine whether the image sequence has a user portrait; if the image sequence The presence of the portrait, the monitoring module performs an eye tracking algorithm on the image sequence; and the monitoring module performs a corresponding marking action on each of the seating blocks based on whether the user is looking at a specified direction. . The monitoring device of claim 7, wherein the monitoring module further comprises: a face recognition module, the face recognition algorithm is executed on the image sequence; and an eye tracking module is executed on the image sequence. The eye tracking algorithm; A marking module performs a corresponding marking action for each of the seating blocks. The monitoring device of claim 8, wherein the monitoring module further comprises: a state recognition module, according to the result of the eye tracking module, determining that the user is currently in a state of concentration and a state of sleep Or an unfocused state; and a closed-eye detection module, performing a closed-eye detection algorithm on the image sequence; wherein, when the eye tracking module determines that the user is not looking at the specified direction, the state recognition mode The group determines that the user is currently in the state of focus; wherein when the eye tracking module determines that the user is not looking at the specified direction, and the closed eye detection module detects that the user is in the closed eye state, The state recognition module determines that the user is currently in the doze state, and when detecting that the user is in a non-closed state, the state recognition module determines that the user is currently in the unfocused state. The monitoring device of claim 9, wherein when the user is currently in the state of concentration, the marking module displays a first mark in the corresponding seat block; when the user is currently in the sleepy state In the state, the marking module displays a second mark in the corresponding seat block; when the user is currently in the unfocused state, the marking module displays a third mark in the corresponding seat block. The monitoring device of claim 8, wherein the marking module displays a fourth mark in the corresponding seat block when the face recognition module determines that the portrait is not present in the image sequence. The monitoring device of claim 8, wherein the monitoring mode The group further includes: a oscillating head detecting module, which determines whether the user turns the head based on a nostril position information, thereby obtaining a face swinging information, wherein when the face recognition module obtains a face area, The face recognition module detects the nostril area of the face area to obtain the nostril position information. When the swing head detection module determines that the user faces the designated direction based on the face swing information, The eye tracking algorithm is executed on the image sequence through the eye tracking module. A monitoring system, comprising: a plurality of image capturing units respectively disposed at a plurality of locations in a space; a monitoring device disposed in the space, wherein the monitoring device comprises: a display unit displaying a user interface, wherein the The user interface includes a plurality of seating blocks, and the seating blocks respectively correspond to the positions; a communication unit obtains an image sequence from each of the image capturing units; and a storage unit stores the image sequence; a processing unit coupled to the display unit and the communication unit, and driving a monitoring module; wherein the monitoring module performs a face recognition algorithm on the image sequence to determine whether the image sequence exists for a user a portrait image; if the image sequence exists in the image sequence, the monitoring module performs an eye tracking algorithm on the image sequence; and the monitoring module is based on whether the user is looking at a specified direction, and each of the seating areas The block performs the corresponding tag action.