WO2020226603A1 - Automated method and system for determining an extent to which information is recognized and automated method for verifying familiarization with an electronic document - Google Patents

Abstract

The invention relates to information technologies and concerns an automated method and system for determining an extent to which information is recognized by a subject during the presentation of multimedia information. Provision is made for presenting multimedia information, capturing and processing data regarding the movement of the eyes of the subject by means of a video capture device (10) and by means of an artificial neural network, with subsequent determination of the extent to which the presented multimedia information is recognized by the subject. The system comprises a computer appliance of the user (1) and a remote computer appliance for interaction with the user (2), a data processing module (71) of which comprises an operating and training unit (81) for the artificial neural network. The invention also concerns an automated method for verifying the familiarization of the subject with an electronic document and the signature thereof.

Description

AUTOMATED METHOD AND SYSTEM FOR DETERMINING VOLUME

RECOGNITION OF INFORMATION, AND AUTOMATED METHOD OF VERIFICATION

ABOUT THE ELECTRONIC DOCUMENT

TECHNICAL FIELD OF THE INVENTION

The present invention relates to technologies that include means and methods for tracking the movement of the eyes of subjects, and relates to a method and system for automatically determining the amount of reading or recognition by a subject of certain areas when presenting multimedia information. The invention makes it possible to establish whether the subject was familiar with the visual information presented to him, in particular text information and / or figurative information, to what extent, and whether this volume was sufficient to confirm the subject's familiarization with the content of text fragments of the presented multimedia information, whether the subject read the presented information, in particular whether the subject has read the textual information.

The invention makes it possible to objectively establish whether a certain subject has read textual information, and to what extent, and whether it corresponds to a certain amount, established as sufficient for the subject to recognize the content of the presented text fragments in the presented multimedia information.

In addition, the invention relates to a method for verifying familiarization with an electronic document, in particular, involving a method and / or a system for automatically determining the amount of reading or recognition by a subject of certain areas when presenting multimedia information.

PRIOR ART

Eye tracking technologies (oculography, eye tracking) provide for the assessment of certain characteristics of eye movements, in particular, determining the point at which the gaze is directed, or the coordinates of the point of intersection of the optical axis of the eye (eyeball) with the plane of a certain object, in particular the plane of the device on which it is presented / a certain visual stimulus is presented. The use of eye tracking systems (eye tracking or oculography) in various fields, and in particular for determining the reading of a subject, is known.

There is a known method of confirming the reading of an email by a user using the technology of tracking the position of the eyes while the user reads an electronic message containing a plurality of lines on an electronic device, in which, based on the position of the eyes, in particular, the determination of the coordinates of the position of the eyes, the number of lines of the message viewed by the user is determined and the read whether the user of the message by comparing the number of lines viewed with the eyes against a threshold number of lines, which is based on the set of lines of the email messages. In addition, it is determined whether the user was surprised while viewing the email by comparing the length of time spent by the eyes while viewing the email with the expected length of time for displaying the submitted email (US 2016/0094705 A1, 03/31/2016).

There is a known method and system for testing the user's knowledge of the language, which automatically determine the level of language proficiency using the technology of tracking the user's gaze when reading a sample of text using a camera (or tracker) for the eyes, which is connected to the processor and is configured to record the fixation time of the eyes for each word of the sample text presented to the user, and the subsequent recording of the saccadic time for each pair of words along which the user's gaze moves between fixations, with the subsequent comparison of the user's gaze parameters with the gaze model parameters of at least one reader who has a certain known level of language proficiency, and the subsequent generation assessment of the user's language proficiency based on the comparison results (US 2019/0080623 A1 03/14/2019).

US 2012/0237084 A1 (20.09.2012) describes a method and system for identifying text and its position in visual media content, for example, a document displayed on a computer monitor or other display device, allowing to determine whether the subject interacted with the text, the level interactions of the subject with the text, for example, whether the subject considered the text, whether the subject read the text, whether the subject perceived the text and understood it, or other types of interaction of the subject with the text. The determination is carried out on the basis of data obtained from the eye tracking device, and the emotional response determination system can be used separately or together. The method provides for the identification of a part of the visual media content constituting the text, determining the position of the identified text of the visual media content, collecting data from the subject's eyes while viewing the visual media content regarding the pupil size, blinking and gaze coordinates, generating the subject's gaze patterns based on the collected data with eye movements, determining whether the subject interacts with the identified text based on at least a certain position of the identified text and the generated gaze pattern, determining the level of interaction of the subject with the identified text, in particular determining whether the subject has read or familiarized with the identified text, and determining the subject's emotional reaction to the identified text.

The known methods and systems provide for the use of specialized equipment for tracking eye movements, which cannot always be provided for laboratory conditions and research institutions, and especially in everyday life. When studying eye movements, even with the involvement of specially adapted equipment for this, there is a question of choosing data for further processing and interpretation. For example, in the case of latent attention, which is observed on recordings of eye movements, when the gaze follows and the fixation points pass by the real object of attention with only a short fixation time, difficulties arise in establishing an unambiguous connection between the results of the study of eye movement and the cognitive process of a particular subject.

Although various eye tracking systems and their applications are known, there are still certain problems that people encounter while reading, and each one solves these problems in a unique way: rereading difficult parts of the text, finding unknown words, writing down details to remember, etc. etc. Automation of these processes using publicly available and widely used equipment, for example, to track a person's reading, estimate speed, distinguish eye movements and behavioral reactions while reading and other activities, comment on fragments of text that are difficult to read or whose content has not been recognized, etc. are functions that can be extremely useful for people who have to process huge amounts of text per day.

Currently, there are no solutions for standardized verification and confirmation of the processing of visual information, and in particular reading, with high accuracy, based not on specialized hardware, but on any video devices or webcams using the principle of eye tracking.

The problem that is solved within the framework of this invention lies in areas where decision-making essentially depends on whether the subject (person) has actually read the information presented, in particular, whether he has read a certain document. The present invention allows it to be applied in such industries as education, legal practice, public sectors and the like.

SUMMARY OF THE INVENTION

An object of the present invention is to provide the ability to automatically determine and evaluate the subject's recognition volume and reading volume with high accuracy when viewing multimedia information on various devices, such as a personal computer or laptop, tablet, smartphone, virtual screen, or any other appropriate device or display. The invention allows in real time to check whether the subject is familiar with the presented information, to determine its volume, to determine the recognition by the subject of text sections, tables, images, etc. when presenting multimedia information and the amount of information, the content of which was recognized by the subject, in particular, to check the reading of text parts in the presented multimedia information or reading any electronic document.

The basis of the invention is to provide a method and a system for automated determination with high accuracy of the volume of information recognition by a subject when presenting multimedia information, in particular the amount of readable information, it is even better to determine the amount of information read using widespread electronic devices suitable for displaying multimedia information, video recording, processing, storing and transmitting data. Another object of the invention was to provide an automated method for signing electronic documents when establishing that it was the verified subject who read a certain volume of a certain document.

In the first aspect of the invention, an automated method for determining the amount of information recognition by a subject when presenting multimedia information, provides for the provision of multimedia information, fixing and processing data on the subject's eye movements using a video device and determining the amount of recognition by the subject of the presented multimedia information, the method being carried out using a system comprising :

- user hardware and software complex (APK-P), in which a video device, a multimedia information display module, a data reception and transmission module, a video stream data recording module from a video device, a data storage module, a data processing module, are combined or integrated, and

- a remote hardware and software complex for interaction with a user (AIC-VP), in which a multimedia information display module, a data reception and transmission module, a data storage module, a user subject verification module, a data processing module capable of working and training an artificial neural network,

and the method includes the following steps:

- in APK-P, calibration is carried out taking into account the individual anatomical parameters of the eyes when presenting to the subject moving visual stimuli and the task of tracking;

- download multimedia information to the remote APC-VP, classify fragments of structural elements according to the characteristics - text, image, table, etc., transfer the classified information from the remote APC-VP to APC-P and present it to the user;

- according to the data of the video stream from the video device, the data of eye movements and / or gaze are recorded on the APK-P, taking into account behavioral reactions in the form of at least one of the blinking of the eyes, and / or head movements, and / or changes in the coordinates of the previously marked points on the face, individual points of view are determined at each moment in time in the coordinates of the plane of the multimedia information display module APC-P when presenting information, and the received information is transmitted to the remote APC-VP in which:

- establish a correspondence between individual points of view and a piece of multimedia information presented at this time, - using an artificial neural network, they form data from a set of views and establish the likelihood of the relationship of each individual view of the subject to reading;

- aggregate data of multiple views into a coordinate system of classified structural elements of the presented multimedia information;

- the density of all gazes and the gaze density defined as reading are calculated from the number in each structural element, and appropriate threshold values for the calculated densities are set, which separate the statistical noise;

- calculate the ratio of the overlapping area of the area where the data density of all gazes and the gaze data density, defined as reading, exceeds the threshold values, to the total area of all structural elements defined in the presented multimedia information, and the amount of recognized information is set by the subject by which the information is determined as read ...

In a further aspect of the invention, the multimedia information presented to the subject is text fragments without images and / or tables or with images and / or tables in well-known formats for presenting electronic documents, or video information that contains text fragments and / or images and / or tables.

In yet another aspect of the invention, the method includes the additional step of verifying the face of the subject based on visual data about the subject entered into the distant APC-VI before providing the multimedia information to the subject.

In yet another aspect of the invention, the method includes verifying the identity of the subject prior to each subsequent step of providing the subject with multimedia information or electronic document.

In yet another aspect of the invention, the method includes calibrating the parameters of the eyes with certain behavioral responses prior to each step of providing the subject with multimedia information or electronic document.

In a further aspect of the invention, a display device for multimedia information or, for example, an electronic document is a television, a computer monitor, a smartphone display, a tablet display, or any other device capable of displaying multimedia information.

In another aspect of the invention, the video fixing device used is, in particular, a web camera or video camera connected or integrated into the APC-P, in particular into a computer, smartphone, tablet or any video device capable of receiving and transmitting video data.

In a further aspect of the invention, the number of gaze transitions from certain text to related materials, in particular tables and images, and the time spent on such transitions, when presenting multimedia information, are determined and optionally presented on the multimedia information display module. In yet another aspect of the invention, a map of gaze data density and / or gaze data density, defined as read, and / or overlapping area of an area where the data density of all gazes and the gaze data density, defined as read, exceed threshold values, is generated, and optionally presented module for displaying multimedia information.

In yet another aspect of the invention, the percentage of the structured units viewed from the total number of units in the presentation of multimedia information in which the view has not been determined to be read is calculated and optionally presented on the display unit of the multimedia information.

In yet another aspect of the invention, the percentage of the building blocks in which the gaze has been determined to be read is calculated from the total number of building blocks when presenting multimedia information, and optionally presented on the multimedia display module.

In a further aspect of the invention, the ratio of the volume of the structural elements viewed, in which the gaze was not defined as reading, to the volume of the structural elements, in which the gaze was defined as reading, is calculated when presenting multimedia information, and optionally presented on the multimedia display module.

In a further aspect of the invention, upon completion of a session for providing multimedia information or a fragment thereof, a read verification certificate is generated in the APC-VP, in particular as an electronic read verification certificate document, and optionally stored in a data storage unit.

In a further aspect of the invention, the read verification certificate is downloaded to the APC-P.

In a further aspect of the invention, it is provided that all received subject data is stored in the data storage unit APC-VP and optionally APC-P.

Another important aspect of the invention is an automated system for determining the scope of information recognition by a subject when presenting multimedia information, including:

a) the user's hardware and software complex (AIC) (AIC-P), in which the following are combined or integrated:

i) video recording device,

ii) module for displaying multimedia information,

W) a module for receiving and transmitting data, configured to receive and transmit the received information to a remote AIC for interaction with a user (AIC-VP),

iv) a module for recording from a video recording device data of a video stream concerning eye movements and / or gaze, and behavioral reactions in the form of at least one of blinking eyes, and / or head movements, and / or changes in coordinates of previously marked points on the face,

V) data storage module,

vi) data processing module, including:

- a block for processing video stream data from a video recording device APK-P,

- block for calibrating eye parameters when presenting multimedia information,

- a unit for determining individual points of view at each moment in time in the plane coordinates of the multimedia information display module when presenting information, and

b) a remote APK for user interaction (APK-VP), in which the following are combined or integrated:

i) module for displaying multimedia information,

ii) a module for receiving and transmitting data, configured to receive and transmit the received and processed information to the APC-P,

W) data storage module,

iv) a module for verifying the identity of the user subject,

v) a data processing module capable of operating and training an artificial neural network, including:

- block of work and training of an artificial neural network,

- a block for classifying fragments of multimedia information into structural elements based on the characteristics of text, images, tables, etc. (any other elements are not texts, not tables, not images),

- a block for processing video stream data from a video recording device APK-P,

- a unit for establishing a correspondence between individual points of view and a piece of multimedia information presented at this time;

- a block for establishing the reliability of the attitude of each individual gaze of the subject to reading,

- a block for generating data of multiple views and aggregating data of multiple views into a coordinate system of classified structural elements of the presented multimedia information,

- a block for calculating the number in each structural element of the data density of all views and the data density of the views defined as reading, and setting the corresponding threshold values for the calculated densities, which separates the statistical noise,

- a unit for calculating the ratio of the overlapping area of the area where the data density of all views and the data density of the views, defined as reading, exceeds the threshold values, to the total area of all structural elements defined in the presented multimedia information, - block for establishing the amount of information recognized by the subject,

- block for establishing the amount of textual information read by the subject,

- block for presenting statistical information and generating reports.

In a further aspect, the APC-VP of the automated system further comprises a module for controlling the modes of presentation of multimedia information on the APC-P.

In a further aspect, the module for displaying multimedia information APC-P and / or APC-VP is configured to present multimedia information to a subject, which is an electronic document, text fragments without images and / or tables, or with images and / or tables in known formats for presenting electronic documents or video information that contains text fragments and / or images and / or tables.

In yet another aspect of the invention, in an automated system, the identity verification module of the user subject is configured to verify based on visual data about the subject that are entered into the remote APC-IP, before the first and / or before each session of presenting the multimedia information or electronic document to the subject.

In yet another aspect of the invention, in an automated system, an eye parameter calibration unit is configured to calibrate the parameters of eyes with certain behavioral responses before the first and / or before each session of presentation of multimedia information or an electronic document to the subject.

In a further aspect of the invention in an automated system, the module for displaying multimedia information or an electronic document in the APC-P and / or APC-VP is a TV screen, a computer monitor, a smartphone display, a tablet display, or any other device capable of displaying multimedia information.

In another aspect of the invention in an automated system, a video fixing device is, in particular, a webcam or video camera connected or integrated into the APC-P, in particular into a computer, smartphone, tablet or any video device capable of receiving and transmitting video data.

In yet another aspect of the invention in an automated system, the APC-VP data processing module comprises a unit for generating a map of gaze data density and / or gaze data density defined as reading and / or overlapping area of an area where the data density of all gazes and the gaze data density determined as reading exceeds the thresholds where the card is suitable for presentation on the media display module.

In the next aspect of the invention in the system, the APC-VP data processing module contains a unit for calculating the percentage of viewed structural elements from the total number of structural elements when presenting multimedia information in which the view is not was defined as reading, in particular the number of transitions of the gaze from a particular text to related materials.

In yet another aspect of the invention, in an automated system, the APC-VP data processing module comprises a unit for calculating the percentage of structural elements in which the gaze was determined to be reading from the total number of structural elements when presenting multimedia information.

In another aspect of the invention, in an automated system, the APC-VP data processing module comprises a unit for calculating the ratio of the volume of viewed structural elements, in which the gaze was not defined as reading, to the volume of structural elements, in which the gaze was determined as reading, when presenting multimedia information.

In another aspect of the invention, in an automated system, the APC-VP data processing module contains a unit for presenting statistical information about the behavior of a subject during reading, in particular, data on the amount of information read, reading speed, information perception complexity, data on various aspects of eye movement and certificate generation reading checks, in particular as an electronic document, upon completion of each session or series of sessions of presenting multimedia information or a fragment thereof.

Another important aspect of the invention is an automated method for verifying the familiarization of a subject with an electronic document, which provides for a preliminary determination of the volume of information recognition, in particular, the volume of a read document sufficient to generate a certificate of signing an electronic document by a verified subject, verification of the subject's identity based on previously entered into the hardware and software a complex (AIC) of visual data about a subject at the beginning of a session of presenting an electronic document to a subject and during each subsequent session of presenting the same document to the same subject, determining the volume of information recognition, in particular the volume of a read document by a verified subject when presenting an electronic document for all sessions of presenting one and the same document using the eye-tracking technology on the APC, and in the event that the subject confirms the end of the session / sessions of acquaintance with the document volume and approves the signing, and if a certain amount of information recognized by the verified subject, including the volume of the read document, is equal to or greater than the predetermined volume, a document signing certificate is generated.

Determination of the volume of information recognition, in particular the volume of a read document, by a verified subject in the method of signing an electronic document can be carried out using the method disclosed herein for automatically determining the volume of information recognition. In yet another aspect of the invention, in the method of signing an electronic document, the determination of the scope of information recognition, in particular the amount of the read document by the verified subject, is carried out using the system for automated determination of the scope of information recognition described herein.

Also, another aspect of the invention is a computer-readable storage medium containing program code executable on a computer for performing operations of the methods described herein.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 - Automated system for determining the volume of information recognition, where

(1) APC-P, (10) video recording device, (20) module for displaying multimedia information, (30) module for receiving and transmitting data, (50) module for recording from a video recording device for video stream data, (40) data storage module, (70 ) a data processing module, (80) a data processing unit for a video stream from a video recording device APK-P, (90) a unit for calibrating eye parameters, (100) a unit for determining individual points of view at each time point in plane coordinates of the multimedia information display unit;

(2) APC-VP, (21) module for displaying multimedia information, (31) module for receiving and transmitting data, (41) module for storing data, (61) module for verifying the identity of the user subject (71) module for processing data, configured for the operation and training of an artificial neural network, (81) a unit for operation and training of an artificial neural network, (91) a unit for classifying fragments of multimedia information into structural elements, (101) a unit for processing video stream data from a video recording device APK-P, (111) a setting unit correspondence between individual gaze points and a fragment of multimedia information presented at this time, (121) a unit for establishing the reliability of attributing each individual gaze of a subject to reading, (131) a unit for generating multiple gaze data and aggregating multiple gaze data into a coordinate system of classified structural elements of the presented multimedia information , (141) calculation block by quantity in each structural element not the data density of all gazes and the gaze data density defined as reading, and setting appropriate thresholds for the calculated densities, (151) block for calculating the ratio of the area of overlap of the area where the data density of all gazes and the gaze data density defined as reading exceeds the threshold values, to the total area of all structural elements defined in the presented multimedia information, (161) a unit for setting the amount of information recognized by the subject, (171) a unit for setting the amount of textual information read by the subject, (181) a unit for presenting statistical information and generating reports.

FIG. 2 - Indicators in time series.

FIG. 3 - Filtering using BPOGV, FPOGV. FIG. 4 - Filters only gaze movements on the screen.

FIG. 5 - Convolutional neural network, architecture with 113,006 trainable parameters.

FIG. 6 - Classification of each group of fixations (observations grouped by FPOGID).

DETAILED DESCRIPTION OF THE INVENTION

The existing methods and systems for tracking eye movements for various purposes provide for the use of specialized video devices - eye trackers. As a rule, attention is focused on determining the exact coordinate positions of the smallest structural elements of the text, in particular a word, even better than a letter, and on tracing the trajectories and basic patterns of movement. There are certain restrictions on the use of eye-trackers, depending on the method on which the system is based, in particular, the dark pupil or the bright pupil, and the significant influence on the obtained data of certain behavioral reactions.

Known methods do not offer methods of remote processing of eye tracking data obtained from non-specialized eye tracking devices, in particular, using a standard webcam, which can be defined for example as web tracking, or any video device that is attached or integrated into any electronic device for personal use, capable of receiving, processing, transmitting and optionally storing data, where such a method has a high definition accuracy and is easily accessible to the user.

For the purpose of determining the amount of recognized and / or read information by a subject without the involvement of specialized equipment, this invention proposes a combination in a certain way into an automated system of software and hardware configured to perform certain functions, as described herein. The use of an artificial neural network is envisaged, through training of which, at the initial stage, template models for determining the density of a gaze in certain structural elements of multimedia information, in particular an electronic document, have been created, and subsequently made it possible to achieve 96% accuracy in determining the volume of recognition of the presented multimedia information and determining the volume of information read.

The described method and system, which was named StoryLook, allows the subject-user to track and save all reading materials and associated read metadata, search and filter content by reading template parameters (for example, the time and speed of fixation allows you to separate the region of interest within text); receive after processing various analytical annotations, receive assistance options in real time while familiarizing with the content of multimedia information, in particular reading (for example, automated translation). The system automatically tracks eye movements and correlates them with information such as text on the screen, which allows for real-time processing of read patterns and saves all metadata for further analysis.

To solve the problem of verifying reading using computer technology, the claimed invention had to transform a common standard video device, in particular a webcam, into an eye-tracking device. For these purposes, machine learning was applied to build a system of eye gaze tracking and recognition of specific eye movements during reading, together with the introduction of new elements and improvement of the known ones, as well as through new connections and a certain sequence of actions, it was possible to effectively solve the problem of reading verification.

The implementation of the invention provides for the use of conventional video devices, in particular, connected, built-in or integrated into any electronic devices, for example a tablet, smartphone, laptop, personal computer or other suitable, which requires the creation of an adapted hardware and software complex (AIC), where the choice of data is also important for video recording, video recording method, data processing method, transmission and selection of criteria for interpretation.

At the beginning, work was carried out on the analysis of the subject's visual activity, where, according to the gaze, read / non-reading was classified, regressions, sweeps and saccades were determined, and many individual views were mapped on the structural elements of multimedia information / document. As a suitable tool for this used the program (in particular as described, Bohomaz R., Chaus M., Melnychuk V. Machine Learning Techniques in Reading Tracking. Meet your [STORYLOOK] / R. Bohomaz, M. Chaus, V. Melnychuk. - Beehiveor Academy and R & D Labs partnered with DataRoot Labs. - Oct 19 2018 https: //medium.eom/@BeehiveorGroup/machine-learning-techniques-in-reading-tracking-meet- your-storylook-6ac760d8a33d), for the creation of which used mainly open source libraries.

All patents, published applications, and references cited in the specification are incorporated by reference, embracing their entirety.

Data collection.

When collecting data to track human activity, a tracker was used, for example GazePoint eye, which made it possible to obtain gaze coordinates with an error angle of no more than 1-1.5 degrees after calibration. During each session, GazePoint Analysis records video and screen images along with tabular gaze movement data.

The entire data set consisted of 2 parts: 51 time series of reads and 85 - "non-reads". Each study participant had to do the following: read 2- minute text; find specific information and things in the images, watch a 3-minute video.

The resulting time series consist of several columns (Fig. 2):

- FPOGX, FPOGY - screen coordinates, relative to screen resolution, algorithm A,

- BPOGX, BPOGY - screen coordinates, relative to screen resolution, algorithm B,

- FPOGID - commit identifier,

- FPOGD - duration of eye fixation,

- BPOGV, FPOGV - information validity,

- BKID - blinking identifier.

Data preprocessing and function selection.

During our research, we found that the tracked coordinates may not be perfect. Blinking, head movements, variable lighting - all these factors interrupt or spoil the data flow. Thus, additional steps were taken that could alleviate the situation, smoothing of gaze movements was adopted as the main option, although the significance of this approach has a limit. Anti-aliasing eliminates one important feature - microsaccades. A saccade is a fast, simultaneous movement of both eyes between two points. Microsaccades are movement within a single fixation that provides an answer to how users fix their gaze. While anti-aliasing is not an ideal choice when it comes to saccades, it does help approximate word detection.

Here's what filtering meant for these research purposes (Fig. 3,4):

- filtering using BPOGV, FPOGV,

- filtering only gaze movements on the screen.

In order to easily manipulate the dataset and training / test models, time sequence processing windows with a width of 100 values were chosen (which roughly corresponds to the average time for reading one section on A4 paper). This resulted in splitting all datasets into 24,568 reads and 14288 non-reads of many time series with a length of 100 values, allowing for 90% overlap.

Reading / non-reading classification.

Further, three main methods of classifying time series were used. Three groups of functions were created: linear trend detection for FPOGX, seasonality and general characteristics, however, the selected functions were not informative and did not describe the data well.

Convolutional neural network was applied next. The functions dX, dY were used. After some tuning, the optimal architecture with 113,006 learning parameters was found (Fig. 5). This model gave 96% accuracy on a subset of tests and was subsequently chosen as the baseline model for further research.

Reading clustering models. The main task was to classify each group of fixations (observations, grouped by FPOGID) as one of three main patterns: saccades, line transitions, regressions (Fig. 6).

The main obstacle that emerged at this stage was the labeling of the dataset, since 60 Hz data is inherently difficult to label. This also turned out to be a problem for clustering. Some of the minor issues that have been addressed are the high similarity between regressions and line transitions, and scrolling fixes that appear to be outliers. The reading classification algorithm was used to exclude fixations during scrolling.

The entire dataset was filtered from points to saccades only. To obtain saccades data, we grouped the points with the fixation identifier (FPOGID) and selected only the last observation from each group. As a result, all the identified saccades were divided into minimum / maximum values with a naive algorithm along the horizontal axis, and the minimum saccades were divided into groups of line transitions and regressions. To achieve the necessary results, clustering was used using the K-means method on three main functions of saccades: the projection of line transitions along the horizontal axis, the angle of line transitions along the horizontal axis and the difference from the previous saccade.

One of the examples of automated processing of multimedia information is the creation of a complete text document with video in the following sequence: identification of static frames in the video (with a threshold deviation between images), separation of sheets (text fragments) from the frame, the following are combined sheets in a single "panoramic" picture, separating text into the displayed document, identifying points of interest.

As a result, we managed to obtain a machine learning model capable of predicting with an accuracy of 97% whether the user read or did not read the text within 1.6 seconds of recording.

Some of the main findings of the study show that an algorithm that builds on the foresight of the previous machine learning (ML) model can account for gaze movements (regressions, sweeps, and saccades) and calculate relative reading speed; an algorithm that provides information about a reader's interest provides a weighting factor for a given word, which may be important to the reader.

Scientists researching human behavior such as eye tracking may discover previously blocked areas in health and business technology. As justified by this algorithm, it is possible to make real-time predictions based on gaze-watching technologies, and possibly go beyond that with more scientific applications that no one thought possible before. Determination of general reading parameters without a clear separation of that part of the text information that was actually read does not allow to predict with high accuracy or establish the volume of the recognized text information content among all multimedia or all text information, the difficulty of its assimilation, the separation of fragments that are especially important in terms of content, etc. .d.

With the involvement of the above, an automated system has been created for implementing a method for determining the amount of information recognition by a subject when presenting multimedia information, in particular, determining the amount of reading by a subject of structural elements of multimedia information or an electronic document defined as text. The selected parameters for the targeted training of the artificial neural network allow the subsequent determination of the reliability of attributing each individual gaze of the subject to reading. At the same time, according to the data of the spatial and temporal characteristics of eye movement, taking into account behavioral reactions, it is possible to distribute sections of text fragments of multimedia information into groups of "attentive reading", "viewing or introductory reading", "scanning reading" and "scrolling" and separate the sections for further analysis text fragments highlighted as "careful reading" or actually reading.

An automated method for determining the amount of information recognition by a subject is carried out as follows.

With the help of the eye calibration unit (90) in the APC-P (1), the parameters of the eyes are calibrated taking into account the individual anatomical parameters of the eyes when the subject is presented with moving visual stimuli and the task of tracking. Calibration can be performed both once, and several times, or multiple times as necessary, in particular at a certain frequency or before each session of presenting the multimedia information to the subject.

The connection of the interaction of the subject with the multimedia information is confirmed by verifying the identity of the subject in the module (61), before providing the subject with the multimedia information based on the visual data about the subject, which is entered into the remote APC-VP, once or several times, or multiple times, for example, at a certain frequency or before each subsequent presentation of the multimedia information to the subject, or after interruption of the browsing session for a predetermined time before each presentation of the multimedia information after such interruption.

Identification / verification of the identity of the subject can occur by entering data about the subject into the database of the data storage module of the corresponding device in advance or during the first access of the subject to the APC-P. In subsequent sessions, the subject's access to the multimedia information in relation to which the determination of the recognition and / or reading volume in any particular mode is provided is compared parameters of the subject with those available in the database during the entire period of the session or sessions of information processing by the subject. When authorizing in the system at the first call, to simplify the identification / verification mechanism and reduce the time for identification, the system can be used to log into the system using existing accounts, including social media accounts, etc., including audio and / or photo and / or video captured data of the subject, in particular his appearance, for example, facial features and the like.

After calibration, or before it, or during calibration, using the module for receiving and transmitting data, configured for receiving and transmitting information (31), multimedia information is loaded onto the remote APC-VP (2) and using the fragment classification unit (91) is classified its fragments into structural elements based on characteristics - text, images, table, etc. (all other information other than text, images or tables).

Then, the classified information is transmitted from the remote APK-VP (2) to the APK-P

(1) and present it to the subject on the multimedia information display module (20).

The display of multimedia information can occur on any suitable display means, for example, a TV screen, a computer or laptop monitor, a smartphone display, a tablet display, and the like.

With the help of the video recording device (10) of the video stream, in the recording unit (50) of the APK-P (1), the data of eye movements and / or gaze is recorded, taking into account behavioral reactions in the form of at least one of blinking eyes and / or head movements, and / or changes in the coordinates of pre-marked points on the face, and in the data processing unit of the video stream (80) of the data processing module (70) using the unit for determining individual points (100), individual points of view are determined at each time point in the plane coordinates of the multimedia information display unit (20 ) APK-P (1) when presenting multimedia information.

It is important that any video device capable of receiving and transmitting video data is used as a video recording device, in particular, a webcam or video camera connected or integrated into the APK-P (1), in particular, it can be used connected to a personal electronic device ( personal computer, telephone, smartphone, tablet, etc.) photo and / or video camera or built into the specified device, or other.

The information received by any known method (wired or wireless) using modules for receiving and transmitting data (30) and (31) is transmitted to a remote APC-VP

(2) and further to the data processing module (71).

Using the unit of work and training of the artificial neural network (81) and the unit of establishment of correspondence (111), a correspondence is found between the individual points of view and the fragment of multimedia information presented at this time, then with using the artificial neural network of the unit (81), data of the set of views is generated and, using the validation unit (121), the probability of the relationship of each individual view of the subject to reading is determined.

The specified data of a set of gaze is aggregated into a coordinate system of classified structural elements of the presented multimedia information in the unit for generating and aggregating a set of gaze data (131).

Further, in the data processing module (71), the data density of all views and the data density of the views defined as reading are calculated in the calculation unit (151) by the number in each structural element, and the corresponding threshold values for the calculated densities are set, which separates the statistical noise.

Additionally, in the data processing module (71), optionally in a separate block, which is associated with blocks (81) - (181), determine the number of transitions from a certain text to related materials, in particular tables and / or images, and / or another, and the time spent on such transitions, which are taken into account when determining behavioral reactions when reading and "not reading".

Then, in the calculating unit, the ratio of the overlapping area of the area (151) determines the ratio of the area of overlap of the area where the data density of all views and the data density of the views, defined as reading, exceeds the threshold values, to the total area of all structural elements defined in the presented multimedia information, and set the amount of information recognized by the subject using the unit (161) for establishing the amount of information recognized by the subject, after which the read information is determined using the unit (171) for establishing the amount of textual information read by the subject.

Multimedia information to be processed by a subject is any kind of information capable of being displayed on various devices, and may contain text or text fragments (in particular, for example, inscriptions, numbers, tables, images, text fragments without images and / or tables, or with images and / or tables, in well-known electronic document presentation formats, for example, it is an electronic document, an electronic book (for example, a production document, educational material, electronic testing material, entertainment material, etc.) and / or video information that contains text fragments and / or images, and / or tables.After processing in block (91) the multimedia information loaded into the APC-VP, its classification is carried out by separating from the entire volume of content fragments containing text features that are text fragments, tables (with text elements and numbers) and images, in particular, to which are associated with text and / or table fragments. The multimedia information is presented to the subject in a conventional format, but the preliminary classification allows focusing on behavioral responses and eye movements in relation to the targeted pieces of multimedia information.

The remote APC-IP also provides the ability to present the results of data processing in relation to the volume of recognition and / or reading of multimedia information on the display device (20) of the user in a different way. The amount and type of presentation of this data can be determined by the operator of the remote APC-VP and / or the owner of this multimedia information, and / or the user, or otherwise.

In particular, in the unit for presenting statistical information (181), data is generated on the behavior of the subject during reading, in particular, the amount of information read, the speed of reading, the complexity of the perception of information, data on various aspects of eye movement, and then a report is generated based on these data, in particular, a read verification certificate, in particular as an electronic document, upon completion of each session or series of sessions of presenting multimedia information or a fragment thereof.

In particular, on the module for displaying multimedia information (20) and / or (21), a generated report and / or a map is presented in any visually acceptable form and / or a diagram, and / or a graph or another, of gaze density and / or gaze data density, defined as reading, and / or the area of overlap of the area where the data density of all gazes and the data density of gazes, defined as reading, exceeds the threshold values, and / or the calculated percentage of viewed structural elements from the total number of structural elements when presenting multimedia information in which the gaze was not defined as reading, and / or the calculated percentage of structural elements in which gaze data was defined as reading, of the total number of structural elements in the presentation of multimedia information and / or the calculated ratio of the volume of viewed structural elements in which gaze was not defined as reading , to the volume of structural elements in which the gaze data were Defined as reading, presenting multimedia information, and the like.

Such visualization of the results of recognition and / or reading of multimedia information and the speed of these processes can occur both at the end of a session of presenting multimedia information, and during the session, upon completion of viewing certain specified fragments of multimedia information in real time. This is an additional tool for controlling the recognition and / or reading process for the user and / or the owner of multimedia information, and / or the operator of the separated APC-VP, and develops into a number of additional capabilities of the method and system described here.

The application of a machine learning approach makes the claimed multimedia familiarization and reading verification method easily applicable and standard on a variety of devices. Thanks to the claimed method and system in the mode real-time using any means of video recording, it is possible to identify with 97% accuracy whether a person is reading at the moment and calculate the amount of information actually read.

The data obtained in this way on the volume and / or speed of recognition and reading of information makes it possible to expand the possibilities of using the methods and / or systems described herein based on eye tracking technology for various purposes.

So, taking into account the possibility of clear identification / verification of the identity of the subject in connection with specific multimedia information, for example, an electronic document that he views, recognizes and reads, it becomes possible for an independent automated confirmation of the familiarization of this subject with the document and confirmation of the amount of information read, which allows a high degree of reliability to assert about the depth of cognitive processing of information by a specific subject. This is useful in various educational processes, when assessing certain professional skills, assessing remote interaction between personnel within one enterprise or between different structures, in particular the public sector, in legal practice, and the like. In addition, the relationship between the identity of the subject, the specific electronic document that he is familiar with, and the amount of content read in this document allows for automated and easily accessible certification of the reading of the document, in particular by creating a document signing certificate, which can be applied in many industries.

Data exchange between APK-P and APK-VP is carried out using wired or wireless transmission, while the settings of APK-VP allows you to expand the system and carry out such data exchange with a variety of APK-P, and also reduces the functional and software load on the APK-P , all the received data of the subject are stored in the data storage unit (41) of the APK-VP, and optionally a part of the data, in particular reports and / or maps and / or certificates, in the APK-P.

While the described embodiments were based on preferred embodiments, those skilled in the art will appreciate that various changes in form and detail can be made without departing from the scope of the embodiments encompassed by the appended claims.

Claims

CLAIM 1. An automated method for determining the amount of information recognition by a subject when presenting multimedia information, providing for the presentation of multimedia information, fixing and processing data on the subject's eye movement using a video fixation device and determining the amount of recognition by the subject of the presented multimedia information, which is characterized in that the method is carried out using the system including - a user's hardware and software complex (APK-P), in which a video recording device, a multimedia information display module, a data reception and transmission module, a recording module from a video recording device of video stream data, a data storage module, an automated data processing module are combined or integrated, and - a remote hardware and software complex for interaction with a user (AIC-VP), in which a module for visual presentation of multimedia information, a module for receiving and transmitting data, a data storage module, a verification module for a user subject, a data processing module, are combined or integrated, configured for work and training of an artificial neural network, and the method includes the following steps: - in APK-P, the calibration of the parameters of the eyes is carried out when the test information is presented to the subject in the form of movable visual stimuli; - download multimedia information to a remote APC-VP and classify its fragments into structural elements according to features - text, images, table, etc. - transfer the classified information from the remote APK-VP to the APK-P and present it to the subject; - a video stream is obtained using a video recording device, according to which data of eye and / or gaze movements are recorded in the APK-P recording module, taking into account behavioral reactions in the form of at least one of blinking eyes and / or head movements, and / or changes in coordinates in advance marked points on the face, define individual points of view at each time point in the plane coordinates of the multimedia information display unit APC-P when presenting multimedia information; - transmit the received information to the remote APK-VP, in which: - establish a correspondence between individual points of view and a piece of multimedia information presented at this time; - with the use of an artificial neural network, data from a set of views are formed and the probability of referring each individual view of the subject to reading is established; - aggregate the specified data of multiple views into a coordinate system of classified structural elements of the presented multimedia information; - calculate from the number in each structural element the data density of all gazes and the gaze data density defined as reading, and set appropriate threshold values for the calculated densities that separate the statistical noise; - calculate the ratio of the overlapping area of the area where the data density of all gazes and the gaze data density, defined as reading, exceeds the threshold values, to the total area of all structural elements defined in the presented multimedia information, and the amount of recognized information is determined by the subject, according to which the information is defined as read. 2. The method according to claim 1, characterized in that the multimedia information presented to the subject is an electronic document, text fragments without images and / or tables, or with images and / or tables in known formats for presenting electronic documents, and / or video information, containing text fragments and / or images and / or tables. 3. The method according to claim 1 or 2, characterized in that it includes an additional step, in which using the verification module, before presenting the multimedia information to the subject, the identity of the subject is verified on the basis of visual data about the subject, which is entered into the remote APC-VK. 4. The method according to claim 3, characterized in that the identity of the subject is verified before each subsequent presentation of the multimedia information or electronic document to the subject. 5. The method according to any one of paragraphs. 1-4, characterized in that the calibration of the parameters of the eyes and certain behavioral reactions is carried out before each session of presentation of multimedia information to the subject. 6. The method according to any one of paragraphs. 1-5, characterized in that the multimedia information display unit is a TV screen, a computer monitor, a smartphone display, a tablet display, or any other device capable of presenting multimedia information. 7. The method according to any one of paragraphs. 1-6, characterized in that the video recording device is, in particular, a webcam or video camera connected or integrated into the APC-P, in particular into a computer, smartphone, tablet or any video device capable of receiving and transmitting video data. 8. The method according to any one of paragraphs. 1-7, characterized in that they form a map of gaze density and / or gaze data density defined as reading and / or overlapping area of an area where the data density of all gaze and the density of gaze data defined as reading exceeds threshold values and does not necessarily represent on the multimedia display module. 9. The method according to any one of paragraphs. 1-8, characterized in that the percentage of the viewed structural elements of the total number of structural elements in the presentation of multimedia information, in which the gaze has not been determined as reading, is calculated and is optionally presented on the multimedia information display unit. 10. The method according to any one of paragraphs. 1-8, characterized in that the percentage of structural elements in which gaze data has been determined to be read out of the total number of structural elements when presenting multimedia information is calculated, and is optionally presented on the multimedia information display unit. 11. The method according to any one of paragraphs. 9-10, characterized in that the ratio of the volume of the viewed structural elements, in which the gaze was not defined as reading, to the volume of the structural elements, in which the gaze data was defined as reading, is calculated when presenting multimedia information and is optionally presented on the multimedia information display module ... 12. The method according to any one of paragraphs. 1-11, characterized in that at the end of the session of presentation of multimedia information or a fragment thereof, a read verification certificate is generated in the APC-VP data processing module, in particular as an electronic document, and is optionally stored in the data storage module. 13. The method according to claim 12, characterized in that the reading verification certificate is loaded onto the APK-P. 14. The method according to any one of paragraphs. 1-13, characterized in that all the received data of the subject is stored in the data storage unit APC-VP and optionally in APC-P. 15. An automated system for determining the amount of information recognition by a subject when presenting multimedia information, comprising a video fixation device, a multimedia information display module, a fixation, transmission and processing module for data on the subject's eye movement, a data processing module, characterized in that the system includes: a) the user's hardware and software complex (AIC) (AIC-P), in which the following are combined or integrated: i) video recording device, ii) module for displaying multimedia information, iii) a module for receiving and transmitting data, configured to receive and transmit the received information to a remote APC for interacting with a user (APC-VP), iv) a recording module from a video recording device of video stream data concerning eye movements and / or gaze, and behavioral reactions in the form of at least one of blinking eyes and / or head movements and / or changes in coordinates of previously marked points on the face, v) data storage module, vi) data processing module, including: - a block for processing video stream data from a video recording device APK-P, - block for calibrating eye parameters when presenting multimedia information, - a block for determining individual points of view at each moment in time in the coordinates of the plane of the multimedia information display module when presenting information, and b) a remote APK for user interaction (APK-VP), in which the following are combined or integrated: i) module for displaying multimedia information, ii) a module for receiving and transmitting data, configured to receive and transmit the received and processed information to the APC-P, iii) data storage module, iv) a module for verifying the identity of the user subject, v) a data processing module capable of operating and training an artificial neural network, including: - block of work and training of an artificial neural network, - a block for classifying fragments of multimedia information into structural elements based on the characteristics of text, images, tables, etc. - a block for processing video stream data from a video recording device APK-P, - a unit for establishing a correspondence between individual points of view and a piece of multimedia information presented at this time; - block for establishing the reliability of the attitude of each individual gaze of the subject to reading, - a block for generating data of multiple views and aggregating data of multiple views into a coordinate system of classified structural elements of the presented multimedia information, - a block for calculating the number in each structural element of the data density of all views and the data density of the views defined as reading, and setting the corresponding threshold values for the calculated densities that separate the statistical noise, - a unit for calculating the ratio of the overlapping area of the area where the data density of all views and the data density of the views, defined as reading, exceeds the threshold values, to the total area of all structural elements defined in the presented multimedia information, - block for establishing the amount of information recognized by the subject, - block for establishing the amount of textual information read by the subject, - block for presenting statistical information and generating reports. 16. The system according to claim 15, characterized in that the APC-VP further comprises a module for controlling the modes of presentation of multimedia information on the APC-P. 17. The system according to PP. 15 or 16, characterized in that the module for displaying multimedia information APC-P and / or APC-VP is configured to present multimedia information to a subject, which is an electronic document, text fragments without images and / or tables or with images and / or tables in well-known electronic document presentation formats or video information that contains text fragments and / or images and / or tables. 18. System according to any one of paragraphs. 15-17, characterized in that the module for verifying the identity of the user subject is configured for verification based on visual data about the subject, which are entered into the remote APC-VP, before the first and / or before each session of presenting the multimedia information or electronic document to the subject. 19. System according to any one of paragraphs. 15-18, characterized in that the unit for calibrating the parameters of the eyes is adapted to calibrate the parameters of the eyes and certain behavioral responses before the first and / or before each session of presentation of multimedia information or electronic document to the subject. 20. System according to any one of paragraphs. 15-19, characterized in that the module for displaying multimedia information or electronic document in the APC-P and / or APC-VP are a TV screen, a computer monitor, a smartphone display, a tablet display or any other device capable of displaying multimedia information. 21. System according to any one of paragraphs. 15-20, characterized in that the video fixing device is, in particular, a webcam or video camera connected or integrated into the APC-P, in particular into a computer, smartphone, tablet or any video device capable of receiving and transmitting video data. 22. System according to any one of paragraphs. 15-21, characterized in that the APC-VP data processing module contains a unit for forming a map of the gaze data density and / or gaze data density, defined as reading and / or overlapping area of the area, where the data density of all gazes and the gaze data density, defined as reading exceeds the thresholds, and the card is suitable for presentation on the media display module. 23. System according to any one of paragraphs. 15-22, characterized in that the APC-VP data processing module contains a unit for calculating the percentage of viewed structural elements from the total number of structural elements when presenting multimedia information, in which a glance has not been determined as reading. 24. System according to any one of paragraphs. 15-23, characterized in that the APC-VP data processing module contains a unit for calculating the percentage of structural elements in which the gaze was determined as reading, of the total number of structural elements when presenting multimedia information. 25. System according to any one of paragraphs. 15-24, characterized in that the APC-VP data processing module contains a unit for calculating the ratio of the volume of viewed structural elements, in which the gaze was not defined as reading, to the volume of structural elements, in which the gaze was determined as reading, when presenting multimedia information. 26. System according to any one of paragraphs. 15-25, characterized in that the APC-VP data processing module contains a unit for generating a read verification certificate, in particular as an electronic document, upon completion of each session or series of sessions for presenting multimedia information or a fragment thereof. 27. An automated method for verifying the familiarization of a subject with an electronic document, which differs in that it involves the preliminary establishment of the volume of information recognition, in particular the volume of a read document, sufficient to generate a certificate of signing an electronic document by a verified subject, verification of the subject's identity based on previously entered into the hardware and software a complex (AIC) of visual data about a subject at the beginning of a session of presenting an electronic document to a subject and during each subsequent session of presenting the same document to the same subject, determining the volume of information recognition, in particular the volume of a read document by a verified subject when presenting an electronic document for all sessions of presenting one and the same document using eye tracking technology on the AIC using the method according to any one of claims 1-14 and / or the system according to any one of claims 15-26, and in case the sub The object confirms the completion of the session / sessions of acquaintance with the document and agrees on the signing, and if a certain amount of information recognized by the verified subject, including the volume of the read document, is equal to or greater than the predetermined volume, a document signing certificate is generated. 28. The method according to claim 27, characterized in that the determination of the volume of information recognition, in particular the volume of the read document by the verified subject, is carried out using the method according to any one of claims 1-14. 29. The method according to PP. 27 or 28, characterized in that the determination of the volume of information recognition, in particular the volume of the read document by the verified subject, is carried out using the system according to any one of claims 15 to 26.