RU2768545C1 - Method and system for recognition of the emotional state of employees - Google Patents

Abstract

FIELD: computer technology.SUBSTANCE: invention relates to the field of computer technology, in particular, to methods and systems for recognizing faces and emotions based on a video stream, and can be used to monitor the emotional state of employees in organizations. The technical result of the proposed technical solution is achieved by performing face recognition on frames received from the video stream, classifying emotions using the first neural network; compare the face image with the database of employees to obtain the employee identification coefficient based on the descriptors, determine the coefficient of the employees' emotional state by counting the aggregated emotions of the employees and the performance indicators of the employees, display the obtained coefficient of the employees' emotional state on the information media screen.EFFECT: improving the accuracy of recognition of the emotional state of employees.7 cl, 6 dwg

Description

FIELD OF TECHNOLOGY

The present technical solution relates to computer technology, and in particular to methods and systems for recognizing faces and emotions based on a video stream and can be used to monitor the emotional state of employees in organizations.

BACKGROUND OF THE INVENTION

The closest analogue is the source of information US 10,496,947 B1, published on 03.12.2019, which reveals a method and an emotion recognition system for personnel analytics. The method includes the steps of receiving a video stream including a sequence of images, detecting a person in one or more images, determining anchor points of the person's features, aligning a virtual face grid with respect to the person in one or more images, based at least in part on control points, dynamically determining from the sequence of images at least one deformation of the virtual face mesh, determining that at least one deformation relates to at least one facial emotion selected from a plurality of reference facial emotions.

The proposed solution differs from the prior art solution in that it does not require a person to be near the camera for a more accurate recognition of the employee's emotional state.

SUMMARY OF THE INVENTION

The technical problem to be solved by the claimed solution is the development of a method and system for recognizing the emotional state of employees, described in independent claims. Additional embodiments of the present invention are presented in dependent claims.

The technical result is to increase the accuracy of recognition of the emotional state of employees. An additional technical result is an increase in the performance of the computing system when solving the task (i.e., it allows processing to obtain a result (product) in less time), thereby reducing the load on the central processor of the computing device, by reducing the number of processed requests.

The claimed technical result is achieved through the operation of a computer-implemented method for recognizing the emotional state of employees, running on a computing device that contains a processor and memory that stores instructions executed by the processor and includes the following steps:

at least one frame from the video stream from at least one video camera is transmitted to the computing device, to the recognition subsystem;

recognizing at least one face on frames received from the video stream by means of the face recognition module of the recognition subsystem, using a linear classifier operating on the basis of the input extracted features of the frame;

classifying the emotions of at least one person recognized in the previous step by means of the emotion classification module of the recognition subsystem using the first neural network;

comparing the face image recognized by the recognition module with the employee database by means of the recognition subsystem identification module based on the descriptors calculated using the second neural network and transmitting the received data to the server;

by means of the server, the coefficient of the emotional state of employees is determined by calculating the aggregated metrics of employees' emotions, recognized by the emotions classification module, and the performance indicators of employees, aggregated over the set time intervals, and transmitting the received data to the client server;

by means of the client server, the obtained coefficient of the emotional state of the employees is displayed on the information media screen.

In a particular implementation of the proposed method, the input extracted frame features are an 80×80 frame area descriptor calculated using the oriented gradient histogram (HOG) method.

In another particular embodiment of the proposed method, the emotional state of the employee is defined as positive or neutral.

In another particular embodiment of the proposed method, the positive emotional state of the employee is recognized when the employee in the frame smiles.

In another particular embodiment of the proposed method, when classifying emotions of at least one person, when the first neural network is running, the convolution operation is divided into convolution in width and depth.

In another particular embodiment of the proposed method, the Euclidean distance between face image descriptors is used as a measure of similarity when comparing a recognized face with a database of employees.

The claimed technical result is also achieved through the implementation of a system for recognizing the emotional state of employees, containing:

at least one video camera configured to record the video stream and transmit at least one frame from the video stream to the computing device;

recognition subsystem, including:

a face recognition module configured to recognize at least one face in frames received from the video stream and transfer the recognized face to the emotion classification module and to the identification module,

an emotion classification module configured to classify the emotions of at least one face recognized in the frame in the previous step,

an identification module, configured to match the face image recognized by the face recognition module with a database of employees;

database;

a server configured to record the emotion classification results of the at least one recognized face into a database, determine the employees' emotional state coefficient, and transmit the employees' emotional state coefficient to the client server;

a client server configured to transmit the coefficient of the emotional state of employees to the information media screen;

information media screen, configured to display the coefficient of the emotional state of employees.

DESCRIPTION OF THE DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

Fig. 1 illustrates a block diagram of the operation of the proposed computer-implemented method for recognizing the emotional state of employees.

Fig. 2 illustrates the architectural scheme of an embodiment of a system for recognizing the emotional state of employees.

Fig. 3 illustrates an architectural diagram of an embodiment of a recognition subsystem.

Fig. 4 illustrates a flowchart of neural network operations for emotion type classification.

Fig. 5 illustrates a block diagram of a type 2 convolutional block of a neural network for emotion type classification.

Fig. 6 illustrates the operation of the computing device.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention can be used, both with and without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to unnecessarily obscure the features of the present invention.

Moreover, it will be clear from the foregoing that the invention is not limited to the present implementation. Numerous possible modifications, changes, variations and substitutions that retain the spirit and form of the present invention will be apparent to those skilled in the subject area.

The following terms and definitions apply in this application, unless otherwise expressly stated. References to techniques used in the description of this invention refer to well-known methods, including modifications of these methods and replacing them with equivalent methods known to those skilled in the art.

On FIG. Figure 1 illustrates a block diagram of the operation of the proposed computer-implemented method for recognizing the emotional state of employees.

From at least one video surveillance camera, a video stream arrives at the computing device by means of network interaction. The recommended image quality is at least 720p (frame size 1280×720 pixels). The frame rate can be 20 frames per second or higher, depending on the computing power of the data processing hardware.

From the received video stream, at least one current frame from the video surveillance camera is obtained in the form of a color image in RGB format (step 105) and the current frame is transmitted to the recognition subsystem (210).

At step 110, at least one face is recognized on frames received from the video stream by means of the face recognition module (310) of the recognition subsystem (210), using a linear classifier operating on the basis of the input extracted features of the frame.

Recognition of at least one face consists in selecting rectangular areas of the image of the minimum size on frames containing the entire face of a person. Recognition of at least one face is carried out using a linear support vector classifier (SVM) trained on a labeled set of examples, namely, a sliding window is applied to the input image. The input information for the classifier model is the vector of numerical features calculated on the basis of the image. In one implementation, a histogram of oriented gradients (HOG) method is used to compute numeric features, comprising the following steps.

1. The resulting color frame image is converted into a grayscale image using known methods.

2. Calculate horizontal and vertical gradients using the formulas

where I(x, y) is the intensity value of the image pixel with x, y coordinates, G is the length of the gradient vector, θ is the direction angle of the vector.

3. An 80×80 pixel image window is divided into 8×8 pixel cells.

Длины векторов градиента, сгруппированные по направлению, суммируются.4. For each cell, a histogram of 9 elements is calculated. The elements of the histogram denote the direction angles of the gradient vectors with a step

The lengths of the gradient vectors grouped by direction are summed.

5. The histograms are concatenated in blocks of neighboring cells sized 2×2. The vector obtained for each block, with a size of 36 elements, is normalized: the operation of dividing the vector elements by the vector norm is performed.

6. The vectors are concatenated over all blocks of the window. The resulting vector thus obtained is a descriptor - a feature vector that describes a frame image fragment.

The descriptor is fed to the input of the SVM linear classifier, the output is the probability of finding a person's face in the considered window, namely: a binary sign of the presence of a face in the image.

7. Steps 3-7 are repeated for the next window, obtained by shifting 1 pixel horizontally and vertically.

8. As areas containing faces, windows with a probability exceeding the experimentally selected threshold value are selected.

One implementation uses examples from the LFW dataset described in Huang G. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments, 2008 to train a linear SVM classifier.

In addition, one of the open source implementations of the face recognition algorithm can be used, for example, from the Dlib library described in King D. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009).

The training examples are pairs: an 80×80 image area descriptor calculated using the HOG method and a binary sign of the presence of a face.

The result of the face recognition module is transmitted to the emotion classification module by means of network interaction.

After at least one face has been recognized in the frame, at step 115, emotions are classified for at least one face recognized by the emotion classification module (315) of the recognition subsystem (210) using the first convolutional neural network.

There are two types of emotional state: positive, in the case when the person in the frame is smiling, and neutral otherwise. Determining the smile of a person in the frame, as already described above, is carried out by using the first neural network, which determines the location of key points in the area of the eyes, cheeks, lips and movements of the facial muscles on the face image. When a person smiles, this process is accompanied by a characteristic facial expression: the corners of the lips rise and at the same time the cheeks rise, the points around the eyes shift. When determining an emotion, the neural network analyzes the face area in the image to identify the above features, namely, but not limited to: the height of the corners of the lips, the rise of the cheeks, the narrowing of the eyes.

To classify the emotions of at least one recognized face, in the frame of the video stream, the first neural network of the convolutional type is used. In order to reduce the number of model parameters to be trained in order to reduce the requirements for the size of the training sample and speed up the operation of the network in the operational mode, when choosing the model architecture, an approach can be used with the division of the convolution operation into convolution in width and depth, for example, similar to that described in the article Chollet F. Xception : Deep Learning with Depthwise Separable Convolutions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017).

The classification of the type of emotional state of a person according to the face image using the first convolutional neural network includes the following steps:

1. Converting a color RGB image of a frame into a grayscale image using known methods.

2. Scale the image to the target size of 64×64.

3. Image normalization with mathematical expectation and standard deviation calculated for the training sample.

4. Performing the transformation of the normalized single-channel image with a size of 64×64 by the first convolutional neural network.

A flowchart for one implementation is shown in FIG. 4. The first convolutional neural network receives a single-channel face image as input (405). Convolutional blocks (410), (415), (420) are applied to the image sequentially. Type 1 convolutional block is applied twice and consists of 8 3×3 pixel size convolutional filters applied to feature map input channels in 1 pixel steps. The convolution operation is described by the formula:

- вес фильтра I в позиции (i, j) для канала с. К выходным значениям признаков применяется функция активации видаwhere K is the filter size, C is the number of feature input channels, L is the number of output channels, x _ijc are feature values at position (i, j) in channel c,

- weight of the filter I at position (i, j) for channel c. An activation function of the form is applied to the output values of the features

Type 2 convolution block is applied 4 times with different number of filters. In one implementation, 16, 32, 64, 128 filters are used, however, one skilled in the art will appreciate that other filters may be used.

A type 2 convolutional block diagram for one implementation is shown in FIG. 5. 2 types of transformations are applied in parallel to the input feature maps.

Type 1 transformation includes successive double application of P filters (where P is an architectural parameter of the block) of a split convolution (510) and an activation function (515) that was described above, the application of pooling (520) for 2×2 regions with a step of 2 pixel. The split convolution operation is the sequential application of a convolution with a filter size of 3x3 for each channel and a convolution with a filter size of 1x1 between the channels:

веса I-го плоского фильтра, w'_c - веса фильтра свертки каналов 1×1. Пулинг выполняется как замена области 2×2 пикселя на максимальное значение в этой области. В результате применения операции пулинга с шагом 2 размер карты признаков уменьшается в 2 раза. Преобразование типа 2 представляет собой применение свертки с Р фильтрами размера 1×1 с шагом 2 пикселя (525), полученные в результате карты признаков имеют вдвое меньший размер относительно исходных. Выполняется конкатенация каналов (530) для карт признаков, полученных в результате применения преобразований типа 1 и 2. Результирующие карты признаков (535) содержат 2Р каналов.where

weights of the I-th flat filter, w' _c - weights of the 1×1 channel convolution filter. Pooling is done as replacing a 2x2 pixel area with the maximum value in that area. As a result of applying the pooling operation with a step of 2, the size of the feature map is reduced by 2 times. Type 2 transformation is the application of convolution with P filters of size 1×1 with a step of 2 pixels (525), the resulting feature maps are half the size of the original ones. The channels are concatenated (530) for the feature maps resulting from applying type 1 and type 2 transformations. The resulting feature maps (535) contain 2P channels.

The type 3 convolutional block is applied once and represents the execution of convolution operations with 2 filters of size 3 × 3, pooling with the choice of the average value in each of the feature maps, application of the function

to obtain elements p _i of the probability distribution vector of the emotional state type (425). The weights of all convolution filters are learning parameters of the model and are selected using stochastic gradient descent and backpropagation algorithms with minimization of the cross entropy loss function.

5. As a result of the algorithm, the most probable type of emotional state is selected based on the probability distribution vector obtained in step 4. This result is presented as an emotion recognition coefficient - a binary sign of a positive emotion or a neutral emotion.

For example, for a recognized face, a probability distribution vector was obtained for the types of emotional state with elements 0.8 and 0.2. A positive emotional state is assumed: the binary sign of a positive emotion is set to 1. The emotion recognition confidence coefficient is taken equal to the highest probability of 0.8.

At step 120, the face image recognized by the recognition module (310) of the recognition subsystem (210) is compared with the employee database (225) by means of the identification module (320) of the recognition subsystem (210), based on the descriptors calculated using the second neural network.

The result of comparing the image of the recognized face and the image from the database of employees is to obtain the coefficient of identification of the employee, to ensure that the employee is recognized correctly. This coefficient is presented as a binary feature.

For comparison, descriptors can be used - representation of the image as a vector of real numbers of features. To calculate a descriptor with a size of 128 elements, a second convolutional neural network is used as an encoder, trained on an open set of sample photos of famous people. In one implementation, the ResNet neural network architecture described in Kaiming He et al. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.

As a measure of similarity when comparing images of recognized faces, using the face recognition module (310) of the recognition subsystem (210) and faces in the employee database (225), the Euclidean distance between the descriptors of images of recognized faces and faces in the employee database is used (225). The smallest of the calculated distances between the input object (recognized face) and face images in the employee database (225) is compared with the threshold value specified as a parameter. In the case when the smallest distance does not exceed the threshold value, the current image of the recognized face is assumed to be identified from the database of employee images, otherwise the face is considered unidentified.

In the example below, the threshold value is set as the distance between the descriptors equal to 0.6, however, the specialist should be clear that this threshold can be changed up or down. The database of employees contains 2 images of an employee, for one image of a recognized face, the distances between descriptors 0.2 and 0.5 were obtained, the smallest distance 0.2 (to the image of employee 1) does not exceed the threshold 0.6, in this case it is assumed that the image of the recognized face corresponds to the image of employee 1.

For another image of a recognized face, distances of 0.9 and 0.7 were obtained, the smallest distance of 0.7 (in relation to the image of employee 2) is above the threshold of 0.6, so the recognized face is considered not identified: it was not possible to find a sufficiently similar example in the database of employees.

The results obtained in a certain format are transmitted to the server. The received results are transmitted as a text message containing a data structure with the following fields: message ID, camera ID, time stamp, number of recognized faces, number of positive emotions, and the “employee details” field. The "details for employees" field is an array of structures with fields: employee ID, binary sign of positive emotion, employee identification coefficient, emotion recognition coefficient.

The result, in the form of a text message, is transmitted over the network, through networking tools, to the server, in the form of text in JSON format. Other textual formats may also be used, such as XML, or a binary representation of the data for compression purposes.

Data transfer is carried out synchronously via the HTTP protocol or asynchronously using a message queue server that can use protocols, but not limited to, AMQP, Kafka, OpenWire, STOMP, MQTT.

At step 125, by means of the server (220), the coefficient of the emotional state of employees is determined by calculating the aggregated metrics of employee emotions, recognized by the emotion classification module (315) of the recognition subsystem (210), and the performance indicators of employees, aggregated over the set time intervals, and transmitting the received data to the client server (230).

An indicator of the work of employees is the work activity during the working day. The work activity indicator is defined as the number of people passing through the camera's field of view per minute, averaged over the last hour.

The positive level of employees' emotional state is calculated as the ratio of detected cases of positive emotions to the total number of people's faces highlighted in the video stream over the last hour. For example, for the time interval T=1 hour, N=100 observations were obtained, of which in P=20 cases a positive type of emotions was classified, the level of a positive emotional state for the time interval T is calculated as r=P/N=20%. In one embodiment, the employee work activity metric is defined as the result of averaging the number of observations n _i over time intervals t _i over a period of time T

where M is the number of intervals into which the time period T is divided. For example, the number of observations can be averaged over minutes over an hour.

By means of the client server (230), the obtained coefficient of the emotional state of employees is displayed on the information media screen (235) by means of network interaction.

The above-described method is performed by implementing the employee emotional state recognition system shown in FIG. 2 which includes: at least one video camera (205); a recognition subsystem (210) containing a face recognition module (310), an emotion classification module (315), an identification module (320); database (225); server (220); client server (230); information media screen (235).

The recognition subsystem (210) includes:

a face recognition module (310) configured to recognize at least one face in frames received from the video stream and transmit the recognized face to the emotion classification module and to the identification module;

an emotion classification module (315), configured to classify the emotions of at least one face recognized in the frame in the previous step;

an identification module (320) configured to match an image of a face recognized by the face recognition module with a database of employees.

The database (225) is a means of storing recognition data, and also contains data that is used to match the face image recognized by the recognition module of the recognition subsystem with images of employees.

Server (220) configured to record emotion classification results of at least one recognized face in a database, as well as to determine the employee emotional state coefficient and transmit the employee emotional state coefficient to the client server via the HTTP protocol.

Client server (230), configured to transfer the coefficient of the emotional state of employees to the information media screen.

Information media screen (235), configured to display the coefficient of the emotional state of employees. The information media screen is a screen that displays the total number of observations (faces), the proportion of positive emotions from the total number of observations, the level of work activity for the last hour, which is displayed as a temperature in the range of 0-30 degrees, the level of positive emotional state for the last hour, which is displayed in the form of clouds with three gradations: sunny (high level of positive emotions), mostly cloudy (medium level), thunderstorm (low level), as well as the change in the level of work activity by hours during the day.

On FIG. 6, the general scheme of the computing device (600) will be presented below, providing the data processing necessary to implement the claimed solution.

In general, the device (600) contains components such as: one or more processors (601), at least one memory (602), storage media (603), input/output interfaces (604), I/O ( 605), networking tools (606).

The processor (601) of the device performs the basic computing operations necessary for the operation of the device (600) or the functionality of one or more of its components. The processor (601) executes the necessary machine-readable instructions contained in the main memory (602).

The memory (602) is typically in the form of RAM and contains the necessary software logic to provide the desired functionality.

The data storage means (603) can be in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The means (603) allows long-term storage of various types of information, for example, the aforementioned files with user data sets, a database containing records of time intervals measured for each user, user identifiers, etc.

Interfaces (604) are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc.

The choice of interfaces (604) depends on the specific implementation of the device (600), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, and the like.

As means of I/O data (605) in any embodiment of the system that implements the described method, a keyboard can be used. The keyboard hardware can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computer device. In this case, the connection can be either wired, in which the keyboard connection cable is connected to the PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports. In addition to the keyboard, I/O devices can also use: joystick, display (touchscreen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Means of networking (606) are selected from a device that provides network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. With the help of tools (605) the organization of data exchange over a wired or wireless data transmission channel, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM, is provided.

The components of the device (600) are coupled via a common data bus (610).

In these application materials, a preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested legal protection and are obvious to specialists in the relevant field of technology.

Claims (22)

1. A computer-implemented method for recognizing the emotional state of employees, including the following steps: at least one frame from the video stream from at least one video surveillance camera is transmitted to the computing device, to the recognition subsystem; recognizing at least one face on frames received from the video stream by means of a face recognition module using a linear classifier operating on the basis of the input extracted features of the frame; carry out the classification of emotions of at least one person, recognized in the previous step, by means of the module of classification of emotions, using the first neural network; comparing the image of the face recognized by the recognition module with the database of employees to obtain the employee identification coefficient, by means of the identification module, based on the descriptors calculated using the second neural network, and transmitting the results to the server; by means of the server, the coefficient of the emotional state of employees is determined by counting the aggregated, according to the established time intervals, metrics, the emotions of the employees recognized using the emotions classification module, and the performance of the employees, and transmitting the received data to the client server; by means of the client server, the obtained coefficient of the emotional state of the employees is displayed on the information media screen. 2. The method of claim 1, wherein the input extracted frame features are an 80x80 frame area descriptor calculated using the oriented gradient histogram (HOG) method. 3. The method according to claim. 1, characterized in that the emotional state of the employee is defined as positive or neutral. 4. The method according to claim 3, characterized in that the positive emotional state of the employee is recognized when the employee in the frame smiles. 5. The method according to claim 1, characterized in that when classifying the emotions of at least one person, during the operation of the first neural network, the convolution operation is divided into convolution in width and depth. 6. The method according to claim 1, characterized in that the Euclidean distance between face image descriptors is used as a measure of similarity when comparing a recognized face with a database of employees. 7. A system for recognizing the emotional state of employees, containing: at least one video surveillance camera configured to record a video stream and transmit at least one frame from the video stream to a computing device in the recognition subsystem; recognition subsystem, including: - a face recognition module configured to recognize at least one face in frames received from the video stream and transfer the recognized face to the emotion classification module and to the identification module, - an emotion classification module configured to classify the emotions of at least one person recognized in the frame in the previous step, - an identification module configured to match an image of a face recognized by the face recognition module with a database of employees to obtain an employee identification coefficient; database; a server configured to write the recognition results to a database, determine the employees' emotional state coefficient, and transmit the employees' emotional state coefficient to the client server; a client server configured to transmit the coefficient of the emotional state of employees to the information media screen; information media screen, configured to display the coefficient of the emotional state of employees.

Images