CN111898420A - A lip language recognition system - Google Patents

Abstract

The invention discloses a lip language recognition system, which comprises: a human-computer interaction interface and an algorithm module; the human-computer interaction interface is connected with the algorithm module through a signal slot; the human-computer interaction interface is used for acquiring a video to be identified; the algorithm module is used for carrying out lip language identification on the video to be identified to obtain a lip language identification result; and the human-computer interaction interface is also used for displaying the lip language recognition result and displaying the lip language recognition process according to time sequence. The lip language recognition system designed by the invention can better observe and analyze the quality of the model performance and each link from the original video to the final recognition process, thereby improving and optimizing the model and the algorithm.

Description

A lip language recognition system

technical field

The invention relates to the field of lip language recognition, in particular to a lip language recognition system.

Background technique

Lip recognition is to make the computer understand the language content expressed by the speaker through the dynamic shape change of the speaker's lips. Lip recognition technology can solve the task of identifying the content of the speaker in a noisy environment or even without an audio collector, which also makes lip recognition applicable to many scenarios in different fields. Automatic lip reading technology can be widely used in Virtual reality systems, information security, speech recognition and assisted driving systems, etc. With the development of the Internet of Things and 5G technology, lip recognition technology will also be used in smart home, smart driving and smart communication in the future.

The traditional lip language recognition technology is based on image processing and pattern recognition to complete the process from sequence images to recognition results, so the recognition tasks can be divided into: lip region location, lip region of interest feature extraction and lip language content recognition. Among them, feature extraction and classifier design are the key and difficult points of the identification process. The mainstream transformation methods generally used are: Principal Component Analysis (PCA), Discrete Cosine Transformation (DCT), Singular Value Decomposition (SVD) and independent component analysis in signal processing method (Independent Component Correlation Algorithm, ICA). The content recognition of lip language is mainly done by artificially designed classifiers. Classification methods can generally be divided into: Hidden Markov Models (HMM), Artificial Neural Network (ANN), Template Matching Algorithm (TMA) and Support Vector Machine (Support Vector Machine, SVM) etc. The current research on lip language recognition mainly focuses on how to realize lip language recognition, and how to improve the existing lip language recognition process to obtain higher accuracy recognition results. To obtain the reason for the large error of the recognition result, it often requires a lot of experiments to first confirm the error-producing link through the elimination method, and then solve the cause of the error. This process is not only inefficient, but also cannot obtain an intuitive lip language recognition process.

SUMMARY OF THE INVENTION

In order to solve the above deficiencies in the prior art, the present invention provides a lip language recognition system, comprising: a human-computer interaction interface and an algorithm module; the human-computer interaction interface and the algorithm module are connected through a signal slot;

The human-computer interaction interface is used to obtain the video to be identified;

The algorithm module is used to perform lip language recognition on the video to be recognized to obtain a lip language recognition result;

The human-computer interaction interface is also used for displaying the lip language recognition result and displaying the lip language recognition process in time sequence.

Preferably, the algorithm module includes:

The fixed frame extraction sub-module is used to extract the video frame for processing from the video to be identified based on the semi-random fixed frame extraction strategy;

Segmentation sub-module for segmenting lip images from the processed video frames to obtain a lip dataset;

The recognition sub-module is used for recognizing each lip image in the lip data set based on the designed model to obtain a lip language recognition result.

Preferably, the identification submodule includes:

The feature extraction unit is used to perform feature extraction on the lip image to obtain image features, and perform slicing operations on the lip images and image features of each convolution layer to obtain the visualized lip images and high-dimensional image features of each convolution layer;

a time series feature extraction unit, configured to perform time series feature extraction on the image features to obtain sequence features, and perform a slicing operation on the sequence features to obtain visualized sequence features;

The classification unit is used to classify the extracted time series features to obtain a lip language recognition result.

Preferably, the feature extraction unit is a convolutional neural network CNN.

Preferably, the time series feature extraction unit is a recurrent neural network RNN.

Preferably, the classification unit is a softmax classifier.

Preferably, the human-computer interaction interface includes:

Select the video option to obtain the video to be recognized when triggered;

Recognition video option for performing lip language recognition on the video to be recognized when triggered to obtain a recognition result;

The visualization option is used to display the lip language recognition process and the lip language recognition result based on the configuration file set by the visualization requirements when triggered;

Wherein, the displayed content includes the video frame to be identified, the lip image segmented from the video frame, the visualized lip image of each convolutional layer, the visualized high-dimensional image feature, the visualized sequence feature and/or the to-be-identified lip image at least one recognition result corresponding to the video.

Preferably, the fixed frame extraction submodule is specifically used for:

Determine the fixed number of frames to be extracted based on prior conditions;

Divide the video to be identified into a plurality of regional blocks according to the total number of video frames;

where the average of the maximization of the area extent in each area block.

Preferably, the video to be identified includes:

The video to be identified is collected for the same target object based on at least one collection device.

Preferably, the human-computer interaction interface is designed and constructed through the PyQt5 framework.

Compared with the prior art, the beneficial effects of the present invention are:

The lip language recognition system provided by the present invention includes a human-computer interaction interface and an algorithm module; the human-computer interaction interface and the algorithm module are connected through a signal slot; the human-computer interaction interface is used to obtain the video to be recognized; the The algorithm module is used to perform lip language recognition on the video to be recognized to obtain a lip language recognition result; the human-computer interaction interface is also used to display the lip language recognition result, and display the lip language recognition process in time sequence, which can Better observe and analyze the performance of the algorithm module and every link in the process from the video to be recognized to the final recognition, so as to improve and optimize the algorithm module.

Description of drawings

1 is a schematic diagram of a lip language recognition system provided by the present invention;

Fig. 2 is the schematic flow chart of lip language recognition system architecture and algorithm flow in the present invention;

Fig. 3 is the functional schematic diagram that the lip language recognition system has in the present invention;

4 is a schematic diagram of a model architecture of a lip-reading recognition system in the present invention;

5 is a structural diagram of a basic unit of LSTM in the present invention;

6 is a schematic diagram of the probability obtained by the SOFTMAX classifier in the present invention;

Fig. 7 is the human-computer interaction interface of the video lip language recognition system in the present invention;

8 is a schematic diagram of selecting video selection in an operating human-computer interaction interface provided by the present invention;

FIG. 9 is a schematic diagram showing a system identification video result display in the present invention;

Figure 10 is a schematic diagram showing visualization results on the human-computer interaction interface in the present invention;

11 is a schematic diagram of the Loss loss function curve in different periods in the present invention;

Fig. 12 is the schematic diagram of the accuracy rate curve in different periods in the present invention;

FIG. 13 is a schematic diagram of each independent digital pronunciation video segment Recall in the present invention.

Detailed ways

In order to better understand the present invention, the content of the present invention will be further described below with reference to the accompanying drawings and examples.

Due to the wide application scenarios of lip language recognition algorithm, for example, the needs of scientific research algorithms are compared with models, auxiliary pronunciation training, the functional needs of making up for auditory speech recognition, and the in-depth development of biometric technology and XR technology. At present, in the research on lip language recognition, no researcher has completed the engineering design of the lip language feature recognition system, and there is no analytical tool for feature visualization. In order to overcome the current lack of research tools that can analyze visual features and video time series features, the present invention provides a lip language recognition system that can provide computer vision researchers with a good feature engineering visualization operation, not only can deeply observe and supervise the convolution process. Feature engineering can also understand the time series feature changes of the recurrent neural network, which can bring great convenience to subsequent researchers in feature engineering design and more effective feature extraction engineering.

As shown in Figure 1, a lip language recognition system provided by the present invention includes: a human-computer interaction interface and an algorithm module; the human-computer interaction interface and the algorithm module are connected through a signal slot;

The human-computer interaction interface is used to obtain the video to be identified;

The algorithm module is used to perform lip language recognition on the video to be recognized to obtain a lip language recognition result;

The human-computer interaction interface is also used for displaying the lip language recognition result and displaying the lip language recognition process in time sequence.

The lip recognition system uses the PyQt5 framework to complete the design and construction of the human-computer interaction interface, and the algorithm module uses Qt multithreading to complete the functions of information processing and calculation. The requirements of this lip recognition system are defined as visual feature visualization, time series feature visualization and other display process operations. At the same time, it also adds Top 3 accurate analysis of which words are similar. It is necessary to optimize and adjust the data and solutions to avoid pure Risk points affect the research progress and results of lip language recognition. In this embodiment, a lip language recognition system crash and abnormality that affect the normal operation of the program are also designed to capture the abnormality processing mechanism, throw and record the abnormality cause and information, and use it to quickly locate the abnormality cause and improve the repair efficiency.

The technical solution constructs a high-cohesion and low-coupling high-efficiency lip language recognition system in the form of a modular interface code. The lip language recognition system is suitable for Ubuntu, Mac, Windows and other platforms, can recognize lip language in video, especially has a good recognition effect on independent digital pronunciation, the lip language recognition system designed by the present invention can be very It is good to bring convenience and in-depth research to the future research on lip language recognition and feature engineering visualization.

As shown in Figure 2, the overall identification process of the system is shown. The front-end human-computer interaction interface and the back-end module algorithm are connected by means of signal slots, and the call of the video to be identified and the acquisition of the display content are realized through the interface call. The video to be identified passes through the decomposition algorithm, then passes through the designed algorithm model, and finally transmits the final result through the signal slot and feeds it back to each front-end component for result display. In Figure 7, the lip language recognition system on the human-computer interaction interface is specifically displayed. As shown in Figure 7, the interface of the lip language recognition system includes video selection, visualization and recognition video options, and a display frame for the lip language recognition results. Top3 accuracy display frame, facial feature display area, mouth feature display area, convolutional layer 1 and convolutional layer 2 display area, image feature display area and sequence feature display area, in Figure 10 with a specific example Example to show the visualization results.

All the characteristic processes displayed on the front-end human-computer interaction interface can be set by configuration files, which can meet the general system design engineering of high availability and high customization. The lip language recognition system provided by the present invention can not only satisfy beginners' understanding of the process of convolution calculation, but also satisfy researchers' new ideas and methods for processing lip language recognition feature engineering.

The lip language recognition system provided by the invention has the characteristics of high cohesion, low coupling, high availability and high customization. The front-end and back-end are designed in the form of modular interfaces, which have the properties of high cohesion and low coupling. At the same time, the system will not be reconstructed due to partial errors that lead to the paralysis of the whole system. The system model interface has the advantage of updating and iterating. With the in-depth study of lip language recognition technology, the system can continue to be used only by replacing the model parameters and reasoning process. In addition, the back-end model only provides an interface to return the results to the front-end, so it is highly separable and highly usable. Both the visual feature display and the time series feature display are highly customizable and can display any intermediate results you want to display. Figure 3 shows the content displayed on the human-computer interaction interface. The system can display functions such as intermediate feature extraction and lip positioning. At the same time, it can analyze similar pronunciations that are easily confused, which will better pave the way for subsequent research.

The lip language recognition system provided by the present invention is divided into six parts: human-computer interaction interface, lip segmentation, CNN feature extraction, RNN time-series feature extraction, fully connected classification and result display, wherein the human-computer interaction interface is also called user interface.

The user interface is based on the python version of the Qt5 framework (PyQt5). The variables of the algorithm inference are transmitted by the signal slot to connect the front-end and the back-end information. The method of lip segmentation is the Dlib library of 68 points of the face, which can be well detected. For the lip position, the segmentation operation is performed using the following formulas (1) and (2).

Among them, the lip coordinate point calculates the position of the center of the lips, denoted as (x ₀ , y ₀ ), let w and h represent the width and height of the mouth image respectively, L ₁ and L ₂ represent the left and right and upper and lower dividing lines surrounding the mouth, respectively, Calculate the bounding box of the mouth according to the above formula.

The method used for lip reading the entire video is to extract a fixed frame, and the algorithm formula for extracting a fixed frame is shown in formula (3) (4):

Among them, x means dividing v into x blocks, A means taking i frames in an orderly manner in block _n , and F means the sequence number of the last frame taken by each block.

In this embodiment, a semi-random Fixed Frame Extraction Strategy (SFFES) is used to extract fixed frames from the video. After a large number of experimental studies, it is shown that the SFFES designed by the present invention has the characteristics of strong flexibility, low time complexity and strong anti-interference. The idea of the algorithm strategy is to first divide the video into regions and blocks according to the total number of video frames. The area range, the number x of the remaining frames is not greater than the number n of the area blocks, and then the range of the first x area blocks is expanded by 1. So far, the range of each area block has been semi-randomly allocated.

The calculation method has the characteristics of robustness, high calculation efficiency and strong eigenvector consistency, and can remove redundant information well.

After the mouth segmentation is completed, the resulting raw lips dataset is processed into standard 224x224 pixels. The overall structure of the model is shown in FIG. 4 . The model designed in this embodiment is only an embodiment, which can complete the basic lip reading recognition function. In the actual application process, a model for lip reading recognition can be designed as required. The model architecture designed in this embodiment uses VGG16 for feature extraction. A 4096x1 feature vector is generated at each moment, and the length of the model is 10. Therefore, the length of the feature vector for the entire action is 4096x10, and then the features are input into lstm for time series features. Finally, according to the output of the last lstm unit structure, the classification work is performed by two layers of full connection and softmax. By taking the top three maximum probability values of the final result sofmax, the top-3 accuracy rate is obtained, and then according to the slicing operation of the middle layer results of the model output, the intermediate results and visual images of feature extraction are obtained.

Due to the long-term dependency problem of traditional RNNs, the problem of gradually reducing the amount of information as time increases during iteration, resulting in outputs with a long time interval have very little impact on the input of the hidden layer at the current moment. Therefore, the traditional RNN is only suitable for processing data with short time series. When the time interval of the sequence is long, it is difficult for RNN to express the implicit correlation between the sequences.

Aiming at the problem that the hidden layer of RNN is prone to gradient disappearance and gradient explosion when transmitting information when processing long sequence data, a long short-term memory (LSTM) structure is adopted, which is specially used for processing long-term dependent sequences. Missing information problem. LSTM stores historical information by introducing memory units, and controls the addition and removal of information flow in the network by introducing three control gate structures including input gate, forget gate and output gate. Remember associations that need to be remembered in long sequences, and forget parts of useless information to better discover and exploit long-term dependencies from sequence data such as video, audio, and text. Figure 5 shows the operations performed inside a single LSTM cell, where x _t represents the input vector of the network node at time t, h _t represents the output vector of the network node at time t, and i _t , f _t , o _t , and c _t represent time t, respectively input gate, forget gate, output gate and memory unit.

The following will introduce the input gate, forget gate, memory unit and output gate inside the LSTM unit:

(1) Input gate: This gate is used to control the input node information. The input information consists of two parts. First, the sigmoid activation function is used to determine which new information needs to be input, and then the tanh function is used to select the new information that needs to be stored in the unit. The mathematical expression of the output i _t of the input gate and the candidate information g _t is:

i _t =σ(U _i x _t +W _i h _t-1 +b _i ) (5)

g _t =tanh(U _g x _t +W _g h _t-1 +b _g ) (6)

where U _i , Wi and b _i represent the weight and bias of the input gate _, respectively, U _g , W _g and b _g represent the weight and bias of the candidate state, respectively, σ represents the sigmoid activation function, and tanh is the activation function.

(2) Forget gate: This gate is used to control which information is discarded by the current LSTM unit. A function value between 0 and 1 is output through the sigmoid activation function. When the function value is closer to 1, it means that the node contains more useful information at the current moment, and more information will be retained to the next moment; When it is close to 0, it means that the node contains less useful information at the current moment, and more information will be discarded to the next moment. The mathematical expression of the forget gate f _t is:

f _t =σ(U _f x _t +W _f h _t-1 +b _f ) (7)

where U _f , W _f and b _f represent the weight and bias of the forget gate, respectively, and σ represents the sigmoid activation function.

(3) Memory cell: This cell is used to save state information and update the state. The mathematical expression of the memory cell c _t is:

c _t =f _t ⊙c _t-1 +i _t ⊙g _t (8)

where ⊙ represents the Hadamard product.

(4) Output gate: This gate is used to control the output node information. First, use the sigmoid function to determine what information needs to be output, and obtain the initial output value o _t , then use the tanh function to fix c _t in the (-1, 1) interval, and finally multiply the initial output value o _t point by point to obtain LSTM The output h _t of the cell. So h _t is jointly determined by o _t and memory unit c _t , and the mathematical expression of o _t and h _t is:

o _t =σ(U _o x _t +W _o h _t-1 +b _o ) (9)

h _t =o _t ⊙tanh(c _t ) (10)

where U _o , W _o and _bo denote the weight and bias of the output gate, respectively.

Logistic and SVM generally solve binary classification problems, and multi-classification can also be composed of multiple binary classifications. From a mathematical point of view, it is better to use SOFTMAX for mutually exclusive events; for non-mutually exclusive events, use a combination classifier such as Logistic or SVM. . In SOFTMAX, the probability that sample x belongs to class j can be expressed as:

Where j=1,2,...,K, then the loss function of SOFTMAX can be expressed as:

where K is the number of categories, p∈{0,1} ^K , and w is the network weight.

Generally, the probability distribution of the input features is obtained by the SOFTMAX classifier after the feature processing layer. As shown in Figure 6, the probabilities of the three categories are [0.88 0.10 0.02], and their sum is 1, which shows that the probability events are independent of each other.

As shown in Figure 7, on the human-computer interaction interface of the lip language recognition system, firstly, it has the function of displaying Top-3 recognition results and probability statistics; secondly, it has the function of displaying the video frames to be recognized; Feature extraction and time series feature extraction process, that is, it has the feature visualization function of CNN and the feature visualization function of RNN. Through the visualization function, the change of the feature vector in the feature extraction process can be observed; finally, it has the function of seeing the lip segmentation process of the video to be recognized. , through the above functions, the system can be used as a good analysis system for the problems encountered in the model verification process. The feature visualization of CNN in this embodiment includes lip images and high-dimensional image features of each convolutional layer, and the feature visualization of RNN includes visualized sequence features.

In this embodiment, different camera collection devices are used to capture video of the same target object at the same angle, and the lip language recognition system provided by this application is used to recognize the videos collected by different camera collection devices, and display them on the human-computer interaction interface The collected video, the lip segmentation process, and the recognition result of the video, based on the above situation, it can be determined that the lip area segmented by different camera collection equipment is inaccurate or deviated, and the accuracy of the video collected by different camera collection equipment can be verified by comparing the recognition results. Ranked by degrees to provide a basis for researchers to choose camera acquisition equipment.

This embodiment also provides video collection of the same target object at different angles through the same camera collection device, and the lip language recognition system provided by the present application recognizes the video collected by the same camera collection device at different angles, and the human-computer The collected video, the lip segmentation process and the recognition result of the video are displayed on the interactive interface. Based on the above situation, by comparing the video display results shot by the same camera acquisition device at different angles, it can be determined that different shooting angles will affect the accuracy of the recognition results. , the lip language recognition system can determine the most favorable shooting angle for recognition, so the system can be used as a problem analysis system in the model verification process, analyzing and predicting various problems encountered by the model from the source of data input.

The present invention provides a specific embodiment, using a lip language recognition system to obtain test results and analyze the test results of the lip language recognition system, specifically including:

The dataset in this example is built on an audiovisual database and consists of 10 independent English digit pronunciations (from 0 to 9) by 6 different target subjects (3 males and 3 females). Each target object pronounced each word up to 100 times. Videos are collected from the frontal view of each target object, which can sit naturally without any movement. The original size of each frame is 1920×1080 resolution, about 25 frames per second. To pinpoint the beginning and end of each articulated unit, using audio as an aid, separate each articulated word, each lasting approximately 1 second. Then, each isolated word video is further extracted to a fixed length of 10 frames, and after processing each video frame, a 224x224 pixel image is obtained as the standard value for the input of the CNN model.

The lip language recognition system aims to complete the recognition of video lip language content. The main functions include the function description of the human-computer interaction interface, the prediction result display Top-3 accuracy rate and the visualization of the algorithm reasoning process. In the prediction and recognition process, the accuracy of Top-3 can be predicted, and which pronunciation videos are similar and confusing in the inference results can be observed. In the visualization process, the feature changes and learning situation of each stage of CNN and RNN can be observed intuitively. It enables the system to better observe the intermediate process of the model in the process of algorithm inference and prediction, which is convenient for subsequent in-depth research and optimization of the algorithm model.

As shown in Figure 7, the human-computer interface will appear after the lip language recognition system is running. There are three function buttons on the human-computer interaction interface, namely, select video, visualize and identify video. On the right side of the function button is the final result of the system identifying the video, and on the right next to it is the Top-3 accuracy rate and result box, showing the prediction result in the style of an LCD liquid crystal display tube. The lower part of the human-computer interaction interface is used to display the intermediate process of algorithmic reasoning, including extracting fixed-length video frames, locating and segmenting the mouth position, visualization of CNN and dynamic visualization of RNN in each period.

As shown in Figure 8, after entering the human-computer interaction interface, click the "Select Video" button, and select the video folder will pop up, select the folder where the video to be recognized is located, the default is the test folder under the current project, if it does not exist is the current folder. Select the video to be recognized, click OK, and the folder window will automatically close, and the fixed-length frame will be extracted, and the backend window will display the background progress.

Taking the video of the digital pronunciation "Six" as an example, after selecting the loaded video of the digital pronunciation "Six", the prediction and reasoning process can be visualized, and click "Recognize Video" to start the recognition and prediction. As shown in Figure 9, the recognition result of the video is the digital pronunciation "Six", the probability of Top-3 being "Six" is 95.96%, the probability of identifying it as "zero" is 2.13%, and finally it is recognized as "Five" The probability is 0.63%. According to the display of Top-3, it can be confirmed that the recognition results are very accurate, and the performance of multiple recognitions is stable. The reason why the probability sum of Top-3 is not equal to 1 is that the classification is mutually exclusive, so each category will have a probability value. its category possibilities.

As shown in Figure 10, trigger the "Visualization" option of the human-computer interaction interface, and wait for about 1 second to display the results. Through the visualization process, the result of the feature extraction layer of the intermediate convolution can be observed, and the changes of image features and time series features can also be observed, so that the reasons for poor recognition in the process of fusion neural network can be analyzed.

The video frames, lip images of each convolutional layer, high-dimensional image features, and/or visualized sequence features displayed on the human-computer interface are a sequential process advancing in time, such as shown in Figure 10, the first row shows The fixed video frame extracted from the video to be recognized, the second row shows the lip feature segmented from the video frame displayed in the first row, and the third, fourth and fifth rows show the feature extraction of the lip feature. In the middle process, the third line is the lip image of convolutional layer 1, the fourth line is the lip image of convolutional layer 2, the fifth line is the high-dimensional lip image feature, and the sixth line shows the sequence features in the time series feature extraction process. The visual display can observe the lip language recognition process in the model, and determine which model has high recognition accuracy and efficiency by showing different model recognition processes. It can also show the recognition results of the same model under different parameters, which is beneficial to Quickly determine model parameters.

Through the visualization of image features and sequence features, the role of the activation function can be well seen, so that the subsequent calculation can reduce the feature dimension and reduce the dependence of computing power. It can be seen that the initial features are not obvious, and the final features are polarized, indicating that the model used by the system has a good fitting force and can meet the requirements of visual lip-reading pronunciation. The lip language recognition system takes into account the intermediate process display of CNN and RNN models, and completes the lip reading segmentation work, which can well complete the background and support for in-depth study of deep learning theory in lip language recognition work.

The specific display content and display method on the human-computer interaction interface can be set in the configuration file according to the visualization requirements. In addition, the human-computer interaction interface can provide bilingual replacement of Chinese and English, which makes the system have better human-computer interaction and brings convenience to more users.

After the interactive interface and system functions are completed, it is first necessary to pay attention to whether the model converges during the training process. When the empirical values of hyperparameters are not set properly or the model construction is not reasonable enough, it is difficult for the model to converge, resulting in the model not working. Therefore, in this training process, the change curve of the loss function of the training set and the test set in different periods was recorded. Through this curve, it can be concluded whether the algorithm model proposed in this paper can learn the characteristics of the data set, so as to evaluate the convergence of the algorithm.

In order to determine the change trend of Loss, the experiment was recorded for 70 rounds, and the results were recorded once per iteration. Figure 11 shows the change curve of the loss function Loss in different periods. One epoch represents the round of training the entire data set. Generally, most data sets will converge after about 10 rounds of training. Continuing training may lead to overfitting. .

It can be seen from Figure 11 that it tends to be stable when the epochs is about 15 times. At this time, the model has reached the optimal solution, and it continuously oscillates in the subsequent training and learning process, indicating that the limit of model learning has been reached. Since the update parameters are updated iteratively based on the training set, the loss of the training set will be relatively smaller than that of the validation set, and both the training set and validation set losses will gradually converge with the number of training iterations, indicating that there is no abnormal problem in the data set and the model is in Good performance on lip recognition. At this point, it can be verified that both the dataset and the model are working.

After verifying that the model has converged, the CNN-LSTM model based on the attention mechanism proposed in this embodiment further tests its performance on the test set. Figure 12 shows the change curve of the recognition accuracy, in which each iteration is recorded once, and the ordinate recognition accuracy is the recognition rate (%). In order to compare the improvement of model performance, the experiment compared the CNN-LSTM network model, so that the introduction of the attention mechanism can improve the performance of the model by controlling the variables. The overall training trend is consistent with the loss function of the loss function, and tends to be stable when the epochs is about 15 times, indicating that the parameter update in the training process is continuously approaching the optimal solution of the model, and the optimal solution has been reached at this time. At the same time, the accuracy of the basic network CNN-LSTM model is much worse than the network of the present invention, indicating that the attention mechanism can well understand the important key frames of the video and reduce the noise of the sequence image. So far, it can be concluded that the attention mechanism has a good improvement and robustness in model performance.

It can be seen from FIG. 12 that the overall performance of the method for introducing attention in this embodiment is better than that of the general fusion neural network method, and as the training continues, the accuracy rate increases significantly. The method of this embodiment needs to calculate the attention weight. Therefore, at the beginning of the learning phase, the accuracy rate will shake significantly, indicating that the attention weight has not been learned, and the model parameters need to be trained. The final epochs is about 15. The method of this embodiment basically completes the training, and the overall performance is better than the general fusion neural network.

The experimental results show that the CNN-LSTM model based on the attention mechanism is better than the CNN-LSTM model on the results of each digit pronunciation. Figure 13 compares the results of each independent pronunciation. The performance of pronunciation "Two", pronunciation "Four" and pronunciation "Nine" is significantly improved, which shows that noise is more likely to appear in the word video of monosyllabic pronunciation, resulting in poor recognition of the results. The performance of the model is greatly improved when the video noise is reduced. For the pronunciation of "Five" and "one" of complex lip movements, the improvement is not obvious, and it shows that it is difficult to learn video noise and spatiotemporal features. The lip movements for pronouncing "zero" change less, and the tongue movement is a key factor in the pronunciation process, so it is difficult for the model to predict.

The lip language recognition system provided by the present invention can better observe and analyze the pros and cons of the model performance and each link in the process from the original video to the final recognition, thereby improving and optimizing the model and algorithm. Therefore, the realized lip language recognition system has important practical significance. On the other hand, the present invention looks at the development direction of lip language recognition technology from the perspective of lightweight model. By reducing the structure of convolution and full connection, the calculation amount of the model is greatly reduced, and the GPU is reduced to a certain extent. Dependency, reducing costs and hardware requirements.

Through the above-mentioned specific embodiments, the validity of the algorithm model is verified by using the test data set to conduct test experiments. The experimental results show that the video lip recognition system designed by the present invention has high availability and feasibility, and the proposed RNN fusion neural network model based on CNN and attention mechanism has high efficiency and feasibility. Compared with other algorithm models, it has a higher accuracy.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above are only examples of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the application for pending approval of the present invention. within the scope of the claims.

Claims (10)

1. A lip language identification system, comprising: a human-computer interaction interface and an algorithm module; the human-computer interaction interface is connected with the algorithm module through a signal slot;

the human-computer interaction interface is used for acquiring a video to be identified;

the algorithm module is used for carrying out lip language identification on the video to be identified to obtain a lip language identification result;

and the human-computer interaction interface is also used for displaying the lip language recognition result and displaying the lip language recognition process according to time sequence.

2. The system of claim 1, wherein the algorithm module comprises:

the fixed frame extraction submodule is used for extracting a video frame to be processed from a video to be identified based on a semi-random fixed frame extraction strategy;

the segmentation submodule is used for segmenting the lip image from the processed video frame to obtain a lip data set;

and the recognition submodule is used for recognizing each lip image in the lip data set based on the designed model to obtain a lip language recognition result.

3. The system of claim 2, wherein the identification submodule comprises:

the characteristic extraction unit is used for carrying out characteristic extraction on the lip images to obtain image characteristics, and carrying out slicing operation on the lip images and the image characteristics of the curling layers to obtain visual lip images and high-dimensional image characteristics of the curling layers;

the time sequence feature extraction unit is used for extracting time sequence features from the image features to obtain sequence features, and performing slicing operation on the sequence features to obtain visual sequence features;

and the classification unit is used for classifying the extracted time sequence characteristics to obtain a lip language identification result.

4. The system of claim 3, wherein the feature extraction unit is a Convolutional Neural Network (CNN).

5. The system of claim 3, wherein the timing feature extraction unit is a Recurrent Neural Network (RNN).

6. The system of claim 3, wherein the classification unit is a softmax classifier.

7. The system of claim 3, wherein the human-machine interface comprises:

selecting a video option for acquiring a video to be identified when triggered;

the identification video option is used for carrying out lip language identification on the video to be identified when the video to be identified is triggered to obtain an identification result;

a visualization option for displaying the lip language recognition process and the lip language recognition result based on a configuration file set by visualization requirements when triggered;

the display content comprises a video frame to be recognized, lip images obtained by dividing the video frame, visual curled layer lip images, visual high-dimensional image features, visual sequence features and/or at least one recognition result corresponding to the video to be recognized.

8. The system of claim 2, wherein the fixed frame decimation sub-module is specifically configured to:

determining a fixed frame number to be extracted based on a priori condition;

dividing a video to be identified into a plurality of area blocks according to the number of overall video frames;

wherein the area coverage in each area block is maximized on average.

9. The system of claim 2, wherein the video to be identified comprises:

and acquiring the video to be identified for the same target object based on at least one acquisition device.

10. The system of claim 1, wherein the human-machine interface is designed and constructed through a PyQt5 framework.

Images